Deconstructing Central Rating in Clinical Trials | Part I

With the unfortunately high rate of clinical trial failure and the urgent need to mitigate threats to trial success, we attempt to clarify one approach to ensuring data quality and trial success - Central Rating. In this two-part series, we define this solution and provide a breakdown of its key benefits and subsequently address common misconceptions about Central Rating.

What is Central Rating? 

There is general agreement in the research community about the definition of Central Rating: The use of a centralized group of raters that are external to the site and typically chosen, trained, and managed by a third-party entity to remotely administer and score diagnostic or severity scales in lieu of site raters. The details are relatively less understood.

What are the potential benefits of Central Rating?

Management of Cognitive Biases 

There are a variety of intended benefits of Central Rating, the most commonly known of which is mitigation of clinician cognitive biases. Central Rating can effectively manage potentially signal-clouding rater or participant biases, including:  

• Enrollment bias, where a rater’s conscious or unconscious wish to help a subject gain entry to a trial adversely affects diagnostic and severity assessment accuracy 

• Expectation bias, which occurs when prior expectations about potential outcomes on the part of a rater, participant, or study partner objectively affect their study behavior 

• Confirmation bias, where information is interpreted in a manner consistent with one’s beliefs, and contrary information is discounted 

• Framing, or the tendency to restrict information flow to increase the likelihood of arriving at a preferred conclusion 

• Attribution error, which involves misinterpretation of causal factors associated with behavior 

• Anchoring bias, or the inability of a rater to expand or flex thinking and decision-making beyond a subjectively familiar level 

• Trajectory bias, wherein a site rater forms a rating-affecting bias of the subject’s expected illness course over time 

Central Raters (CRs) are dissociated from any site-related enrollment quotas or incentives, thus injecting objectively into the rating process. If CRs vary from visit to visit for a single subject, they do not have the degree of interaction that can lead to potential affiliation with the subject as is the case for site raters. CRs are generally blinded to protocol inclusion and exclusion criteria, a subject’s study visit, and a subject’s duration of time in a study, resulting in the potential for a truer picture of a subject’s actual status at each visit. Central Rating can facilitate enrollment of appropriate subjects for which there is high diagnostic confidence and from which valid and reliable data can be captured. The objectivity afforded by Central Rating is critical in studies whose treatments involve tell-tale side effects and so-called “functional unblinding,” in which the effects of the study treatment are sufficiently apparent as to compromise blinding. Regulators may ask sponsors to assure rating blindness by using CRs for efficacy scales in studies with highly sedating investigational treatments, or those that cause unusual alterations in sensory perceptions such as ketamine or other psychoactive agents.  

But the benefits of Central Rating go beyond bias mitigation. For example:  

• Central Rating is associated with lower data error variance due to the use of a small group of highly trained, well-calibrated raters. Rather than relying on the contribution of hundreds of raters across hundreds of sites, one is able to reduce inter-rater variability by using only a small tightly calibrated group.   

• The Central Rating solution provides the opportunity for expert raters in studies with sites whose raters may lack experience in research assessments, for example in the rare and ultrarare disease space, where patients are often followed by a small number of physicians worldwide who possess disease expertise but are inexperienced in research assessments. In such studies, CRs can perform assessments with a level of expertise that could not be obtained at the site. This occurs also when a measure external to the disease is required, such as a psychiatric safety scale required in a dermatology study. CRs can be employed to administer such scales expertly, freeing site raters to administer the dermatology ratings.

Central Rating also affords several relatively lesser-known benefits:

• Central Rating allows for an expanded scope of initial and ongoing training and calibration activities due to smaller rater group size and easier access to CRs.

• The use of more intensely trained CRs ensures standardization of scale administration and consistent use of administration and rating conventions.

• CRs generally demonstrate tight calibration, and the Central Rating model facilitates more frequent quality monitoring. These practices lead to decreased frequency of rater administration and scoring errors and less data ‘noise,’ thus improving data quality.

• Remote rating by CRs decreases the burden on busy sites and increases the ease of assessments for participants and caregivers.

In Part II, we will discuss commonly held misconceptions about Central Rating.

About the Authors

Juliet Brown, Director of Endpoint Reliability and a Clinical Thought Leader at Signant Health, has over 25 years of clinical and research experience, specializing in MDD, Bipolar Disorder, Anxiety Disorders, Psychotic Spectrum Disorders, Substance Use Disorders, and Cognitive Behavioral Psychotherapy. She holds a PhD and Master's Degrees in Clinical Psychology from Drexel University. Before joining Signant Health 8 years ago, Dr. Brown provided psychotherapy to individuals with Severe Mental Illness and treated Substance Use Disorders. At Signant, she oversees phase 1-3 global trials, offers clinical guidance, and serves as a Blinded Data Analytics Scientist and Subject Matter Expert.

Dr. Joan Busner has over 35 years of experience as an academic psychiatric researcher, serving as Principal Investigator for 49 clinical trials and Sub-Investigator for 35 more. She has authored or co-authored over 140 peer-reviewed articles and presentations. Before joining Signant Health, she directed psychiatric clinical trials at two major medical schools and served on University IRBs for 20 years. Currently an Affiliate Associate Professor of Psychiatry at Virginia Commonwealth University, Dr. Busner leads studies at Signant on pediatric, rare, and psychiatric disorders, and has trained thousands of clinical trial investigators worldwide.

Dr. Daniela Chereches is a Clinical Scientist at Signant Health and has been with the company for the past 5 years. She has over 10 years of clinical research experience in various indications, to include Psychotic Disorders, MDD, Bipolar Disorder, Anxiety Disorder, Alzheimer’s Dementia, Lupus and Myasthenia Gravis. Prior to joining Signant Health Dr. Chereches conducted clinical assessments to patients with psychiatric disorders in both inpatient and ambulatory settings and provided leadership to a team of physicians, clinical raters and investigators. At Signant Health Dr. Chereches provides clinical oversight in phase 1-3 global trials and consultative guidance to various clinical teams, manages study endpoint reliability programs, and has served as both a Central Rater and a Central Quality Reviewer.

Margot Oakley is a Masters-level Registered Nurse with a diverse clinical background. Her extensive nursing experience covers various medical diagnoses and settings. She has prior experience in clinical trial work at research sites and as a Clinical Research Associate for a CRO. For the past 16 years with Signant Health, she focuses on pediatric and adult CNS clinical studies, with recent emphasis on pediatric rare diseases.