Deconstructing Central Rating in Clinical Trials | Part II

In Part I of this series, we defined Central Rating and outlined the valuable benefits that it provides, including assurance of data validity and reliability and increased likelihood of study success. In Part II, we analyze several commonly held misconceptions about Central Rating.

Central Rating: There must a downside, right?

Respecting the Subject Experts
Given their independence from study sites, CRs will not have a full clinical ‘picture’ of each subject, both in relation to the subject’s manifestations of study indication symptoms, and in terms of the subject ‘Gestalt’ that represents all psychiatric, medical, and psychosocial subject details across time. Furthermore, subjects may be more inclined to share particularly sensitive information, such as details about a traumatic event or suicidal ideation, with a more familiar site rater. Indeed, site raters should be considered subject experts and remain integral in clinical trials. The involvement of site raters and other site-based personnel is pivotal in enhancing recruitment, mitigating subject attrition, ensuring smooth Central Rating scheduling and processes, and most importantly, protecting subject safety. Continual site engagement is critical in study success.

Mistaken Beliefs
Throughout the years, we at Signant Health have seen evidence of Central Rating misconceptions. Let’s evaluate these misconceptions.

Misconception: CRs take too long to administer assess
Reality: An ideal Central Rating training program will include an interview skills component that teaches best practices for balancing the thorough interviewing geared at obtaining all needed information for accurate ratings with efficiency-enhancing interviewing choices. Signant Health’s experience is that outlier long scale durations for our CRs are uncommon.

Misconception: There is a long wait period for sites to receive ratings after a CR has evaluated a subject.
Reality: In fact, when a solid Central Rating logistical infrastructure is built well in advance of FPI, CR ratings can be provided to sites within minutes, avoiding delays in study procedures and enrollment decisions.

Misconception: Aside from being blinded to visit, CRs are no different than site raters and are just as biased.
Reality: Psychologist Nathaniel Branden said, “The first step toward change is awareness. The second step is acceptance.” CRs certainly have human brains, but bias-related risks can be considerably lowered by purposively training CRs to understand the cognitive biases that may be activated for site raters during administration of clinical trial assessments and those that may be held by study participants. Risk can be further mitigated by training CRs on the ways that such biases can adversely impact scale outcomes through distortion of their own and study participants’ perceptions of symptoms, and through the teaching of neutral and objective administration behaviors and scoring decisions.

Misconception: Use of CRs for screening and baseline assessments leads to inappropriate screen failures.
Reality: CRs will ideally be visit-blinded to protocol inclusion and exclusion criteria. Therefore, a study participant’s randomization potential will be unknown to the CR. CR-generated scale total scores and item ratings that fall outside of protocol-dictated inclusion thresholds can therefore be held in high confidence as indicative of participants whose symptoms are inconsistent with the sub-population being sought by the sponsor.

Misconception: CRs are distant and cold with study participants and do not elicit as much information during interviews.
Reality: CRs should be trained robustly on interviewing best practices. An optimal CR interview will strike a balance between maintenance of clinical research boundaries and neutrality and healthy rapport with the study participant. At Signant Health, our CRs understand this balance and achieve standardization while avoiding unnecessarily rigid administration. 

Misconception: CRs are often inexperienced.
Reality: “A good decision is based on knowledge and not on numbers.” – Plato

Savvy vendors are well aware that the choice of CRs must be methodical and based on multiple layers of information, including CV, additional experience information gathered in a targeted manner, language fluency, ability to synthesize information that can at times be disparate, ability to successfully complete complex training, cognitive objectivity and flexibility, and interviewing ability.

Sound CR training and management will ensure that the Central Rating process is time-efficient, that CR interviews involve the optimal application of scale conventions and obtain rich subject information, and that CR-generated data are unbiased, valid, and reliable.

Are you sure it’s Central Rating?

Here at Signant Health, we have also encountered confusion about other clinical trial approaches that are mistaken for Central Rating.

• Central Quality Reviews
  • Central Quality Reviews are appraisals by an independent clinician of the quality of a site rater’s scale administration and scoring based on review of scale data and, if available, a video- or audio-recording of scale administration. It is differentiated from Central Rating in that Central Quality Reviewers (CQRs) do not meet directly with participants and CQRs do not generate scale source data.
• Independent Scoring
  • Central Quality Reviews of site-administered scales ideally involve secondary, visit-blinded, independent rating based on review of a video- or audio-recording of scale administrations by site raters. This approach allows the degree of concordance between the two sets of ratings – one from the site rater and one from the secondary clinician - to serve as an indicator of the quality of the site rater’s performance. Independent ratings are not used in the sponsor’s study data analytics as is the case for Central Rating data, but rather are used for quality comparison purposes.
• Central Scoring
  • Central Scoring involves the independent rating of audio- or video-recorded site rater assessments by a secondary clinician. It is differentiated from Central Rating in that the Clinical Scorer (CS) does not interview the subject directly and does not generate source data, but rather formulates a second set of ratings for the same scale and visit based on the original site rater’s recorded assessment. CS scores are used for quality data comparison, consensus rating decisions, or, in some studies, endpoint statistical analyses.
• Tandem Rating
  • In a Tandem Rating model, the site rater interviews and rates the study participant using an electronic Clinical Outcome Assessment (eCOA) scale designed with a series of stem and probe questions for each item, and there is an additional computer-administered and computer-scored electronic Patient Reported Outcome (ePRO) version of the scale designed to present targeted questions based on pre-programmed algorithms. This approach generates a pair of site rater and computer ratings that can be compared for quality purposes. No CRs are involved in the Tandem Rating process.

Distinguishing Central Rating from a Decentralized Trial Model

Since the start of the Covid-19, there has been an increase in Decentralized Trials, those in which trial activities occur outside of the trial sites, such as in the home of a study participant or caregiver, a local healthcare facility, or a community laboratory. It is common for trials to take a hybrid approach, where key endpoint scales, or scales that require collection of observational data or physical examination are administered and scored by site raters, while other study activities are completed remotely, sometimes by CRs. However, it is noted that although CRs evaluate study participants and interview caregivers or study informants remotely outside of study sites, not all Decentralized Trials employ CRs. The main goal of Decentralized Trials differs from that of Central Rating. Decentralized Trials seek to decrease the burden for sites, participants, and caregivers/informants, while the mission of Central Rating is to ensure valid and reliable data through removal of potential biases in exchange for enhanced clinical objectivity and incorporation of scale or indication experts.

Good News! Signant Health provides all of these solutions – Central Rating, Central Quality Reviews with Independent Scoring, Central Scoring, Tandem Ratings, and support for Decentralized Trials.

With over 13 years of Central Rating and over 20 years of experience with Central Quality Review, Central Scoring, and Tandem Rating across an expansive list of CNS and non-CNS indications and scales, Signant Health is well-positioned to provide targeted, data quality-oriented trial solutions for small and large global trials alike. Signant Health has a cohort of approximately 250 contracted expert Clinical Consultants across 40+ countries and 35 languages and boasts a 93% Clinical Consultant retention rate. Our Clinical Consultants are carefully selected and robustly trained at the universal [study and indication agnostic] level and the study-specific level. Clinical Consultant selection, training and calibration, monitoring, and management is led by a team of over 50 Signant Health Digital Health Science Clinical Scientists working in collaboration with a seasoned Global Project Management Team.

Signant Health’s Clinical Consultant approach includes:

• Clinical Consultant vetting and selection - Collaboration between Signant Health Clinical Scientists and the Signant Health Consulting Management Team
• Tailored Consultant contracts
• Legal and regulatory process oversight
• HIPAA and GDPR compliant telemedicine capabilities​
• Universal Consultant training on diagnostic indications, scale administration and scoring, clinical interviewing, mitigation of placebo response and cognitive biases, GCP/GDP, and Signant Health quality monitoring processes
• Universal inter- and intra-rater reliability-focused scale calibration exercises
• Study-specific pre-FPI online and live training on scales and processes
• In-study, dynamic training and guidance, recalibration exercises, and case consultation
• Use of secure and compliant Zoom Healthcare platform for remote assessments with recording capability
• Web portal for CR rating documentation and real time data access ​for sponsors
• In-study secondary quality monitoring of data and conclusions generated by Clinical Consultants
• Audit trail reporting
  
  

References

Decentralized Clinical Trials for Drugs, Biological Products, and Devices Guidance for Industry, Investigators, and Other Stakeholders DRAFT GUIDANCE. (May 2023). https://www.fda.gov/media/167696/download

Hammond, M, Stehlik, J, Drakos, S. et al. Bias (2021). Medicine: Lessons Learned and Mitigation Strategies. J Am Coll Cardiol Basic Trans Science. 2021 Jan, 6 (1) 78–85.

Powsner, S., Goebert, D., Richmond, S, J., Takeshita, J. (2023). Suicide Risk Assessment, Management, and Mitigation in the Emergency Setting. Focus, 21(1) 11.

Psychedelic Drugs: Considerations for Clinical Investigations Guidance for Industry DRAFT GUIDANCE. (June 2023). https://www.fda.gov/media/169694/download

Targum, S., Wedel, P. C., Robinson, J., Daniel, D. G., Busner, J., Bleicher, L. S., Rauh, P., & Barlow, C. (2013). A comparative analysis between site-based and centralized ratings and patient self-ratings in a clinical trial of Major Depressive Disorder. Journal of Psychiatric Research. 47 (7) 944-954.

Williams, J. B., Popp, D., Kobak, K. A., & Detke, M. J. (2012). P-640 - the Power of Expectation Bias. European Psychiatry, 27 (S1), 1–1.

About the Authors

Juliet Brown, Director of Endpoint Reliability and a Clinical Thought Leader at Signant Health, has over 25 years of clinical and research experience, specializing in MDD, Bipolar Disorder, Anxiety Disorders, Psychotic Spectrum Disorders, Substance Use Disorders, and Cognitive Behavioral Psychotherapy. She holds a PhD and Master's Degrees in Clinical Psychology from Drexel University. Before joining Signant Health 8 years ago, Dr. Brown provided psychotherapy to individuals with Severe Mental Illness and treated Substance Use Disorders. At Signant, she oversees phase 1-3 global trials, offers clinical guidance, and serves as a Blinded Data Analytics Scientist and Subject Matter Expert.

Dr. Joan Busner has over 35 years of experience as an academic psychiatric researcher, serving as Principal Investigator for 49 clinical trials and Sub-Investigator for 35 more. She has authored or co-authored over 140 peer-reviewed articles and presentations. Before joining Signant Health, she directed psychiatric clinical trials at two major medical schools and served on University IRBs for 20 years. Currently an Affiliate Associate Professor of Psychiatry at Virginia Commonwealth University, Dr. Busner leads studies at Signant on pediatric, rare, and psychiatric disorders, and has trained thousands of clinical trial investigators worldwide.

Dr. Daniela Chereches is a Clinical Scientist at Signant Health and has been with the company for the past 5 years. She has over 10 years of clinical research experience in various indications, to include Psychotic Disorders, MDD, Bipolar Disorder, Anxiety Disorder, Alzheimer’s Dementia, Lupus and Myasthenia Gravis. Prior to joining Signant Health Dr. Chereches conducted clinical assessments to patients with psychiatric disorders in both inpatient and ambulatory settings and provided leadership to a team of physicians, clinical raters and investigators. At Signant Health Dr. Chereches provides clinical oversight in phase 1-3 global trials and consultative guidance to various clinical teams, manages study endpoint reliability programs, and has served as both a Central Rater and a Central Quality Reviewer.

Margot Oakley is a Masters-level Registered Nurse with a diverse clinical background. Her extensive nursing experience covers various medical diagnoses and settings. She has prior experience in clinical trial work at research sites and as a Clinical Research Associate for a CRO. For the past 16 years with Signant Health, she focuses on pediatric and adult CNS clinical studies, with recent emphasis on pediatric rare diseases.