5 Reasons to Prioritize Rater Training

The quality of assessments is an essential clinical trial component that is often overlooked. In placebo-controlled trials, accurate ratings are needed to differentiate the effects of the investigational treatment from placebo to determine whether the proposed treatment works. In open-label trials, accurate, replicable ratings are needed to describe symptoms and changes that may occur over the course of the trial.

Without proper training, though, raters may not evaluate each clinical trial participant – or even each symptom – accurately, reliably, or consistently over study visits. This puts the study at considerable risk for inconclusive results or failure.

Our Science and Medicine team members are experts in clinical trials methodology, including rater training. They provided the following five reasons sponsors should consider implementing a comprehensive training program at the outset and throughout the course of a clinical trial.

1. Raters vary greatly.

Site raters vary greatly in academic backgrounds, indication experience, clinical trial experience, and exposure to diagnostic, safety, or efficacy scales. Scales that are common in one country may not be used in another. Additionally, scales that are common in clinical practice may not be used in clinical trials. For these reasons, it’s essential to assess rater experience and design a training program that will enable raters to achieve the desired competency and skill level needed to rate proficiently.

It is a costly mistake to assume that raters will score consistently solely based on their experience or medical degree. All raters, regardless of their educational level or background, should receive study-specific training and calibration to ensure scoring practices and conventions are followed, especially since these can vary between studies and sponsors. Establishing interrater consistency at the outset of a trial is a critical step in enhancing the likelihood of a trial’s success.

Our team can provide recommendations as to preferred backgrounds for given scales and will work with you to customize enriched training programs for less experienced raters. 

2. Placebo response rates are increasing.

Placebo response rates are increasing for several CNS and non-CNS indications. For some conditions, the presence of a newly marketed treatment may raise patients’ and raters’ expectations that all treatments (even investigational ones) will work. Sometimes it’s the absence of any current treatment that may excessively raise hopes and expectations.

Regardless of the cause, rising placebo rates threaten a study’s ability to identify the effectiveness of the investigational treatment. Skilled rater training can help raters recognize and correct common behaviors, well-meaning comments, and ways of handling patient questions that may be increasing placebo response. Skilled rater training can also help raters recognize and minimize heightened patient expectations, allowing raters to elicit, probe, and score efficacy and safety assessments in a manner that maintains neutrality.

3. Hybrid and decentralized trials are becoming more common.

For many trials, the pandemic necessitated the use of remote visits. There has been an increase in new studies that are purely remote (“decentralized”) and those that use a mixed visit methodology, with some visits in-clinic and some remote (“hybrid”). These models remain on the rise as they arguably increase efficiencies and retention rates. Leveraging secure, digital technology, sponsors are now running studies with remote visits more frequently.

To ensure quality data, it is important to adequately prepare study staff and patients for remote assessments. Raters must be specifically trained in best administration and scoring practices for remote assessments. If the study involves both in-clinic and remote ratings of the same scale, training should cover methods for improving similarity of measurements across venues.

4. Diagnostic practices differ.

Diagnostic practices differ from country to country, state to state, clinic to clinic, and clinician to clinician. Even when raters use a similar diagnostic system, such as DSM-5 or ICD-10, the interpretations of what constitutes a symptom are often idiosyncratic and may vary greatly even among established experts.

For a study to be successful, all enrolled patients must meet the agreed upon set of diagnostic criteria for which the investigational product is targeted. Establishing nuanced and consistent diagnostic practices across a wide range of rater backgrounds is an art, and the potential to offend or offput experienced raters is high.

Signant has always based its training on its skillful and sensitive methods for building consensus and encouraging adherence to conventions “for the good of the study.”   

5. Training companies differ.

Not all rater training programs are created equal. Your choice may make the difference between a trial that succeeds and a trial that fails. Careful consideration and evaluation of the company’s scientific expertise, training experience, operational wherewithal, and global reach should be a priority for sponsors and CROs planning trials. 

Often neglected is the training company’s ability to engage collegially with raters and investigators in a manner that encourages learning and allows for remediation. Rater training companies not expert in this art often alienate investigators, making retraining efforts counterproductive and sometimes, even threatening investigators’ willingness to enroll and retain patients.

How does Signant Health’s Rater Training & Qualification help clinical trials?

Signant Health has been a rater training leader for the past 18 years. The evidence generation company’s Science and Medicine team, supported by an international staff of clinical and operational delivery experts, ensures the highest quality in training, tracking, and follow up to enable studies of all sizes, designs, and indications to succeed. Our expert trainers are internationally recognized for their scale and disease expertise, audience engagement, and abilities to not only set conventions but achieve needed consensus amongst raters of disparate educational and clinical backgrounds.

Under the direct supervision of an indication and methodology expert from the Science and Medicine team, Signant will provide your study raters the knowledge and tools to accurately and consistently assess patients, as well as detect changes over time.

Here’s what our proven program includes:

• Protocol-customized, expert-led, interactive training in diagnostic, safety, and efficacy outcomes
• International training that helps standardize practices while ensuring cultural sensitivity to patient and ratings nuances
• Workshops and assessment of interviewing skills to ensure sensitive, neutral, consistent scale administration
• Placebo mitigation training to manage patient and rater expectations
• Provision of standardized scoring conventions with consensus-building and scoring certification
• Ongoing monitoring and calibration to prevent raters from “drifting” from trained conventions (This is accomplished through review of incoming data scoring as well as review of audio video-recorded interviews with local language expert remediation as needed.)

Diagnosis and clinician-administered assessments form the foundation on which a study is built, making it crucial to ensure that raters within and across sites are accurate and consistent. Establishing and facilitating uniform, credible standards, informed by disease and scale expertise, will reduce intra- and inter-rater variability. It will also allow your team to detect investigational treatment-related changes over time and ensure the collected data are of the highest quality.   

Want to learn more? Contact our team today.

Signant Health

• facebook
• twitter
• google-plus
• linkedin

Signant Health is the evidence generation company. We are focused on leveraging software, deep therapeutic and scientific knowledge, and operational expertise to consistently generate quality evidence for clinical studies across traditional, virtual, and hybrid trial models. For more than 20 years, over 400 sponsors and CROs of all sizes – including all of Top 20 pharma – have trusted Signant Health solutions for remote and site-based eCOA, eConsent, IRT, supply chain management, and data quality analytics.