Clinician Ratings
Transitioning from paper-based to electronic ClinRO assessments in clinical trials demands meticulous planning and adherence to best practices. Signant Health is at the forefront of this evolution, leveraging years of expertise to define and refine methodologies that ensure the fidelity and reliability of ClinRO data capture.
Let's explore essential recommendations developed by industry experts to guide sponsors and eCOA providers through the seamless migration and implementation of ClinROs, highlighting critical steps and considerations for optimizing study outcomes.
Filling in the Gap in eClinRO Guidance
Nearly 15 years ago, the FDA first published its final guidance on migrating validated paper patient-reported outcome (PRO) assessments to electronic formats for the delivery of ePRO. In 2009, a paper by Stephen Coons and ISPOR colleagues offered a set of recommendations to address the critical need to demonstrate equivalence between paper and ePRO assessments. In particular, that group focused on the level of modification in the migration from paper to ePRO, and the potential impact on equivalence.
To date, there has been no established similar regulatory guidance or industry consensus on the migration and implementation of clinician-reported outcomes (ClinROs). To fill that critical gap, a panel of member representatives from the Critical Path Institute’s (C-PATH) eCOA and PRO Consortiums collaborated over several years to develop consensus-defined best practices and recommendations for migrating and implementing ClinROs for electronic data capture. A summary of the panel’s recommendations is provided in this article, and a more comprehensive overview can be found in the full publication.
Differences Between PROs and ClinROs
A PRO presents a question or similar request for information directly to a participant, or “patient” in a clinical trial. The participant records their response to the question without assistance or interpretation by clinicians or anyone else. In contrast, a ClinRO is administered by a clinician, or rater, with appropriate training and experience. The rater is required to interpret the patient’s responses, symptoms and/or presentation when scoring a ClinRO.
ClinROs can vary in complexity, with the administration and scoring procedures, clinical content, and navigation elements among the intricacies of individual ClinROs. The inherent variability of ClinROs creates challenges for eCOA migration, particularly compared with the standardized structure of PROs. The C-PATH panel recognized these considerations as we sought to develop a set of recommendations and best practices for eClinRO equivalence and migration, similar to those published and refined for ePROs. From the start, the consortia acknowledged the role that clinicians with expertise and experience would need to play in the process, and that ClinROs present unique challenges relative to most PROs.
Impact of eCOA for ClinROs on Data Quality
eCOA today is preferred by many stakeholders, including regulators, for reasons ranging from the ability to standardize data to reduction of basic data management issues such as queries and missing data. There are substantial efficiencies and benefits to study stakeholders in using a well-designed, validated eCOA platform.
The move from paper to eCOA for ClinROs, when faithfully migrated and designed, can standardize administration across the study, and, by implementing administration and scoring conventions, improve the quality of the outcome data collected. For example, Alzheimer’s studies were among the first to widely implement eCOA ClinROs with experienced vendors that understood how to integrate the scale conventions into eCOA. Collaborative sponsor publications with those experienced vendors have demonstrated that error rates on common Alzheimer’s scales, such as the CDR-sb, ADAS-Cog, and MMSE, are reduced by up to 75% when administered as eCOA, compared to paper administration.
Optimal Migration Practices
The consortia members recognized that properly designed electronic ClinROs have tremendous potential to improve the quality of data collected, many in therapeutic areas such as Alzheimer’s that have tremendous unmet needs. The paper proposed a set of best practices for “faithfully” migrating and implementing ClinROs electronically that can be utilized by sponsors and eCOA vendors alike. Faithful migration in this context means that modifications to an assessment’s instructions, presentation and display of content, navigation and branching, and scoring requirements will allow a rater to interpret and respond to items in the same way the validated paper scale intended.
The capabilities of eCOA platforms have substantial variation, and delivery modes and devices must be considered initially when evaluating the feasibility of ClinRO migration. Migration of ClinRO assessments should follow best practices as for PRO measures, but differences in ClinROs can demand more robust capabilities than ePRO form-capture platforms may support. For example, ensuring faithful migration of clinical content, navigation, and scoring algorithms may require scripting and logic within the eCOA system. In addition, some highly complex requirements of individual ClinROs may require additional evidence to support comparability. Finally, there are some ClinROs that are not suitable for eCOA administration, and the best practices support that conclusion when appropriate.
To determine the level of evidence needed to produce faithful migration, the consensus panel developed a classification system that hinges on whether changes to a particular scale will impact interpretation of the administration conventions and responses. The table below provides guidelines for changes that are likely to impact interpretation. The approval of engaged copyright holders is the optimal starting point for any migration. Expert clinical review, as well as full testing to ensure the content and psychometric properties of the original scale are preserved, are additional requirements for any migration.
Four Steps for Implementation
Implementation of electronic ClinROs also shares some best practices with PROs. The consensus panel offers additional recommendations for each of the four steps involved in implementation: planning, startup, system development, and deployment to live production environment. Reinforcing a key takeaway: Copyright holders as well as clinical and COA experts should be involved throughout each step for reviews, approvals, and functional clinical testing.
The primary goal in the planning phase is to select a fit-for-purpose assessment and obtain the validated reference scale, along with scale administration and scoring conventions, from a credible source, which is typically the scale author and/or copyright holder. Changes to instructions, layouts, and scoring presentation should minimized to avoid impacts on the clinical interpretation.
Once the proper engagement is established, the next step is to determine the complexity. The panel created tiers of complexity and guidelines for sponsors and eCOA developers, labelled as ‘Level I’ and ‘Level II’ in the previous table. Classification into a one of the tiers guides the recommended level of evidence required for a faithful migration and potential steps such as additional psychometric testing. The complexity assessment should be done as early in the study planning process to identify risks to study timelines.
eCOA vendors may want to avoid making changes that place it into class two complexity, or as noted, determine that the scale is not suitable for migration to an electronic format. Planning should finish with a protocol review outlining how the scales will be used on electronic devices, how they should appear and behave in the eCOA system, the schedule of administration, order of assessments, and other study-specific configuration variables.
The startup phase should begin with a kickoff meeting with all stakeholders to align on objectives, timelines, roles and responsibilities. Other potential key elements in this phase would be to obtain any required training materials from the copyright holder and establish timelines for translations if needed. The panel did not focus on rater training, however, whether they’re administered via eCOA or paper, raters must undergo training and qualification on ClinROs prior to administering the scales in a study. eCOA systems should leverage their electronic platforms to ensure that access to individual eCOA scales should be restricted to raters that have met qualification requirements.
Once the planning and startup activities are complete, system development can begin using migration best practices outlined in the complexity classification level guidance. Once the scale is built in the eCOA platform, related activities such as user acceptance testing, definition of data specifications, and implementation of translated versions can begin. These activities are typically governed by vendor SOPs, guided by compliance requirements.
In the final phase of electronic implementation of a ClinRO – deployment to a live production environment – site staff and clinicians should receive training on the ClinRO assessment and the technology to minimize variability, and an eCOA system help desk should be available. The table below summarizes the five recommendations developed by the consensus panel.
How eCOA Reinforces Accuracy and Consistency
Each ClinRO will have its own set of unique and sometimes complex requirements. Following the framework established by this panel’s three plus years of effort enables sponsors to improve the precision of ClinRO data to support medical product registration through faithful migration and careful implementation. The evidence shows that properly implemented eClinROs have a positive impact on the quality and reliability of data collected, justifying addressing the challenges that ClinROs present.
While much of consensus panel’s work focuses on the clinical and scientific aspects of maintaining measurement equivalence, eCOA providers play an important role by providing in-house scientific expertise, experience with the scales, and platforms and technical teams that are capable of meeting the requirements. Signant Health has been a leader in the field of ClinRO eCOA since 2005, has utilized many of these best practices in successful pivotal programs worldwide, and has published data to demonstrate the impact of our approach to ClinRO eCOA across multiple therapeutic areas.
You can learn more about Signant SmartSignals eCOA here.