Selecting a Fit-For-Purpose Device

In today’s protocols, we continue to see an increased appetite to study clinical endpoints derived from wearables and sensors (i.e., digital health technologies – DHTs) in addition to our traditional study endpoints. Using DHTs, we have the opportunity to measure constructs that have previously been difficult or impossible to measure before (such as free-living daily activity), or to measure constructs more frequently (at home) for richer insights.

While there is little regulatory guidance specifically on the topic of selection and implementation of DHTs, a number of industry consensus groups have published recommendations and guidelines that are useful. Examples include the Critical Path Institute’s ePRO Consortium [1], the DIA’s Study Endpoints Community [2], the Clinical Trials Transformation Initiative (CTTI) [3], and the Digital Medicine Society (DiMe) [4]. Signant Health’s scientists have played a major role in the first two of these consensus groups.

In this blog, we describe our thinking on selection of a fit-for-purpose device when used to derive clinical endpoints in clinical trials.

Before we examine DHT selection, we must assume that using a DHT has been identified as a suitable strategy to measure one or more of the study’s concepts of interest. That given, there are several considerations in selecting a device that is fit-for-purpose.

Analytical validity 

Do we have data to support the accuracy and precision of measurement of the device when used in patients representative of the target patient population? DiMe refer to this as the second “V” of their V3 process, evaluating whether the DHT and its associated data processing produces outcomes data from the original sensor signals of an appropriate level of accuracy [5].  For example, does a wearable step counter provide an accurate estimate of the number of steps taken? In some cases, the evidence for this may come from data collected in studies of healthy volunteers or other patient groups. In other cases, we may need data from the specific patient population to be confident of this. For example, measuring gait parameters in a Parkinson’s population will require specific analytical validity data due to the difference in gait patterns specific to that patient population. 

Selecting a device with demonstrable analytical validity is important when it comes to defending the endpoint data it provides in regulatory review. Moreover, selecting a device with superior measurement properties, where possible, is recommended. For example, while all blood glucose meters meet the required ISO 15197-2013 accuracy standards for market approval, the accuracy and precision different models achieve varies widely. Models with superior measurement properties should be selected for use in clinical trials, such as those listed by the Diabetes Technology Society Blood Glucose Monitoring System Surveillance Program [6].

While DiMe and others recommend data to support device “verification” (showing the accuracy of the device’s raw, unprocessed data – for example, showing that an accelerometer can accurately measure defined accelerations), these data are not always easy to obtain from device manufacturers. In such cases, analytical validation suffices.

Clinical Validity

Clinical validity establishes that the endpoint measure derived from the processed sensor data adequately represents the concept of interest, and that the concept of interest does truly represent a meaningful aspect of health for measurement. The latter, content validity, is less related to the choice of a specific DHT within a defined class of devices (e.g., accelerometers), but still a vital consideration that should be documented as part of the conceptual framework to support the endpoint. Also important to demonstrate clinical validity are other endpoint properties such as reliability and sensitivity, which are specific to the DHT selected.

Feasibility and Usability 

Usability assessment should include determination of whether the mobile sensor technology can be used effectively by the target patient population. This may include consideration of form factor and wear location / usage requirements. Feasibility research, while not always needed, evaluates whether the DHT used in the context of the study protocol will be acceptable to sites and patients.  For example, this research can help determine whether requiring that patients use a particular DHT poses too great a burden alongside all the other aspects of the trial, or whether the required measurement period is likely to represent a challenge for patient adherence. 

Technical Characteristics 

From the perspective of a technology vendor, important characteristics include how the data are made available – can I integrate with my ePRO app using Bluetooth, or can I pull data in a timely manner from the DHT solution provider’s cloud database? Are the data available in a format I can ingest easily? Can I ensure the same firmware version is maintained throughout the study so as not to affect the measurement properties of the device? 

Can patients be enrolled to use the DHT through a behind-the-scenes data integration to simplify site and patient workflows? Does the device have adequate data privacy and security properties, and does it comply with 21 CFR part 11?

Other technical characteristics, linked to feasibility, include (for example) whether the battery life and data storage of the device supports the measurement interval required by the study.

Geographic Availability

When selecting a DHT, depending on what it is measuring, we may aim to maintain the same device model in all studies within a development program. In some cases, devices are not interchangeable due to differences in their measurement properties [7]. In these cases, the device needs to be able to be used in all the territories required by the development program. While not always required, selecting a device with good geographic footprint in terms of market clearances makes it easier to supply devices to sites across the globe.

Track Record in Clinical Trials 

This is a reassuring property and nice to have. While not essential, this knowledge may swing one device in favor of another if all things are equal.

Affordability 

Devices, and their use (e.g., data connectivity fees) should be affordable within the context of use required by the study.

DHTs offer great potential to measure constructs more frequently, more accurately or conveniently, or to measure things we have been unable to measure before. These technologies enable greater oversight and monitoring of patients between site visits and within decentralized trial models. 

While no formal regulatory guidance exists specifically on the use of DHTs, we know enough about the evidence needed to support device selection and endpoint validation requirements to scale up the use of DHTs in clinical trials. 

References

[1] Byrom B, Watson C, Doll H, et al (2018) Selection of and Evidentiary Considerations for Wearable Devices and Their Measurements for Use in Regulatory Decision Making: Recommendations from the ePRO Consortium. Value Heal 21:631–639. https://doi.org/10.1016/j.jval.2017.09.012

[2] Walton MK, Cappelleri JC, Byrom B, et al (2020) Considerations for development of an evidence dossier to support the use of mobile sensor technology for clinical outcome assessments in clinical trials. Contemp Clin Trials 91:105962. https://doi.org/10.1016/j.cct.2020.105962

[3] Clinical Trials Transformation Initiative (CTTI) (2017) Recommendations on developing technology derived novel endpoints. https://www.ctti-clinicaltrials.org/projects/digital-health-technologies. 

[4] DiMe Society (2020) The Playbook: Digital Clinical Measures. https://playbook.dimesociety.org/

[5] Goldsack J, Coravos, A, Bakker JP et al (2020).  Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs). npj Digit. Med. 3, 55. https://doi.org/10.1038/s41746-020-0260-4

[6] Diabetes Technology Society. Blood Glucose Monitoring System Surveillance Program.  https://www.diabetestechnology.org/surveillance.shtml

[7] Byrom B (2021).  Utilizing DHTs for Clinical Trial Endpoints.  Applied Clinical Trials 30(5).  https://www.appliedclinicaltrialsonline.com/view/utilizing-dhts-for-clinical-trial-endpoints