Pharmaceutical companies are actively seeking deeper insights related to the effectiveness of prescription therapies by gathering and analyzing real-world data (RWD).
When RWD from available sources such as electronic health records (EHRs) is subjected to appropriate analytical methodologies, real-world evidence (RWE) is generated, showing how a treatment works in a real-world setting. When the process is done right, the resulting RWE is sufficiently accurate to allow clinical assertions to be made. Such clinically relevant findings can inform both regulatory decision-making (with regard to product approvals and label expansions, comparative effectiveness, and more) and payer decision-making (with regard to drug pricing, reimbursement, and formulary status).
However, the process of developing RWE to make clinical assertions is still relatively new. Much work is required to assess available sources of RWD and select the most appropriate analytical techniques to develop regulatory-grade RWE, as well as develop standard protocols to guide such efforts. A recent study published in the Journal of the American Medical Informatics Association (JAMIA) aimed to accomplish just that—to determine whether traditional RWE techniques used in cardiovascular medicine are able to allow regulatory-grade clinical assertions to be made.
Regulatory-grade RWE and Why it Matters
The study defined regulatory-grade RWE, or advanced RWE, as evidence making clinical assertions that are sufficiently accurate and reliable to support decision-making by regulators and payers. Such RWE can help pharmaceutical stakeholders expand the findings of the underlying randomized clinical trials (RCT). Specifically, regulatory-grade RWE allows for studies that help everyone—patients, physicians, payers, and pharmaceutical firms—to better understand how prescription drugs are used in specific groups of patients that may not have been included in the trials. RCT have strict inclusion and exclusion criteria that make it difficult to generalize the clinical findings to broader groups of patients who were not included in the trial. When regulatory-grade RWE can influence regulatory and reimbursement decision-makers, it has the potential to give both patients and physicians broader access to today’s safe and effective, often lifesaving therapy options.
The Study Design
The JAMIA retrospective observational study analyzed EHR records from an academic hospital. Different analytical techniques were used to create data-driven clinical assertions related to the use of a cardiovascular therapy option. The study used two types of available EHR data:
- Structured EHR data (EHR-S) refers to coded medical information related to medical issues, medications and lab data, which are typically used for billing or to provide an overview of medical care.
- Unstructured EHR data (EHR-U) refers to physician notes and other narrative text that is typically collected in a patient’s EHR during the provision of care.
The intent of the study was threefold:
- To assess the occurrence of a predefined set of clinical events related to cardiovascular medicine (such as coronary artery disease, myocardial infarction, LDL/HDL cholesterol, statin use, and many more) that are captured in the EHR data sources;
- To evaluate the accuracy of available analytical techniques when applied to clinical concepts in EHR-S and EHR-U data; and
- To compare the accuracy of findings that were developed using traditional query techniques (such as Standard Query Language) applied to EHR-S data, to the findings that were developed using AI-based approaches (such as natural-language processing and machine-learned inference) applied to EHR-U data.
The study also aimed to define a reasonable measure of sufficient accuracy in advance, and then to determine which approaches can meet those defined accuracy requirements.
The Study Findings
The study assessed whether the resulting RWE insights could be considered regulatory-grade based on concepts of precision and recall:
- Precision evaluates how many patients that had a specific clinical event, such as a heart attack, were correctly identified in a dataset.
- Recall evaluates how many patients that had a specific clinical event, such as a heart attack, were accidentally missed in a dataset.
The approach that applied traditional database-query techniques to EHR-S data typically achieved less than 50% recall. This did not meet the thresholds proposed in advance by the study authors to achieve regulatory-grade RWE, which was to achieve recall >85% and precision >95%.
In other words, if traditional query techniques were used on EHR-S data alone, the findings would potentially invalidate the resulting study. For example, a data-driven study that needed to exclude myocardial infarction would accidentally include more than 70% of patients who had a prior heart attack, and no one would know. This could result in findings leading to inappropriate changes in the standard of care and potential patient harm. These results suggest that recall is an important metric that should be routinely measured during EHR-based studies that are intended for regulatory use.
The study highlights that to derive a more reliable and clinically relevant RWE, an approach should involve 1) stating accuracy requirements within a RWE protocol in advance of running the study and 2) ensuring accuracy requirements are met. To achieve this, it is likely that unstructured data and advanced data enrichment approaches such as AI may be required.
Application in Real World
The validity of any RWE study is heavily influenced by the quality of the underlying data and the analytic techniques used to derive insights from it. To enable regulatory-grade RWE studies, measuring accuracy should now be required. Advanced approaches using advanced data sources and AI-based methodologies may be needed to achieve regulatory grade. If standard query-based approaches are used, the resulting data-driven assertions may be incomplete or inaccurate.
The landscape of evidence-based care is changing rapidly, and these changes are creating both exciting and challenging prospects for pharmaceutical companies. The ability to develop broad and deep experience in how to identify the most robust data sources, and how to use the most appropriate analytic techniques and validation methodologies will become a differentiator for leading pharmaceutical firms. The ability to develop and demonstrate such capabilities has direct implications for pharmaceutical firms looking to use RWE for label expansions and post-marketing commitments for regulators, and comparative effectiveness for market access.
