There is an emerging crisis in the development of drugs, biologics and complex new medical devices. Clinical trials take too long, cost too much and often produce imperfect knowledge. Many promising medical products are not developed because of the difficulty and expense of proving safety and efficacy—a loss that is costly to society.
FDA Matters believes that the key lies in developing new approaches to generating rigorous data and analysis. Ultimately, this will require the re-invention of the clinical trial.
Clinical trials produce the knowledge that makes FDA approvals possible. Without them, we would all become test subjects in a dangerous game of medical trial and error. FDA (and patients) want a reasonable level of certainty about safety, efficacy and risk-benefit before medical products are marketed. Except in extraordinary cases, FDA should never be put into a position to accept less.
The clinical trial is, and must remain, the gold standard. To understand why, it is useful to look at another type of medical knowledge that is increasingly in vogue: analysis of real-world data. The Medicare Claims database would be an example. Another would be patient data compiled by large health plans. Analysis of real-world data sets is becoming a cornerstone of reimbursement policy and plays a significant role in comparative effectiveness determinations.
The supposed advantage is the ability to look at hundreds of thousands of patients and discern patterns that might not be seen in clinical trials. However, the association of data points tells us nothing about causality. It only signals where additional analysis is needed. Real-world datasets also lack rigor:
Real-world data sets → post-hoc analysis using uncontrolled variables + inconsistent definitions + incomplete data collection + questionable data accuracy
By comparison, clinical trials produce a wealth of reliable knowledge (albeit far from infallible). This can be expressed as:
Clinical trial data sets → prospectively-defined analysis using controlled variables + randomization of patients + double-blind protocol + placebo controlled + pre-defined standard for data collection and data integrity
“Prospectively planned” means a drug or device sponsor must declare in advance the precise findings that will determine whether the treatment caused a beneficial outcome. Sponsors are limited in their ability to go back afterward to “data dredge” for positive correlations that might be spurious. To some extent, all analysis of real-world data sets is data dredging.
“Controlled variables” means that the outcomes of patients in the clinical trial can be compared with some degree of reliability. In real-world data sets, you can never be sure.
“Randomization” and “double blind” work together to assure there is no bias in patient selection (e.g. putting healthier patients in one arm of the trial) and that neither patients nor medical staff knows who is getting the study drug.
“Placebo controlled” allows a reliable determination of the impact of treatment. Since some patients will improve regardless of whether they are getting treatment or placebo, treatment effectiveness is the differential between those who improve in one study arm over the ones who improve in the other.
“Pre-defined protocols for data collection and data integrity” assures that definitions stay constant and results from different trial sites and different investigators can be combined. In real-world data sets, no one has yet figured out why medicine is practiced differently in Boston compared to Hartford.
Taken together, these features of the clinical trial serve to produce reliable data that support a conclusion (or not) that the treatment caused the benefit. The challenge is to improve upon this gold standard while maintaining confidence in the results.
Future columns will explore how this might be done. Meantime, readers are encouraged to post their thoughts or send me their ideas.
Steven
Here are two earlier columns that partially address this topic:
Long-term Challenges Need Short-term Attention
December 13th, 2009
We are less than 7 months into the new Commissioner’s tenure. Three or four years from now, she will be judged by whether she moved the agency forward in these areas. I think she has gotten off to a very good start, but there is immense amount of work still required. Read the rest of this entry »
Turning Data into Knowledge
June 2nd, 2009
Through statute and directive, FDA has been asked to collect, analyze, interpret and utilize massive amounts of data. This includes biological, clinical, adverse event, production and distribution data, medical and food product tracking, and the Sentinel system for early discovery of potential drug safety problems. The systems are not in place to do any of this, at least not at the required level of sophistication. Even if they were, sifting valuable information from background noise is extraordinarily hard. Read the rest of this entry »
I look forward to your ideas on reinventing clinical trials. However, your lengthy lists of “disadavantages” associated with trying to learn from real world datasets, coupled with your list of “advantages” associated with randomized controlled trials, pre-specified endpoints, placebo controls and all the other trappings of our current, statistics-driven system, seems to lead straight back to the status quo: clinical trials that take too long, cost too much and often (I would argue always) produces imperfect and frequently misleading knowledge.
Have you considered trying to identify all the disadvantages of RCTs (there are many, and they are inherently insurmountable from a “real world” standpoint) before embarking on a process of reinventing clinical trials? It is because we have chosen in a very narrow-minded and rigid way to anoint RCTs as the “gold standard” that must be applied to drug development and approval in almost every case, that clinical trials take too long, cost too much and produce imperfect data.
Don’t we have to get out of the RCT cattle chute to reinvent our clinical trials system? It is now scientifically clear that the relative frequentist statistical approach that serves as the basis for RCTs does not fit our current and rapidly expanding knowledge of the biology of most of the diseases that are still killing us. The RCT approach isn’t working because these diseases are not population-based, which is the only kind of disease where RCTs are truly useful, and even in population-based diseases, RCTs take too long, cost too much and produce imperfect data.
Isn’t it time to set our stats programs aside for a while and consider truly new, more scientific approaches than the simplistic narrow comparisons possible with RCTs.
A good root cause analysis puts everything on the table, and until we acknowledge the severe limitations of RCTs and start questioning the “gold standard,” we are going to end up back where we are now, with a system that takes too long, costs too much, produces imperfect data, is unattractive and largely inaccessible to patients, and is fraught with a host of serious ethical problems. In short, RCTs aren’t working very well at all, and tweaking them isn’t going to make them work better.
When science is advancing, there are no “gold standards,” and methods change and advance quickly. We have been rigidly tied to the RCT for almost 50 years, but change may be coming.
Quoting Peggy Hamburg (FDA Commissioner), who quoted Albert Einstein in a recent speech:
“At the dawn of the atomic age, Albert Einstein said, “everything has changed except for our way of thinking.” In these, the early days of the genomic age, we are trying to adapt our thinking, our regulatory system, our models of drug development, research, clinical trials and the very way we look at, gather and analyze data to a new reality.”
She went on to say:
“While the evidence-based method gave us great confidence that the clinical trial data we were evaluating gave us a reliable picture of the benefits and potential risks of a new medical product, we have always known that these methods are not infallible; that particular risks may emerge in the post-market population that we frankly did not see and, perhaps, could not detect in the evaluation of a new drug.”
“While we might have a statistically accurate picture, it lacked the nuance we really need to account for human variability. And a statistical average fails to recognize the fundamental truth that patients aren’t really homogeneous populations or sub-populations at all, but individuals. In terms of answering the critical questions we must ask in any trial for a new molecular entity, the randomized controlled trial could give us the “what” –a quantifiable measure of the effectiveness of a particular drug therapy, but it could not tell us the “why”—why do these patients respond to this treatment and others don’t, and it could not tell us the “who” –who are the responders, and what is the underlying biology that enables them to have this response.”
“In fact, in some cases, looking at the average response rate for a particular group of patients may provide a false picture, if instead of one group, the data actually reflects the response rates of different sub-groups as identified by genetic variant. For example, a 60 percent response rate may, in fact, be the combination of one patient group with an 80 percent response rate and another patient group with only 40 percent. ”
The Commissioner of the FDA is openly questioning the “gold standard,” she is right to do so, and it is one of the most scientific and hopeful developments in the debate about how to best make progress against disease in years.
Again, I look forward to your ideas.
Steve Walker, Co-Founder, Abigail Alliance