Through statute and directive, FDA has been asked to collect, analyze, interpret and utilize massive amounts of data. This includes biological, clinical, adverse event, production and distribution data, medical and food product tracking, and the Sentinel system for early discovery of potential drug safety problems. The systems are not in place to do any of this, at least not at the required level of sophistication. Even if they were, sifting valuable information from background noise is extraordinarily hard. As a result, FDA needs to manage Congressional and public expectations as to “what is possible and when.”
Above and beyond its day-to-day information technology needs, FDA will require substantial monies to develop these new databases. Further, the agency lacks the data-mining experts needed to make sense of the data that is received. Such individuals are few and in high demand. Even were the agency given the funds to hire 100 of them this year, it would probably not come close to reaching this goal. Even when the required expertise is on-board, it is a difficult, iterative process to turn data into usable knowledge.
FDA must take account of the littered field of others’ past failures and limited successes. Even programs that work well may have limited applicability to FDA’s daunting task. The VA system operates with electronic medical records, a finite number of known participants and a partially closed system of care. The Drug Abuse Warning Network (DAWN) tracks only drug-related hospital emergency department visits and drug-related deaths reported by coroners and medical examiners. Each has key characteristics that are not present in tracking food and product use–and consequences–in a real world setting.
FDA has been told that the Medicare claims data set (provided by CMS) gives them a fast start along this difficult path. Perhaps not. I often use the following equation to explain:
Real world data sets = uncontrolled variables + inconsistent data collection + questionable data accuracy.
FDA is used to the type of knowledge derived from controlled clinical trials. In contrast, claims data is notoriously unreliable with few tools to identify and correct for systemic bias.
With very limited staff, FDA has done yeoman work on the initial phases of developing its Sentinel initiative. The goal is a nation-wide electronic safety monitoring system for post-market surveillance. It will use existing data from health systems, hospitals and insurers. An operational system will be a challenge. Creating one that minimizes false positives is more than just a challenge.
There are also some larger issues that will need to be addressed. Even with new hires throughout the agency, medical knowledge and food surveillance needs are growing faster than FDA expertise. Not every “cluster” leads to a medically or scientifically valuable insight. The consequences of premature responses and over-reactions are substantial and often severe. Nonetheless, the tendency in our society, even sometimes at FDA, is to forget that the association of data points often tells us nothing about causality. Finally, it is hard to know the implications of studies that show that more information often does not produce better decisions. This runs counter to FDA culture and is counter-intuitive to the way most of us think.
In sum, FDA has a large job in the area of information technology, data collection and data interpretation. In addition to the need for monies and expertise, there are inherent, as well as institutional barriers to success. Managing expectations will be a test for the new FDA leadership and the senior staff.