FDA Matters Blog
Register to get regular updates from FDA Matters FDA Matters Home

Lies, Damned Lies and Statistics


FDA’s regulations, policies and actions are multi-faceted and complicated. Oftentimes, it is hard to interpret what the agency is doing and why. We all depend on good analysis to understand where the agency has been and where it is headed. Unfortunately, some of what we read about FDA is poorly reasoned or distorted by the media and others.  


Three recent analyses have particularly troubled FDA Matters. They claimed to draw broad and important conclusions about FDA behavior and were uncritically circulated through mainstream and trade press. Yet, the analyses they offer are unremarkable or misleading.



My first example is a recent analysis from a healthcare research firm analyzing the number of FDA refuse-to-file (RTF) letters over the last dozen years. These involve situations where companies file drug and biological applications for approval and the FDA returns them to the company rather than accepting them for evaluation.


There is a methodological problem: FDA does not disclose RTF’s and traditionally companies have not disclosed them. Thus, any analysis of trend data (“more now, fewer a decade ago) is speculative.



Likewise, not knowing which companies received RTF’s means there is no basis to conclude that RTF’s were previously associated with small, inexperienced firms, but now are being received by larger companies. The shift, we are told, might reflect the agency’s enforcement mentality under the new commissioner. And maybe standards have been raised. A commentator (not the author) even suggests that the alleged uptick may be FDA maneuvering to improve its success rate under the user fee program (while, presumably, returning meritorious applications?).



To its credit, the analysis does mention that “the wave of RTF’s” may be related to FDA’s 21st Century Review Initiative. One aspect of that initiative is for FDA to weed out applications that are likely to be rejected later in the process. This is more efficient for companies, as well as FDA.



Front-loaded reviews are going to create more RTF’s. This seems obvious, if not unassailable. But trying to embellish this with time/trend data and allusions to changing standards raises issues that have no bearing on the question of whether RTF’s are becoming more important in the review process…and whether this is a good trend.



However, the implication picked up by the media was that increased industry interest in orphan drugs was not being met by increased commitment by FDA to get these drugs approved. But is that really the case? I don’t know and, despite appearances, the numbers don’t answer the question.



Showing same-year data for applications, designations and approvals implies that they relate to each. However, any given year’s orphan drug approvals reflect designations that were made 2 to 6 years previously. The meaning of the surge in the number of designations in 2010 cannot be assessed until we see if there are more orphan drugs approved in a few years.



Lastly, there is this week’s headline that: Biopharmaceutical Product Approvals in the U.S. Rose Dramatically in the 2000’s.” As a result of a Tufts University study, we are told that “during the 2000-09 period, 65 biopharmaceutical products received U.S. marketing, approval, up from 39 in the 1990s and 13 in the 1980.” Media seemed to treat this as a revelation. 


However, there was no biotechnology industry to speak of in 1980 and no products. As chronicled in previous columns (link below), the growth of this new industry has occurred over several decades. Is anyone surprised there were more approvals in the 2000’s?  


These three examples are a reminder that those of who write about and critique the FDA have an obligation to be accurate and not misleading. All of us, including myself, will fail sometimes. The media that report on our analyses rarely check to see if our conclusions are valid or make sense. 






The first analysis is at: http://portal.leerink.com/IRPDocumentViewer/Web/DocumentViewerCache.aspx?docId=4E2F752B612F6B5646706F3D&pad=52384B6E6E74573478656F45317951416D4B6A506E673D3D&userId=52636261346B39577A34343D


The second analysis is at: http://www.fdalawblog.net/fda_law_blog_hyman_phelps/2011/01/orphan-drug-designations-and-applications-took-off-in-2010-while-orphan-drug-approvals-tapered-off.html


The third analysis is at: http://csdd.tufts.edu/files/uploads/may-june_2011_ir_report_summary.pdf



Forget the Hype: Change Takes Time     March 21st, 2011

FDA Matters is always impressed by how much FDA does. The everyday tasks are overwhelming: reviewing, approving, monitoring and inspecting the products and facilities responsible for 80% of our food supply and 100% of drugs, biologics, medical devices, vaccines, and animal drugs. Then there are the policy issues, big and small, that must be tended to.

These are largely functional tasks—someone has a job (or several) and does them. Yet, FDA has another life, as the bridge to the future of foods, drugs and devices. This responsibility is vitally important to our nation. It also takes time to bear fruit. Read the rest of this entry

Leave a Reply

You must be logged in to post a comment.

© 2009-2012 by HPS Group. All rights reserved. Permission is hereby granted to those wishing to quote or reprint from this site, providing it is properly attributed to FDA Matters: The Grossman FDA Report™.