Among clinical studies used to support the approval of high-risk medical device modifications by the U.S. Food and Drug Administration (FDA), fewer than half were randomized, blinded or controlled, and most primary outcomes were based on surrogate end points, according to a study published in the Journal of the American Medical Association (JAMA). These findings suggest that the quality of studies and data evaluated to support approval by the FDA of modifications of high-risk devices should be improved.

Many high-risk medical devices undergo modifications approved by the FDA through pre-market approval (PMA) supplements. However, there have been multiple high-profile recalls of devices approved as PMA supplements. Study authors Sarah Y. Zheng, MD, et al., examined the use of randomization, blinding, controls, post hoc analyses, and the reporting of age and sex and other metrics used in FDA-approved panel-track supplements, a type of PMA supplement pathway used for significant changes in a device or use that always requires clinical data, between 2006 and 2015.

The approval of 78 panel-track supplements were supported by 83 clinical studies, with 71 supplements supported by a single study. Of the 83 studies supported, 45 percent were randomized clinical trials and 30 percent were blinded. The median number of patients per study was 185 and the median follow-up duration was 180 days. Of the primary end points, 81 percent were surrogate end points and 38 percent were compared with controls. Age was not reported in 40 percent of the studies, and 30 percent did not report sex for all enrolled patients.

Zheng, et al., write, “studies without randomization are prone to various types of bias, making it difficult to ascertain whether these modified devices are safer or more effective than previous iterations, conventional treatments or no procedure,” and that “for surrogate end points to be useful to patients and clinicians, they must be shown to predict meaningful clinical outcomes, which rarely happens. Therefore, use of surrogate measures can lead to uncertainty about clinical outcomes.”

“These findings suggest that the quality of studies and data evaluated to support approval by the FDA of modifications of high-risk devices should be improved,” write Zheng, et al.

In a related study also published in JAMA, authors Huseyin Naci, PhD, MHS, et al., found that among 22 drugs with 24 indications granted accelerated approval by the FDA between 2009 and 2013, efficacy was often confirmed in post-approval trials a minimum of three years after approval, despite similar design elements in confirmatory and pre-approval trials, including reliance on surrogate measures as outcomes.

The median number of participants enrolled in the pre-approval studies was 132, with eight studies including fewer than 100 participants. At a minimum three years of follow-up, 50 percent of required confirmatory studies were completed. Among these studies, 66 percent examined clinical efficacy, 18 percent evaluated longer follow-up and 16 percent focused on safety. Completed post-approval requirements demonstrated efficacy in 42 percent of indications based on trials that evaluated surrogate measures. Among the 58 percent of remaining indications that had not yet completed all requirements, at least one of the confirmatory studies failed to demonstrate clinical benefit, and at least one of the studies was terminated.

Naci, et al., write, “Even though the majority of completed studies showed positive results in the post-marketing period, all completed confirmatory studies demonstrating drug benefit evaluated surrogate measures of disease activity rather than clinical outcomes. Drugs granted accelerated approval receive market authorization on the basis of fewer studies, smaller patient populations, shorter follow-up, and less-established surrogate measures than drugs approved via the traditional pathway.”

In an editorial comment addressing both studies, former FDA commissioner Robert M. Califf, MD, MACC, writes, “When medical products are found to be ineffective or have serious adverse effects after reaching the market, concerns are raised about dangers to the public health – people are harmed when risks are not identified prior to marketing. On the other hand, when effective therapies are developed, the interval between the time the product could have conceivably been available and when it actually became available raises concerns about depriving people of access to life-improving or lifesaving therapies – people are harmed when they lack access to effective products. These issues, subject as they are to the probabilistic uncertainty that makes medical product development so risky, are often much clearer in hindsight.”

“A sweeping overhaul of the overall system is needed. Fortunately, such a transformation is under way. Regardless of the type of medical product, substantial progress in balancing safety with access to effective therapies will come from systemic changes in the ecosystem rather than incremental modifications made by imposing more severe demands on individual products,” writes Califf.

Keywords: Drug Approval, Uncertainty, Follow-Up Studies, United States Food and Drug Administration, Pharmaceutical Preparations, Public Health, Drug Evaluation, Bias

< Back to Listings