The FDA vs. cost-benefit analysis

Critics on the left and right are pointing to the same problem

(David Ryder/Getty Images)

The issue of full FDA approval for COVID-19 vaccines is finally starting to get more media attention, and I’m fairly optimistic that we can get some motion on this in August rather than “by the end of September” as the FDA is currently saying now that it’s on the public's radar.

But I want to talk in a more general way about the pushback I’ve gotten from several people who say that the FDA can’t deviate from its settled procedure because to do so would undermine confidence in the FDA. I don’t believe that anyone participating in this debate really has empirically valid information available about the short-term impact on mass psychology of the FDA going one way or another. What I do know, because we also saw this with the Johnson & Johnson pause debate, is that whenever public health authorities do something that does not make sense in first-order cost-benefit terms, the go-to explanation is that they had to do it to uphold confidence in the agency.

And I would beg people to consider that separate from the question of short-term mass psychology, there is a question of the long-term functioning and legitimacy of institutions.

I think the best way, over the long run, for U.S. public health agencies to have the confidence of the American public is for them to make decisions that are defensible on the merits. Things that you say can have unpredictable impacts on short-term mass psychology. But a solid track record of saying true things and giving advice that makes sense helps bolster your reputation over time. Resorting to Wizard of Oz tactics can work, but it also creates an increasingly brittle system that’s subject to catastrophic failure.

But stepping back from the specific contours of the vaccine EUA, I think it’s becoming increasingly clear that the standards the FDA uses to evaluate these things are — in the delicate words chosen by Actual Medical Doctor-Turned-Journalist Elisabeth Rosenthal — “outdated” and in need of a rethink. Basically, if we are in a situation where there are vaccines that every public health official is saying everyone should take, but the public health regulatory process doesn’t give them a thumbs up, that goes to the show that the process is not fit to purpose.

The two critiques of the FDA

The way the FDA thinks about this is that there are certain scientific criteria a drug has to meet in order to qualify as approved, and then there is a factual question as to whether or not a drug has in fact met that standard. The FDA then takes two forms of criticism.

One, which is popular in the lefty nonprofit space and the public interest community, is that the FDA is too captured by industry and too eager to give drugs the thumbs up. These critics tend to point to things like the approval of Aducanumab, excessive enthusiasm for prescription opioids, and a tendency to go soft on copycat drugs that provide on-patent alternatives to cheaper medicines without necessarily offering commensurate benefits. The other critique, which is popular among libertarians, is that the FDA is too slow and too hesitant to approve drugs.

If you dive into the guts of these criticisms, the critics are in many ways actually saying the same thing, which is that the FDA is not applying a cost-benefit lens to these decisions. And in their defense, as best as I can tell, that’s not their statutory mandate. So in the sense that the FDA is a bunch of human beings with jobs to do, I am not sure they are actually doing anything wrong at all. And I want to be clear about that — I’m not here to insult people or impugn their work.

But one lens you can use to examine this process is a functionalist one. In other words, what happens when the FDA puts different drugs into different regulatory categories.

• There is a very strong presumption that Medicare will cover FDA-approved medications for appropriate patients.

• There is a less strong but still strong presumption that private insurers will do this too.

• An EUA-authorized medication, by contrast, can be used, but the presumptions about who will pay for what are much less clear.

• There is a very strong presumption among public sector entities that you cannot mandate a vaccine based on an EUA.

So one question the FDA could ask about a medication is “is this medication safe and effective?” That is a little vague, but they can build processes to formalize it and the processes have their own times and schedules. A more precise question an agency could ask is “should Medicare pay for this?” or “should the Army mandate this?” or “is it good for society to flood television with ads for this drug?”

The FDA does not ask those questions, but the FDA in effect does make the decision on those questions. And since they are important questions, I think a better process would be one where they do ask the questions.

Mandatory vaccines could be very good

I think obviously the politics of trying to impose vaccine mandates in red states are dead. And frankly, I am pretty skeptical of even trying to put them in place in “purple”-type states where it would turn into a huge divisive fight.

But large urban school systems located in blue states are operating in a context where there is a lot of anxiety among both teachers and parents about Covid risk. And there is also a lot of anxiety among both teachers and parents about learning loss. Far and away the best way to address those concerns is to mandate the vaccines for students aged 12 and older, teachers, and staff absent a valid medical excuse. Mandating vaccines for public safety officials and mass transit operators also seems like a no-brainer.

Action in this regard would have direct benefits, but also secondary benefits.

Every right-wing cop who grudgingly gets vaccinated and lives to tell the tale is someone who can tell his friends and family that it was all fine in the end. Every bus driver who gets the shot and sees that it cost $0 can convey that information forward into his community. And to the extent that the U.S. military, blue states, and blue cities are moving forward with vaccine mandates, that will serve as validation to private enterprises that want to do the same. It would be good for United’s marketing to be able to say to customers “all of our pilots and flight attendants are vaccinated,” and it would be good for America to get those people vaccinated.

I’m dwelling on the benefits because to me, that is the issue — we should give the vaccines the regulatory treatment that is most likely to be beneficial to society.

But trust me, it’s not that I “don’t understand” that this isn’t how the process works. I am saying the process is bad.

The FDA deals in binaries

To make the point that this focus on cost-benefit isn’t an ideological right-wing thing, I’m going to quote Arindrajit Dube, who’s an excellent left-wing economist, talking about his experience with Covid vaccines.

He initially got a Johnson & Johnson shot and then, looking at the data that came in over the months, chose to supplement it with an mRNA booster which is clearly supported by the evidence but not yet officially recommended.

So what happened here?

Well the way the vaccine approval process works is that the manufacturer needs to say in advance what it is that they are testing, and then run a clinical trial based on that protocol. There isn’t a special magic property of the Johnson & Johnson vaccine that makes it “work” with only one shot whereas the mRNA vaccines require two shots. The evidence is clear that the first mRNA shot provides protection, and our knowledge from general immunological principles is that a booster could improve Johnson & Johnson. The difference is that Johnson & Johnson designed a clinical trial based on a one-dose regime while Pfizer and Moderna designed different regimes. So you run your trial, then the FDA looks at the data and makes a ruling. The data showed that one J&J shot provided a good amount of protection against SARS-Cov-2, so they approved it. And the data also showed that two mRNA shots provided excellent protection against SARS-Cov-2, so they approved those.

Officials in Canada and the UK are, I think, more accustomed to thinking of healthcare as a resource-optimization problem because they have single-payer systems. So they looked at the data and saw that a single mRNA shot clearly does provide a meaningful degree of protection and opted for a “first shots first” strategy, prioritizing getting as many people as possible a single shot before looping back to get boosters. Canada does not have a vaccine manufacturing industry, and the United States initially banned vaccine exports, so we started with a large vaccination lead. But then you can see how first shots first worked by looking at how they caught up to and then surpassed us.

This, to me, is the kind of flexible decision-making that we need from our public health authorities. It is absolutely reasonable to ask drug companies to preregister their dosing designs and to have some skepticism of proposals to deviate from that. But a public health agency does not have to act like it’s populated by robots who only know how to follow the established procedure. A holistic look at the data supported a deviation, so Canada deviated.

Similarly, early in the vaccine rollout, doses were very scarce so it was entirely correct to emphasize the idea of “take whatever shot you can get,” even though the J&J single-dose was not quite as good. Today, doses are no longer scarce, and all vaccines seem somewhat less effective against Delta, but the mRNA shots have held up best. Dube is not a medical doctor, but you don’t need to be a medical doctor to look at the statistical evidence and see that the risks of J&J people getting an mRNA booster are essentially zero while the benefits are fairly large. But again, you would need public health institutions that focus on this kind of analysis rather than focusing on questions that are within the four walls of the clinical trial.

Safe, effective medicine can also be bad

Superficially, the left’s critique that the FDA approves too many drugs is the opposite of the right’s critique that it approves too few. But we are really talking in both cases about a failure of cost-benefit analysis.

Suppose I created an All-Purpose Miracle Pill that, regardless of patient health status, extended life by somewhere between four and eight months in all cases. Unfortunately, the Miracle Pill costs $10 million and also often causes headaches and nausea. The questions of “should this pill be illegal?” and “should Medicare pay $10 million for this pill?” are really totally distinct.

But because we created the FDA, then created Medicare, then created the Medicare prescription drug purchasing program with a stipulation that Medicare won’t “bargain” with drug companies, we have a situation where the FDA is both a consumer safety regulator and also a de facto price regulator. So a common sense category of “look, we’re not telling you it’s so dangerous that it should be illegal, but we don’t really think we should pay for it” doesn’t exist.

This is, I think, Rosenthal’s basic point about Pfizer’s push to get the FDA to approve booster shots. It’s not going to be hard to show that boosters are safe. So are they effective? In the context of a global pandemic where we’re wrestling with anti-vax sentiment domestically and supply shortages globally, what kind of bar are we looking for? Say a Zeta Variant comes around against which the standard two-shot regime gives you an 85% reduction in the odds of hospitalization. But with a booster, it’s an 87% reduction. Well, that’s not not effective.

She writes that the “eh, it’s better than nothing” standard leads to approvals of drugs with little value and, critically, doesn’t require pharmaceutical companies to prove that their products are better than other products on the market:

The United States’ continued use of this standard to let drugs into the market has led to the approval of expensive, not necessarily very effective drugs. In 2014, for example, the FDA approved a toenail fungus drug that can cost up to $1,500 a month and that studies showed cured fewer than 10% of patients after a year of treatment. That’s more effective than doing nothing but less effective and more costly than a number of other treatments for this bothersome malady.

It has also led to a plethora of high-priced drugs to treat diseases like cancers, multiple sclerosis and Type 2 diabetes that are all more effective than a placebo but have often not been tested very much against one another to determine which are most effective.

Now I would say it’s good to have multiple drugs on the market for a given condition. Because of side effects or because of poorly understood individual factors, a drug that is better on average might still be worse for you. But I think it’s hard to dispute Rosenthal’s basic point that in many of these cases, it would be nice for the regulator to be able to order more detailed trials to generate useful information — information is a public good, and the cost of delay in a case like this may be low.

Lobbying as loophole

Now obviously, no agency is actually fully rigid in practice.

But the issue at the FDA is that it seems to mostly be flexible in the direction of the volume of lobbying. Not only do companies spend a lot directly on trying to work the refs, but they finance “patient advocacy groups” to push for approvals, and that’s how you get a $50,000 Alzheimer’s drug that doesn’t work.

And to an extent, this can be useful. Pharmaceutical companies are often made to wear the black hat in politics for bad reasons. People have screwy intuitions about it being immoral to sell life-saving medicine for profit which ignores the fact that it’s actually really good for smart people to work on trying to develop life-saving medicine rather than build ever-bad ad-targeting algorithms. So I don’t think it’s per se bad that pharma pushes to get more drugs approved more quickly.

The trouble is that when a policy community gets used to the idea of lobbying as a safety valve against overregulation and rigidity as a safety valve against lobbyist domination, you can get odd orphaned areas of policy.

In the short term, Pfizer is selling literally every dose of vaccine it can produce. So Pfizer doesn’t care about how quickly it moves off of EUA. To them, it’s completely irrelevant. Global demand for the vaccine is outstripping their productive capacity, they are eschewing price increases because that would make people angry, and so while full approval could be a big deal for American public health, it’s nothing to Pfizer. By contrast, getting the FDA to agree that everyone should get periodic booster shots is worth a lot to Pfizer over the long term. So they are hard at work on working the regulatory process with an aim of boosting sales in 2023, and are beyond and sort of checked out from the question of whether or not all eligible personnel are vaccinated in time for the school year to start.

But this just goes to show that trying to strike a balance between corporate hunger for profits and regulatory inflexibility doesn’t mean you’re finding a happy medium. There are tons of American institutions waiting for the FDA to shift to full authorization; they just don’t happen to be pharmaceutical companies accustomed to lobbying the FDA.

Build confidence by doing a good job

The various iterations of the discussion around “confidence” and various FDA decisions (the AstraZeneca vaccine, the Johnson & Johnson pause, first doses first, now the lack of full approval) always remind me of debates we used to have about monetary policy in 2010-13.

If you looked at the facts, the economy was suffering from inadequate demand, so I would say the Fed should do more to boost adequate demand. Others would note that the Fed was already taking tons of criticism from the right for having done too much, and members of Congress were getting mad. They would say that to do even more would be to risk blowing public confidence in the central bank. My response was that at the end of the day nobody cares about the details of monetary policy — they were mad because the economy was so shitty. If the Fed made the economy better, people would be happy, and their political legitimacy would be saved.

I can’t prove that I’m right of course. But I would note that the Fed’s emergency actions in March 2020 were very controversial in March 2020, but now it seems like everyone is happy with Jay Powell’s conduct of monetary policy because Jay Powell did a good job.

Lots of the public sector in the United States is conducted by institutions that are not particularly technocratic in their design or their operation. But the Fed and the FDA are. These are institutions that employ a lot of technical specialists and that have a tradition of independence from the White House. In the long run, I think the way technocratic institutions safeguard their independence and build their legitimacy is by doing a good job. You want people to think “these guys seem like they know what they are doing.”

And to me, the pattern of ignoring first-order cost-benefit considerations and then appealing to public confidence as the explanation of why that’s okay does the opposite. It suggests that the large-ish minority of people who are quantitatively competent need to all be doing their own analyses of issues like supplementing an initial J&J dose with a booster, and that basically every policy decision needs to be litigated and re-litigated in the public square. That’s not going to foster long-term confidence. If you want that, you need to make decisions that are justifiable on the merits.