The case for vaccine challenge trials
We need a better way, even after Warp Speed
Early in the pandemic, I was very bullish on the idea of challenge trials to speed vaccine development. No government on earth went that route, and the conversation has largely been dropped since Pfizer and Moderna managed to get vaccines approved in record time. The vaccine from AstraZeneca has already been approved in the UK, and most observers believe it will be approved in the US around April or May. There is also a Johnson & Johnson vaccine that they are hoping to submit for Emergency Use Authorization within the next month or so.
The previous record for vaccine development was four years, so this is a big achievement.
And as far as I can tell, the main thinking in the medical ethics world is that this is all great. They didn’t like the idea of challenge trials, we didn’t do challenge trials, and now we’re making vaccines faster than ever, so there’s no problem.
I think this is seriously wrong. The new mRNA technology that powers the Pfizer and Moderna vaccines appears to be a very powerful tool that could do incredible good in the future. But to really take advantage of it, we need to test vaccines much more quickly. Delays in approval are more costly now that we know we can make vaccines faster. And, critically, the current way of doing Phase 3 trials depends on failing to contain novel pathogens and getting lots of people killed. Indeed, there’s a feedback loop where taking longer to approve vaccines undermines containment, and then the failures of containment lead to vaccine approval. We ought to institutionalize a better way of doing it.
Challenge trial vs Phase 3 trial
If you’ve seen Contagion, Jennifer Ehle’s character, Dr. Ally Hextall, develops an attenuated virus vaccine. She then injects herself with it and goes to visit her father who’s already sick with MEV-1. She does not become infected, and the vaccine appears to be approved on that basis.
There are a lot of serious scientific problems with that study design, starting with the n = 1 sample size and the lack of placebo control.
But the basic concept is sound. If you give a bunch of people a vaccine candidate and a bunch of other people a placebo and then expose them all to the virus, you will quickly get statistically powerful information about whether infection is rarer in the treatment group than in the control group.
That would be a challenge trial, and it’s not how it’s done.
Instead, in a Phase 3 clinical trial, you give a bunch of people a vaccine candidate and a bunch of other people a placebo and then tell them to go about their business as usual. Over time, you track both populations and if many more infections occur in the placebo group than in the vaccine group, that tells you the vaccine works.
Compared to the challenge trial, the non-challenge model has several downsides:
It’s slower, since instead of exposing everyone at once, you need to wait for exposures to trickle in over time.
It requires more subjects, since lots of people in both groups actually end up not being exposed at all, so you need a huge sample to get reliable information.
The experiment itself has somewhat undetermined scope. Since you can’t know when people will be exposed, you can’t know when you’ll have your data or necessarily even define the terms of success in advance.
But most of all, a non-challenge trial isn’t just slower. It specifically requires large numbers of people outside the experiment to get sick and die.
Is this ethics?
The problem with a challenge trial (allegedly) is ethics because it is (allegedly) problematic to deliberately expose people to a virus you don’t have a cure for.
But even in the non-challenge trial, members of the control group becoming sick are how you know the treatment is working on the treatment group. The interesting thing Hextall skips in the movie is the placebo check. Doing it her way is considered bad science. Once you accept the need to test the candidate against a placebo, there’s no way to avoid the fact that your trial isn’t done until a bunch of people vaccinated with a placebo get sick.
The difference is that in the non-challenge model, everyone is supposed to behave normally rather than deliberately get exposed. But for the trial to succeed, a bunch of people with the placebo need to get sick. Which in turn means a much larger number of people in the general population need to get sick, since the people in the experiment aren’t supposed to be doing anything different.
I confess to a lack of imagination here, but I sincerely cannot understand why the scenario that involves giving people a placebo and then waiting for them (and millions of others) to get sick is “more ethical” than the scenario where we give people a placebo and then get them sick quickly in order to spare others.
If anything, the ethical question here seems to me to be about placebos, not about challenge trials. Giving someone a fake vaccine for science feels a little dodgy, but it’s perhaps a necessary evil. Certainly with a non-challenge trial, it’s necessary because without sick people in the placebo group, you can’t tell whether the people with a lack of sickness in the vaccinated group is because of the vaccine or just good luck. In a challenge trial, there might be less need for placebos since you know the treatment group has been exposed. But I don’t know. I’m not here to question the medical science of placebo effects.
I am here to say that this vision of ethics doesn’t make sense. And in broader public health terms, it’s particularly bad because it involves planning to fail.
A sidebar on expertise
Whenever I write on public health issues in a way that takes issue with the decisions that have been made by the leaders of America’s public health institutions, a discourse launches on Twitter that pits loudmouth know-nothing pundits like me against “the experts.”
So before anyone goes that route, I hope they will peruse the open letter on challenge trials assembled by the advocacy group One Day Sooner over the summer. I was a philosophy major in college and it’s the ethics side of this that I feel most personally invested in, so I was struck to see Peter Singer and Christine Korsgaard — probably the world’s leading consequentialist ethicist and the world’s leading deontological ethicist — both sign the letter. They’ve also got Shelly Kagan, Tyler Burge, David Chalmers, Elizabeth Harman, Daniel Dennett, and many others.
There’s a bunch of economists, of course, which is what you would expect from this sort of thing. But also fourteen science Nobel Prize winners. And many, many people from the world of public health and bioethics headlined by Nir Eyal and Marc Lipsitch.
This is a big interdisciplinary issue and a huge interdisciplinary group of experts who supported doing things a different way. Whatever the issue is here, it’s not that the world’s regulators and pharmaceutical companies were “listening to the experts.”
The defeatism of Phase 3 trials
Like many people, I find it regrettable that non-pharmaceutical interventions (NPIs) did not go better in the United States.
As of June, we were getting the pandemic under control. At that time, we should have encouraged people to take advantage of the summer weather to shift all social activity outdoors. We should have done a bar and restaurant bailout and prohibited indoor dining. Nearly everything other than dining can be done approximately safely by wearing masks. That should have been mandatory, but we specifically should have been ramping up production of the highest-quality masks and encouraging their use rather than relying on cloth. Then we should have invested in large-scale surveillance testing (especially of people working outside the home) to identify asymptomatic spreaders combined with centralized quarantine under comfortable conditions.
The pandemic could have been wrestled into submission with better policy, and I think we’d have been living in a much better world if it had.
But note that if cases had been low and falling in October rather than high and rising, the Pfizer and Moderna clinical trials wouldn’t have wrapped up nearly so quickly. The superficial way that defenders of the clinical trial status quo square this with being critical of America’s failures on non-pharmaceutical interventions is to say companies organize multiple trials in different countries. But that just pushes the problem out. The “give a bunch of people a placebo and wait around for them to get sick” method of assessing vaccine efficacy requires the disease to rage out of control somewhere. That’s not good. When new pathogens arise, we should try to suppress them and we should deploy state of the art medical technology to defeat them. That means testing the technology without waiting for the NPIs to break down.
Planning for success
I think this is an important point because the cost-benefit of stringent NPIs varies according to the vaccine timeline.
If it were the case that a new vaccine would take 4-6 years to develop, then what you want to do is basically just flatten the curve. You adopt enough restrictions on activity to prevent the hospitals from becoming overwhelmed, but fundamentally the population achieves herd immunity by everyone getting sick. The virus then bounces along as a low-level endemic problem and eventually gets cleaned up with a vaccine.
The reason is that keeping tough NPIs in place for years is just much too costly. As we’ve seen, getting even moderate NPIs to stick is pretty hard. Even in a place like California where the weather is good all the time, lots of people seem to have wanted to insist on holding indoor maskless social gatherings. It’s tough. And trying to take the kind of steps that would have truly suppressed the virus and kept them in place until 2025 would have been too much to ask.
But consider the actual timeline.
As David Wallace-Wells reports, Moderna designed its vaccine candidate on January 13, 2020. The entire saga over the next 11 months was testing it, not doing the science of making it. It took until mid-July to get the early human testing done, and then the Phase 3 trial on 30,000 people started on July 27. With human challenges, we could have been ready to start vaccinating in September. And I think it’s important to understand that while all the virologists I’ve spoken to affirm the importance of actually doing the tests before you start injecting people, all of them expected the vaccines to be approved. By the time a vaccine gets to Phase 3, in other words, there is already a strong evidence base that a vaccine is going to be effective — they’re not just out there guessing.
Wallace-Wells himself isn’t even focused on the human challenge aspect of it; the thesis of his piece is that in the future we could speed the earlier phases of the trials by essentially pre-testing mRNA vaccines before pandemics start:
None of the scientists I spoke to for this story were at all surprised by either outcome — all said they expected the vaccines were safe and effective all along. Which has made a number of them wonder whether, in the future, at least, we might find a way to do things differently — without even thinking in terms of trade-offs. Rethinking our approach to vaccine development, they told me, could mean moving faster without moving any more recklessly. A layperson might look at the 2020 timelines and question whether, in the case of an onrushing pandemic, a lengthy Phase III trial — which tests for efficacy — is necessary. But the scientists I spoke to about the way this pandemic may reshape future vaccine development were more focused on how to accelerate or skip Phase I, which tests for safety. More precisely, they thought it would be possible to do all the research, development, preclinical testing, and Phase I trials for new viral pandemics before those new viruses had even emerged — to have those vaccines sitting on the shelf and ready to go when they did. They also thought it was possible to do this for nearly the entire universe of potential future viral pandemics — at least 90 percent of them, one of them told me, and likely more.
To me, though, all these ideas are additive.
If the basic science of developing a vaccine is going to take years, then a 16-week versus four-week Phase 3 trial doesn’t seem like that big of a deal. But if vaccines can be made really fast, and if they can go through preliminary testing in advance, and if we can use human challenges to rapidly test for efficacy, then we have a strong argument for really tough NPIs to beat the pandemic while we wait for the vaccine to be deployed.
It’s a little forgotten now, but back in March and April, compliance with NPIs was really high. And in retrospect, the restrictions governments put in place (closing playgrounds, etc) were stricter than they needed to be. The problem was that people had the impression that this was a short-term measure, and when it instead turned out to be essentially indefinite, things all started to fall apart. Even given that context, our policy choices were far from perfect, and the implementation also had a lot of flaws. But that’s life. To do better in the future we need a better plan with a faster route to vaccination.
I want a world where we tell people we’re really going to hunker down for a finite — and not that long — span of time while a team of heroic volunteers steps up to save the country.
But what about … ?
Every time I’ve brought this up over the past year, some wise-ass asks “well why don’t you do it, asshole?”
But of course I’ll do it! I signed up on the One Day Sooner site as a potential volunteer months and months ago. If you don’t want to do it, that’s fine. They got tens of thousands of volunteers with zero support from world governments. Recruiting more with a big national program would be easy.
More generally, asking people to do something risky for the greater good is just not that unusual. I can tell I’m getting old because when I saw a bunch of National Guard troops heading downtown the other day to help protect the Inauguration, I was struck that it’s a bunch of teenagers. This dude from MPDC was in hand-to-hand combat with a mob trying to overthrow the government.
I don’t want to say that our society needs more wars or more raging battles with insurrectionaries. But it’s good for more or less ordinary people to have opportunities to volunteer to do extraordinary things and win status and praise from their fellow citizens.
Last but by no means least, one key anti-challenge talking point is the idea that if you run the trial exclusively on young and healthy people, you don’t get data about more vulnerable populations. If you start from the premise that challenge trials are “unethical” and are only being brought up by annoying trolls and you want to avoid thinking about them, this seems like as good a reason as any. But if you think of challenge trials as offering incredibly large potential benefits, then it’s just a minor roadblock to address.
For starters, I don’t think it’s out of the question that elderly or more vulnerable people might give informed consent to participate in even a very risky challenge for the sake of human betterment. The medical ethics world seems to take a very dim view of the human capacity to deliberately self-sacrifice, but other segments of our society (including medical personnel treating Covid patients!) don’t operate on this basis.
We could also do rolling trials. Once the vaccine is proven to work in twenty-somethings, you could start dosing young people while running a new challenge with forty-somethings. Then if it works there, you further update your priors about the risks and do it with sixty-somethings.
In all cases, recall that inaction is very costly. I started writing this post on Saturday, and over 10,000 people have died since then. Something that has a 70 percent chance of ending a pandemic two months sooner saves a staggeringly large number of lives in expectation — to say nothing of all the non-mortality benefits.
Things could be worse
Thinking back to Dr. Hextall’s hasty challenge, the fictional MEV-1 had a basic reproduction rate of 4 and an Infection Fatality Rate of 25 percent.
Under those circumstances, you might opt for genuinely desperate measures.
Covid-19 has been the worst thing to happen to the world in two or three generations, and yet a virus could be much worse. Already we seem to be seeing new, more transmissible variants of the virus spreading. What if it had been like that from the beginning? Measles is much more transmissible that Covid, and while its IFR is somewhat lower, the deaths are concentrated in small kids. The Spanish flu mostly killed young adults. In either case, the statistical loss of life was much higher than with the elder-focused Covid.
So when a new, worse virus emerges, what do we do?
The status quo, I think, traps us in a doom loop. Public health advocates think we should impose stricter NPIs and keep them in place until a vaccine is available. But the more effective the NPIs, the slower vaccine approval gets. And the vision of strict NPIs that need to stay in place indefinitely undermines compliance. Of course an individual country can successfully suppress the virus with NPIs (hello, New Zealand!), but they count on the clinical trials being done elsewhere where NPIs didn’t work. As a planet, we’re stuck.
This was a good enough approach to vaccine development when we were talking about long-endemic diseases to which the majority of the population already had acquired immunity. But while I hope it’ll turn out that Covid-19 was a freakish one-off and nothing like this happens again for hundreds of years, I’m not that confident. We should be making plans to succeed against the next pandemic — meaning rigorous adherence to effective NPIs paired with challenge trials to ensure fast approval of effective vaccines.
Again it all seems obvious until you design the experiment. “Just expose people to the virus” does a lot of work for you but it by no means an experimental design. Please interview a scientist on how they would choose to do this experiment and your argument will be stronger. Do you put the test subject in a room with sick people? Inject it? Gas people with it? How large is the exposure? When you do it the standard way, people are exposed to “normal” levels of the virus not some standard that must first be scientifically established. Not all viral infections are the same, quantities and methods matter.
I’m all for challenge studies to get a quick read on efficacy but I don’t see that doing anything about Phase 3 studies. Vaccine Phase 3 studies are not the size they are simply because of efficacy. It’s because you need large trials to identify safety signals that are too rare to pop up in a small Phase 1 trial. Covid is bad but most people haven’t had it and the vast majority of people survive. The vaccine is going to go into *everyone on the planet* in an ideal world. The risk/benefit ratio is hugely critical for a scenario like this, and it’s very challenging to track adverse events in a non-controlled environment. To avoid Ph3 studies you would need a new way to accurately track safety in the general public.