Background
Throughout the pandemic, policy makers from local levels (county and state health officials, school boards, and governors) to national and federal levels such as CDC directors and White House officials, relied on modeling to guide decisions. Public health has a long history of using epidemiologic models for a variety of purposes: (i) To gain understanding of infectious disease dynamics, (ii) to predict future health care needs to ensure sufficient capacity, and (iii) to fill in for missing real world data. When using models to make public-health policy decisions, it is crucial that politicians, policy makers, and public health officials clearly understand data weaknesses, underlying assumptions used to generate models and forecasts, the nature of input parameters, and uncertainties inherent in any model.
At the outset, models from the Institute of Health Metrics and Evaluation at the University of Washington (IHME) and Imperial College in London, as well as models generated by the CDC, were influential both locally and nationally. These models tried to forecast COVID-19 cases, hospitalizations, and deaths under different pandemic lockdown strategies, by modeling the effects on COVID-19 from school closures, public gathering restrictions, suspension of health care services, business closures, limiting restaurant capacity, quarantining people, travel restrictions, and mass asymptomatic testing. Mask models were used as support for mask mandates and models assuming that vaccination halted transmission were used when approving, recommending and mandating vaccines.
Infectious Disease Forecasts
Models used to forecast infectious disease cases, hospitalizations, and deaths are complex, with arcane assumptions built into mathematical formulas. These models are sensitive to assumptions about input parameters that violate real-world conditions. Assumptions and limitations are not always understood by the ultimate consumers of the model, including policy makers. It is important to conduct sensitivity analyses, because if model parameters are overly reliant on specific inputs, this greatly limits their usefulness and predictive ability at forecasting using real-world data, which tend to be messy and variable.
In March 2020, professor Neil Ferguson and colleagues at Imperial College published alarming COVID-19 mortality forecasts. At the same time professor Sunetra Gupta, an infectious disease epidemiologist at Oxford University, suggested that various scenarios of spread were compatible with available COVID-19 data. The Gupta model highlighted three key sources of uncertainty in these forecast models: (1) the date of initial seeding of the virus in populations; (2) the inherent infectivity of the virus; and (3) the infection fatality rate. These sources of uncertainty are related, meaning that a virus with both high infectivity and high infection fatality rate is highly unlikely. Gupta and colleagues called for these uncertainties to be resolved before policy makers relied heavily on these models to craft policy.
Why did world leaders overly rely on models that made unverified assumptions about the pandemic’s trajectory rather than trying to verify these assumptions and their implications? Did politicians and public health officials understand inherent limitations in epidemiologic COVID-19 models?
While technical aspects of modeling are complex, it is important to understand that any model, in order to make accurate predictions, must be based on accurate data on initial disease prevalence in the population. Why did the CDC not conduct seroprevalence surveys? Why did policy makers assume that Chinese reports about initial disease spread, released in December 2019, were accurate? Published in the fall of 2020, antibody detection assays in Italy and France indicated a late summer 2019 spread. Why were these data not factored into subsequent models?
Once it became obvious it would be very difficult to limit COVID transmission in the general community, why didn’t policy makers prioritize models focusing on the age gradient in risk?
Why were the most influential models from IHME, Imperial College, and CDC, only accompanied by limited sensitivity analyses, instead of by an extensive evaluation with many different possible input parameters? Were experts with relevant knowledge included in discussions of model parameters?
Why didn’t more modelers speak up about the difficulty of accurately predicting COVID-19 cases, hospitalizations, and deaths? Did epidemiological disease modelers sufficiently explain inherent model limitations to politicians and other consumers?
Websites to enable open-source modeling exist and are critical to promote transparency and peer-review of model assumptions. Were influential models, particularly at the state level, critiqued transparently?
Around 15 years ago, to prepare for a potential pandemic NIH launched the Models of Infectious Disease Agent Study (MIDAS), funding a network of more than one hundred infectious disease modelers, including Neil Ferguson and six of his colleagues at Imperial College. Considering how poorly their models performed at predicting the behavior of the COVID-19 pandemic, will NIH continue to fund MIDAS?
After forecasting models failed for COVID-19, the CDC launched the Center for Forecasting and outbreak Analysis (CFA). How does CFA plan to avoid repeating the modeling failures during the pandemic?
Why did some states and governors rely on local models to shut down schools and businesses when those models were not vetted or made transparent and the model creators did not necessarily have experience in epidemiological modeling?
Why did many models appear to ignore aspects of human nature, such as the desire to gather?
Did models consider the disparate impacts that lockdowns would have on different socioeconomic groups?
Pandemic Concepts and Parameters
Epidemiological models are important for estimating pandemic parameters such as infection fatality rate, case fatality rate, person-to-person transmission, and reproductive number.
In 2020, health agencies and the media confused the case fatality rate (CFR) with the infection fatality rate (IFR). The former is the risk of death among known cases. The latter is the risk of death if infected, which, in the case of SARS-CoV2, is much lower since many cases are asymptomatic or mild and go undetected by health officials. Why was there confusion about these basic epidemiological concepts? Why did the CDC and NIH not clarify this misunderstanding? How did confusing the two concepts drive panic in the general population?
Studying transmission on the Diamond Princess cruise ship demonstrated that the asymptomatic transmission rate was around 18%. Furthermore, data collected on the Diamond Princess cruise ship suggested age stratification of severe disease. While the exact numbers are debatable, as they have been adjusted by reported Chinese data, the IFR from this outbreak was significantly lower than initial calculations from the WHO, and should have raised questions about the high IFR used to instigate restrictions such as school closures. Were policy makers aware of these data and of the major age-stratified risk from COVID-19?
Modeling Collateral Lockdown Damage
Nearly all the modeling efforts used by public health officials during the pandemic focused on predicting COVID-19-related parameters, such as trajectories of cases, hospitalizations, and COVID-19-related mortality, as well as on predicting effects of non-pharmaceutical interventions such as masking and distancing in schools. However, public health measures had a broad range of collateral consequences beyond COVID-19, such as learning loss from closed schools, worsening mental health from fewer social contacts, canceled cultural events and religious services, more substance use and weight gain due to isolation and depression, and worse cancer outcomes from delayed cancer screenings and missed cancer treatments, to name a few.
Why did public health scientists develop models to forecast COVID-19 but not to forecast health and economic outcomes resulting from collateral damage due to non-pharmaceutical interventions?
Why did public health authorities accept models forecasting health consequences from COVID-19, without insisting on models also forecasting collateral public health damage due to pandemic mitigations?