| Property | Value |
|---|
|
?:abstract
|
-
Dozens of coronavirus (COVID-19) forecasting models have been created; however, little information exists on their performance. Here we examine the performance of nine commonly-used COVID-19 forecasting models, as well as equal- and performance-weighted ensembles, based on their knowledge - i.e., accuracy and precision, and their \'self-knowledge\' - i.e., \'calibration\' and \'information\'. Calibration and information are measures commonly employed in structured expert judgment to assess an expert\'s ability to meaningfully communicate the extent and limits of their knowledge. Data on observed COVID-19 mortality in 4 states, selected to reflect differences in racial composition and COVID-19 case rates, over eight weeks in the summer of 2020 provided the basis for evaluating model predictions. Only two models showed little bias (geometric mean of observed/predicted < 10%) and good precision (geometric standard deviation of observed/predicted < 1.6). Three models demonstrated good calibration and information. However, only one model exhibited superior performance in both dimensions. Nearly all models under-predicted COVID-19 mortality, some quite substantially. Further, model performance depends on racial composition and case rates, and forecasts in the short-term outperform forecasts in the medium-term on all criteria. The performance-weighted ensembles also outperformed the equal-weighted ensemble on all criteria. The ability of models to accurately and precisely predict mortality and the ability of the modelers to provide meaningful characterizations of the uncertainty in their estimates are potentially important to model developers and to those using model output to inform decisions.
|
|
is
?:annotates
of
|
|
|
?:creator
|
|
|
?:doi
|
|
|
?:doi
|
-
10.1101/2020.12.09.20246157
|
|
?:license
|
|
|
?:pdf_json_files
|
-
document_parses/pdf_json/4aae5510d9c573eb7338150a72276dc2f6867022.json
|
|
?:publication_isRelatedTo_Disease
|
|
|
?:sha_id
|
|
|
?:source
|
|
|
?:title
|
-
Assessing the Performance of COVID-19 Forecasting Models in the U.S.
|
|
?:type
|
|
|
?:year
|
|
![[This page as RDF]](https://research.tib.eu/covid-19/static/rdf-icon.gif){kind=icon}