A Note on Calibration of Clinical Prediction Models with Copas Statistics
Abstract
Background: Calibration of clinical prediction models often entails assessing goodness of fit with independent, non-identically distributed Bernoulli random variables. We here investigate two statistics studied by Copas in this setting.
Materials and Methods: We present distribution theory and a simulation study to compare the operating characteristics of the Copas statistics.
Results: In our simulation study with relatively small sample sizes, we found a simple Cornish-Fisher approximation tail quantiles of the distributions of the Copas statistics to perform adequately. Upon illustrating their use in a calibration study relating to prediction of atherosclerotic cardiovascular disease risk, power properties appear to reflect differential weighting accorded to observations, as evinced with other goodness-of-fit statistics.
Conclusion: The Copas statistics are easily implemented, have proven value in other contexts, and appear to be underutilized in calibration studies. They ought to be part of the armamentarium of calibration tools for all researchers
2. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Statistics for Biology and Health). New York: Springer, 2009.
3.Harrell FE. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis , 2nd ed.(Springer Series in Statistics). New York: Springer, 2015.
4.Hosmer DW, Lemeshow S. Goodness of fit tests for the multiple logistic regression model. Commun Stat Theor Meth 1980;9:1043-69.
5.Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness of fit tests for the logistic regression model. Statistics in Medicine 1997;16:965-980.
6.Hosmer DW, Lemeshow S, Sturdivant RX. Applied Logistic Regression 3rd Edition. New York: John Wiley, 2013.
7.Hand DJ. Measuring classifier performance: a coherent alternative to the area under the roc curve. Machine Learning 2009;77:103-23.
8.Austin PC, Reeves MJ. The relationship between the C-statistic of a risk-adjustment model and the accuracy of hospital report cards: a Monte Carlo study. Medical Care 2013;51:275-284.
9.Cornish EA, Fisher RA. Moments and cumulants in the specification of distributions. Revue de l'Institut International de Statistique / Review of the International Statistical Institute 1938;5:307–320.
10.Goff DC, Lloyd-Jones DM, Bennett G et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association task force on practice guidelines. Circulation 2014;129:S49
11.Muntner P, Colantonio LD, Cushman M et al. Validation of the atherosclerotic cardiovascular disease pooled cohort risk equations. JAMA 2014;311:1406-1415.
12.Paul P, Pennell ML, Lemeshow S. Standardizing the power of the Hosmer-Lemeshow goodness of fit test in large data sets. Statistics in Medicine 2013;32:67-80.
13.Brier GW. Verification of forecasts expressed in terms of probability. Monthly Weather Review 1950;78:1-3.
14.Gerds TA, Cai T, Schumacher M. The performance of risk prediction models. Biometrical Journal 2008;50:457-479.
15.Bradley AA, Schwartz SS, Hashino T. Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather and Forecasting 2008;23:992-1006.
16.Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine 2014;33:517-535.
17.Ensor J, Snell KIE, Martin EC. PMCALPLOT: Stata module to produce calibration plot of prediction model performance. Statistical Software Components S458486, Boston College Department of Economics, revised 1 November 2018.
| Files | ||
| Issue | Vol 6 No 4 (2020) | |
| Section | Methodology | |
| DOI | https://doi.org/10.18502/jbe.v6i4.5687 | |
| Keywords | ||
| calibration clinical prediction models Copas statistics | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
