Formalising Sample Size Calculations for the Development of Risk Prediction Models: The Importance of Accounting for Performance Variability

Sample size calculations for developing prediction models typically aim to ensure that the expected value of a performance measure meets a prespecified target. For example, a key measure is the calibration slope (CS) which quantifies model overfitting; the sample size is often chosen so that the expected CS equals 0.9, close to the ideal value of 1. However, because of sampling variability, model performance can vary substantially across development samples of the recommended size. When variability is high, the probability of obtaining a model with performance close to the target may be unacceptably low.
We propose a framework which enables sample size calculations that incorporate both the expected value and the variability for a given performance measure. The framework is illustrated for binary outcomes and logistic regression but applies to other outcomes and model types. To explicitly account for variability, we introduce the probability of acceptable performance (PrAP). For example, a model may be considered acceptable if the CS lies within a prespecified range (e.g., 0.85 to 1.15) and the sample size might be chosen to ensure that PrAP exceeds some target (e.g., 80%). Under existing approaches PrAP can be low, especially when the specified number of predictors is small, which can also translate into large variability for individual predicted probabilities. The use of shrinkage tends to improve PrAP.
Our findings highlight the importance of accounting for performance variability to ensure robust model development.