Prediction-Powered Active Testing

Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled.

However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive.

To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate.

Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance.

Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator.

Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates.

Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.