학술

How Much to Experiment? Admissible Sample Sizes in Test-and-Roll Experiments

arXiv Econ
CC BY
이 매체는 공공·자유 라이선스로 본문을 직접 표시합니다.

Abstract

This paper studies welfare-based sample-size choice in finite-population test-and-roll experiments.

A decision-maker conducts a balanced randomized experiment on $m$ of $N$ units and assigns the remaining $N-m$ units to the arm with the higher sample mean.

Regret combines the welfare cost of experimentation with the risk of an inferior rollout.

Direct minimax regret is uninformative when treatment-effect magnitudes or noise scales are unrestricted, while minimax-regret choices under parameter bounds can depend materially on those bounds.

We therefore derive a sharp characterization of admissible sample sizes, defined as those not dominated in regret by any other sample size under the same rollout rule.

Under a Gaussian approximation and continuous relaxation, the admissible set for two treatments is exactly $(0,N/3]$, with $N/3$ strictly dominating every larger sample size.

This characterization requires neither a prior nor externally specified parameter bounds, while partial information on the signal-to-noise ratio further narrows the admissible set.

We extend the analysis to covariate targeting and multiple treatment arms.

Under covariate targeting with a plug-in rule, the largest admissible sample size depends on the convergence rate of the conditional average treatment effect (CATE) estimator and falls below $N/3$ for slower-than-root-$m$ convergence.

With three or more treatments, even arbitrarily small variance heterogeneity makes every sample size admissible, whereas under common variance the largest admissible sample size lies in $(N/3,N)$ and depends on the number of treatments.

관련 뉴스