When and How to Pilot: Design Rules for Two-Wave Experiments

Experimenters often run pilots, but how much a small pilot should shape the main-wave design has no settled answer.

This paper shows how noisy pilot evidence should guide treatment assignment probabilities in two-wave experiments.

Two canonical rules mark the extremes.

Balanced assignment guards against worst cases but ignores evidence that one arm is noisier.

Feasible Neyman allocation adapts, but with a finite pilot it can overreact to noise, producing arbitrarily large precision losses.

We propose a Conditional Minimax Regret (CMR) rule that minimizes worst-case regret over a finite-sample confidence set for the treatment and control variances.

CMR retains balance's worst-case protection with high probability, converges to the Neyman allocation as the pilot grows, and attains the minimax-regret rate up to constants.

It extends to multi-arm and stratified designs, and simulations calibrated to four field experiments show it avoids feasible Neyman's severe small-pilot losses while capturing most of its large-pilot gains.