Optimal Designs with Robust Inference for Binary Treatment Effects

We study randomized experiments with binary outcomes under Neyman's nonparametric model, where covariate measurements are fixed but potential outcomes are random.

In this setting we derive the exact variance of the difference-in-means estimator and characterize designs that minimize it.

We show that any balanced design satisfying a covariate-balance condition is asymptotically optimal, and we prove that a broad class of blocking designs satisfies this condition under mild smoothness assumptions.

Because the variance depends on unknown success probabilities, unbiased variance estimation is impossible.

We therefore develop two conservative estimators: a generalization of the Cochran-Mantel-Haenszel (CMH) statistic applicable to any balanced design, and an extension of Robins' variance estimator for blocking designs.

We establish conditions under which the CMH-based estimator is asymptotically tight under local alternatives, thereby yielding asymptotically valid confidence intervals.

Our theoretical and simulations results show that blocking and other designs that achieve covariate balance and sufficient degree of randomness, perform well when they are equipped with the CMH-based inference.

Thus, we provide an experimental design and inference framework for incidence outcomes that is simultaneously variance-optimal, conservative in finite samples and asymptotically tight.