Beyond Publication-Bias Detection: Estimating Bias under Uncertainty in Heterogeneous Literatures

Meta-analysts routinely test for publication bias, but a nonsignificant test is inconclusive because it may be a Type 2 error.

In a large factorial simulation, I show that publication-bias tests have low power even with 1,000 studies once effect sizes are heterogeneous and the data do not meet the model's assumptions.

I argue that the goal should shift from detecting bias to estimating it with confidence intervals.

A confidence interval does not only bound the hypothesis that bias is absent; its upper bound also indicates how much bias remains compatible with the data.

When the interval is wide, the upper bound does not exclude a large amount of bias, even if the formal test is nonsignificant.

I then show that confidence intervals from step-function selection models are too narrow, covering the true amount of bias only about half the time.

In contrast, z-curve's confidence interval obtained by transforming the expected discovery rate into a selection weight parameter has nominal coverage.

Z-curve is therefore a valuable tool for examining publication bias in meta-analyses, especially when heterogeneity is high.

I illustrate the implications of bias estimation with a meta-analysis of social priming and a set of applied-psychology meta-analyses.