Reconciling Bayesian and frequentist approaches to robustness against outliers

Heavy-tailed models are used as a way to gain robustness against outliers in Bayesian analyses.

In frequentist analyses, M-estimators are often employed.

In this paper, the two approaches are tentatively reconciled by considering M-estimators as maximum likelihood estimators of heavy-tailed models.

From this perspective, it is realized that a fundamental difference exists as frequentists, contrarily to Bayesians, do not require these heavy-tailed models to be proper.

For instance, a popular robust estimator in linear regression, Tukey's biweight M-estimator, does not correspond to a proper heavy-tailed model.

Thus, a Bayesian practitioner does not have access to the same range of tools as a frequentist practitioner.

It is shown through two real-data linear regression analyses that the former may in consequence obtain significantly different estimation results than the latter, where the difference is due to a more pronounced influence by the outliers in the former case.

It is highlighted that a way to give these practitioners access to the same range of tools is for the Bayesian to adopt the generalized Bayesian framework of Bissiri et al.

(2016) which allows the use of improper models, in combination with proper prior distributions yielding proper generalized posterior distributions.

A complete reconciliation of the Bayesian and frequentist approaches to robustness is then achieved.

An extensive theoretical study of the generalized Bayesian counterpart of Tukey's biweight M-estimator is provided, which includes a robustness characterization result, a strong consistency result and a Bernstein--von Mises result, the latter allowing to calibrate the generalized posterior distribution for meaningful uncertainty quantification.

After adopting the generalized Bayesian framework, the Bayesian practitioner obtains similar results as the frequentist practitioner in the aforementioned examples.