Calibration of answer probabilities in verbal autopsies: Working Paper

We consider the problem of calibrating a posterior estimator of a categorical latent variable given a fixed-length ternary string using only unlabelled observations and without a specified likelihood. We primarily consider a setting in which the estimator is parametrised by estimated conditional probabilities of elements of the string given the latent variable, with calibration if the estimates are correct.
Our motivating application is the `Verbal Autopsy' procedure, whereby a cause of death is probabilistically inferred following a structured interview with associates of the deceased. More generally, our setting applies to circumstances where experts can more readily describe posterior beliefs than likelihoods, due to similarity with diagnostic practices.
We argue combinatorially that in general the problem is intractable without a simplifying assumption on the data distribution, though some posterior estimators can be ruled out as incompatible. We propose an assumption of block-conditional independence on substrings, allowing calibration procedures based on substring frequency, imputation, and pairwise and three-way distributions of string elements. We give theoretical results on identifiability, on consistency for distributions of either fixed support or fixed entropy, and on robustness to assumptions, finding essentially that three conditional independence blocks of size at least the number of latent categories are necessary and sufficient for calibration. We empirically evaluate methods on data simulated to resemble realistic verbal autopsy questionnaires, and find substantial promise for the approach in the practical problem of calibrating posterior estimates for causes of death.