Active Exploration via Autoregressive Generation of Missing Data

We pose uncertainty quantification and exploration in online decision-making as a problem of training and generation from an autoregressive sequence model, an area experiencing rapid innovation.

Our approach rests on viewing uncertainty as arising from missing future outcomes that could be revealed through action choices, rather than from unobservable latent parameters of the environment.

This reformulation aligns naturally with modern machine learning capabilities: we can i) train generative models through next-outcome prediction rather than fit explicit priors, ii) assess uncertainty through autoregressive generation rather than sampling latent parameters from posteriors, and iii) adapt to new information by extending the sequence model's context rather than explicit posterior updating.

Our main theoretical result establishes a reduction from online learning to offline next-outcome prediction, showing that Bayesian regret is controlled by the offline sequence prediction loss.

Semi-synthetic experiments show our insights bear out in a challenging news recommendation setting, where effective performance requires leveraging article headline text as prior information to focus exploration on resolving remaining uncertainties.