Estimating Causal Effects from Data Generated by Stochastic Algorithms

Recommendation systems and chatbots present content to users, typically using stochastic algorithms that select the content based on user characteristics or context.

Examples of content include chat responses, videos, or items available for purchase.

Scientists and application developers are often interested in whether characteristics of content increase outcomes such as user engagement.

Estimates of such causal effects may guide content providers to generate content that emphasize desirable features.

However, in settings with a large content library or where content is generated uniquely for a given user, it can be difficult to use observational data to learn the causal effect of content features, because the content a user sees is tailored to that user, and because content varies in many dimensions.

This paper proposes a new method for estimating the impact of content features using observational data, when the algorithm that determines user exposure incorporates some randomization, and when two additional data elements are logged for each user: $(i)$ the identity of at least one item that could have been exposed to the user, but was not (the unexposed item); $(ii)$ an estimate of the ratio of the probability that the unexposed item would have been shown to the probability that the exposed item was shown.

We show that causal effects of features are identified in this setting, even in the presence of unobserved confounders that affect both user preferences and the identity of the considered pair of items (exposed and unexposed).

Our estimator differs from prior approaches in terms of what data is used and how the estimator is constructed.