Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

Data valuation is a natural framework for understanding which preference datasets matter most when aligning a Large Language Model (LLM) using multiple sources.

The standard game-theoretic approach assigns each dataset a contribution score via the Shapley value.

In practice, however, Shapley-based valuation is computationally prohibitive because it requires fine-tuning a separate model for every possible coalition of preference datasets, i.e., an exponential number of alignments.

We address this challenge for a broad family of preference-optimization objectives, including DPO and IPO, that learn directly from log-policy ratios with respect to a reference policy.

We introduce Sequential Preference Optimization, an offline procedure that applies existing preference optimization methods sequentially, source by source, updating the current policy after each dataset.

Under exact optimization, this procedure yields an additive composition rule in reward space and an equivalent arithmetic composition rule in policy space.

This observation enables an efficient approximation of the Shapley value: we train one model per preference dataset and reconstruct coalition policies at inference time from the singleton models, reducing the required alignments from exponential to linear in the number of sources.

Leveraging this property, we compute Shapley values for several real-world preference datasets and reveal how each source drives model alignment.