Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback

Reinforcement learning from human feedback (RLHF) is a central post-training tool for aligning large language models, but its training reward is only a learned proxy for true human utility.

This creates a decision problem under objective misspecification: the policy is optimized against an estimated reward, while deployment performance is governed by an unobserved population preference.

The resulting gap leads to reward over-optimization, where proxy reward keeps improving after true quality deteriorates.

We propose distributionally robust regret optimization (DRRO) for RLHF with a Wasserstein ambiguity set over reward laws, using promptwise $\ell_p$ distances between reward vectors as transport costs.

Unlike standard distributionally robust optimization, which pessimizes worst-case value, DRRO pessimizes worst-case regret relative to the best policy under the same plausible reward perturbation.

We show that the expressive-policy problem decomposes into promptwise regret problems.

For each prompt, the inner adversary has a dual-norm closed form; under the $\ell_1$ transport cost used by our algorithm, the optimizer has a water-filling structure.

These results lead to a practical policy-gradient algorithm that adds a simple sampled bonus to GRPO-style training.

Theory and experiments both show that DRRO is less over-pessimistic than standard DRO and mitigates over-optimization more effectively than existing baselines.