Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning.

However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks.

Directly pooling heterogeneous source data can therefore lead to negative transfer.

To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs.

TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility.

The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.