Cross-Cultural Value Attribution in Large Vision-Language Models

The rapid adoption of large vision-language models (LVLMs) in recent years has been accompanied by growing fairness concerns due to their propensity to reinforce harmful societal stereotypes.

While significant attention has been paid to such fairness concerns in the context of social biases, relatively little prior work has examined the presence of stereotypes in LVLMs related to cultural contexts such as religion, nationality, and socioeconomic status.

In this work, we aim to narrow this gap by investigating how cultural contexts depicted in images influence the judgments LVLMs make about a person's moral, ethical, and political values.

We conduct a multi-dimensional analysis of such value judgments in nine LVLMs using counterfactual image sets, which depict the same person across different cultural contexts.

Our evaluation framework pairs descriptive analyses (Moral Foundations Theory categorization, lexical analyses, and value sensitivity) with a novel grounding analysis that compares LVLM cross-context variation against two large-scale human surveys (MFQ-2 and WVS Wave 7).

Across 4.8 million LVLM generations, we identify three bias patterns that replicate across architecturally diverse models: an inversion of the socioeconomic-status-to-Authority relationship found in WVS, and two race-conditional failures that override cultural context cues when depicting Middle Eastern persons.

Additional ablations show that the socioeconomic-status-to-Authority inversion bias is amplified by image conditioning and persists across different model sizes.