Calibrating Semantic Uncertainty from Observable Language-Model Probabilities

Language models produce probabilities over words, but professional decisions require uncertainty over meaningful states such as diagnoses, hypotheses or operational conditions.

A model's printed numerical confidence does not establish reliability.

We introduce a semantic map: a prespecified, testable bridge from probabilities over verbal responses to probabilities over declared states, formulated as semiparametric inference for a finite-valued latent state.

A reference model defines the target posterior, the language model supplies an unrestricted conditional distribution over verbal responses, and held-out calibration connects them.

We derive posterior-error bounds and conditions for existence, uniqueness, stability and sequential Bayesian updating.

Crucially, language probabilities depend on the prompt's lexical form, whereas the target posterior is unchanged by information-equivalent rewording.

We test the method on professional market text compiled from Federal Reserve economic and financial series and on controlled simulations with exact posteriors.

Across two fitted language models, language-derived probabilities outperform printed numerical probabilities, recover held-out posteriors with valid uncertainty coverage, remain largely stable under paraphrasing, and respond appropriately to altered evidence.

The broader implication is that prompt engineering optimizes a wording-dependent response, whereas scientific and professional use requires validated stability of application-relevant meaning.

The semantic map turns this general concern into a testable statistical problem and, when its acceptance conditions hold, yields an auditable posterior estimate.

The same principle offers a template for auditing classifications, recommendations and other fluent responses that may conceal semantic instability.