The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

Standard decoding strategies for text generation, including top-$k$, nucleus sampling, and contrastive search, select tokens based on likelihood, restricting outputs to high-probability regions.

In contrast, human language production prioritizes communicative appropriateness, allowing the use of contextually suitable but statistically rare tokens.

This mismatch induces a \emph{truncation blind spot}, whereby such tokens remain accessible to humans but are systematically excluded by likelihood-based decoding.

We investigate this phenomenon using over 1.8 million machine-generated texts from eight language models, including large proprietary systems (GPT-3.5-turbo, Claude-3-Haiku), across five decoding strategies and 53 hyperparameter settings, alongside 5,261 human-written references.

We find that 8--18\% of human-selected tokens fall outside typical truncation boundaries.

This exclusion is not random: content-bearing tokens are omitted at rates $2.9\times$ higher than grammatical function tokens.

As a consequence, simple classifiers based on predictability and lexical diversity separate machine-generated from human-written text with mean AUC-ROC above 0.97.

Detectability persists across model scales, architectures, and alignment procedures, and instead tracks the intensity of truncation.

A classifier trained only on the oldest model in our study (GPT2-XL, 1.5B) detects outputs from substantially more recent and capable systems at near in-distribution accuracy, indicating that the detection signal is shared across generators rather than model-specific.

These results indicate that detectability is a structural consequence of likelihood-based token selection rather than a limitation of model capability.

We release code, datasets, and analysis at this https URL