Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection

Large Language Models (LLMs) often hallucinate, limiting their reliability in sensitive applications.

In black-box settings, several self-consistency-based techniques have been proposed for hallucination detection.

We empirically show that these methods perform nearly as well as a supervised (black-box) oracle, leaving limited room for further gains within this paradigm.

To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM.

With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods.

We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases.

It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier.

We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights.

Extensive experiments on QA-style hallucination detection benchmarks show that this approach maintains high detection performance while significantly reducing computational cost.