AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation

Millions of people now use generative AI chatbots for psychological support.

Despite their promise, the most pressing question in AI for mental health is whether these tools are safe.

The field currently lacks a validated, automated benchmark for evaluating AI chatbot safety, particularly for users at risk of suicide.

The Validation of Ethical and Responsible AI in Mental Health (VERA-MH) evaluation was recently proposed to address this need.

This human validation study examined the alignment of VERA-MH safety ratings with expert clinician judgments.

We simulated conversations between large language model (LLM)-based users spanning a range of suicide risk levels and disclosure styles and general-purpose AI chatbots.

Licensed mental health clinicians from Spring Health independently rated chatbot safety using the VERA-MH scoring rubric.

An LLM-based evaluator ("judge") applied the same rubric to the same conversations.

We examined agreement among clinicians, between clinician consensus and the LLM judge, and across different judge LLMs.

Clinicians also rated user-agent realism, suicide risk, and disclosure.

Clinicians showed strong agreement in safety ratings (chance-corrected inter-rater reliability [IRR] = 0.77), establishing a reliable clinical consensus reference.

The LLM judge was strongly aligned with this consensus (IRR = 0.81), and ratings were stable across judge models and repeated evaluations.

Ratings of user-agent realism and fidelity to intended suicide risk and disclosure styles were mixed.

These findings support the reliability of VERA-MH as an open-source, fully automated benchmark for evaluating AI chatbot suicide risk detection and response.

Because these results reflect an earlier version of the benchmark, future work should validate updated versions, assess generalizability and robustness, and expand VERA-MH to additional domains of AI safety in mental health.