Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.

Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability.

We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems.

It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring.

The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions.

Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability.

The framework processes approximately 50,000 records daily and has evaluated more than two million interactions.

Validation used 12,980 stratified-random human-labeled records from four trained annotators.

Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains.

The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.