EN
FR
ES
DE
IT
KO
JA
AI judging, or LLM-as-a-judge, means each answer is scored by a language model judge drawn from a multi-vendor pool. A sample is double-scored by a second judge to measure agreement. Mixing vendors limits self-preference bias.
The principle
Rather than scoring an AI answer with a single model — which would tend to favour itself — you convene a panel of several judge models from different providers. Each one scores the answer against precise rubrics (presence, accuracy, sentiment), and the scores are then aggregated.
Why several judges?
- Anti self-preference: a model cannot favour its own answers if the panel is diverse.
- Robustness: the occasional errors of one judge are smoothed out by the others.
- Detecting hallucinations: an unverifiable claim flagged by the judges lowers the score.
Reliability and traceability
The agreement between judges is measured by an inter-rater reliability coefficient. The exact composition of the panel for each audit is tracked via a configuration fingerprint, ensuring comparability over time. This panel is at the core of the AGS; see the methodology and the demonstration.
Going further
When your brand is absent from an AI answer, a competitor takes its place 37% of the time. AI Labs Audit data: sub-sample of our 3,002-answer study.