LLM-as-Judge: How to Use a Model to Evaluate a Model

1Key Takeaways

• LLM-as-judge uses a capable model to score or compare agent outputs against a rubric — filling the gap where quality is open-ended and human judgment doesn't scale.
• Done well it approximates human judgment cheaply; done carelessly it produces confident nonsense.
• The keys: a specific rubric, pairwise over absolute scoring, and validating the judge against human labels.
• Why use a model as a judge Many of the qualities that matter most — helpfulness, faithfulness, reasoning quality — have no….