Why I Stopped Using LLM-as-a-Judge (And What I Use Instead)

Six months ago I was running evals on a coding Q&A dataset and my OpenAI bill hit $47 before I finished my test run. I wasn't doing anything exotic — just asking GPT-4 to score 500 model responses on a rubric. That's the hidden cost of LLM-as-a-Judge: every evaluation call is itself an API call, which means the more you evaluate, the more you pay. I spent a weekend building an alternative. The idea was simple: train a local binary classifier using Sentence Transformers and logistic…