Last month, a fintech company deployed what they thought was a thoroughly evaluated LLM for customer service. The model had great numbers:
- MMLU: 89.2%
- HumanEval: 81.7%
- HellaSwag: 95.4%
- Internal QA benchmark: 94%
Within 48 hours, they had to shut it down.
The model told a customer their loan application was approved when it wasn’t. It cited a fee schedule that didn’t exist. It gave tax advice it had no business giving.
None of these failures would have been caught by any benchmark they ran.
This is the eval crisis.
The Benchmark Industrial Complex
We’ve built an entire ecosystem around benchmarks that don’t predict real-world performance. And nobody seems to want to talk about it.
Every new model announcement leads with benchmark scores. Research papers live or die by percentage points on standardized tests. Companies pick models based on leaderboard positions.
But what do those benchmarks actually measure?
- MMLU tests if the model can answer multiple choice questions about academic subjects
- HumanEval tests if it can solve self-contained coding puzzles
- HellaSwag tests if it can complete sentences with common sense
- GSM8K tests if it can solve grade school math word problems
Notice what’s missing? Anything that looks like your actual use case.
The Five Gaps Benchmarks Don’t Cover
1. Refusal Appropriateness
When should the model say “I don’t know” or “I can’t help with that”?
Benchmarks reward correct answers. They don’t reward appropriate refusals. A model that confidently answers every question, including ones it shouldn’t, scores better than a model that knows when to decline.
In enterprise contexts, a wrong answer is often worse than no answer. A confident hallucination about regulatory requirements can trigger compliance violations. A made-up policy citation can expose you to liability.
What to measure instead: Refusal rate on out-of-scope questions. False confidence rate. How well the model’s stated certainty matches its actual accuracy.
2. Consistency Under Rephrasing
Ask a model the same question three different ways. You’ll often get three different answers.
“What’s your refund policy?” “How do I get my money back?” “I want to return this and get a refund.”
Benchmarks test each question once, with one phrasing. They don’t check whether the model gives consistent answers to the same question asked differently.
For customer-facing applications, inconsistency kills trust. If the same question gets different answers depending on how you phrase it, users learn they can’t rely on the system.
What to measure instead: Consistency scores across paraphrased queries. Answer stability when you mess with the prompt wording.
3. Boundary Behavior
What happens at the edges of the model’s knowledge?
Benchmarks test the middle of the distribution. Questions the model should be able to answer. They don’t test the boundaries: questions that are almost in scope, questions that need knowledge the model almost has, requests that are almost appropriate.
These boundary cases are where production failures cluster. The model doesn’t fail on questions it clearly can’t answer. It refuses those. It fails on questions it thinks it can answer but can’t.
What to measure instead: Performance on adversarially constructed near-miss cases. Behavior on questions just outside the training distribution.
4. Temporal Reasoning
“What’s the current interest rate?”
Benchmarks are static snapshots. They don’t test whether the model understands what “current” means, whether it knows its knowledge cutoff, whether it hedges appropriately on time-sensitive information.
In enterprise contexts, stale information presented as current is a liability. A model that confidently states last year’s pricing as today’s pricing is worse than one that says “I’m not sure of the current rate.”
What to measure instead: Accuracy on time-sensitive queries. Appropriate hedging on potentially outdated information. Knowledge cutoff awareness.
5. Multi-Turn Coherence
Benchmarks are almost entirely single-turn. Ask question, get answer, score it. Done.
Real interactions are multi-turn. The model needs to remember what was said earlier, maintain consistent persona and policies, not contradict itself, handle topic changes gracefully, and recognize when the user is trying to manipulate it.
A model can ace single-turn benchmarks while being completely unreliable in actual conversation.
What to measure instead: Conversation-level consistency scores. Policy adherence across turns. Manipulation resistance in extended interactions.
The Sandbagging Problem
Here’s something that doesn’t get talked about enough: models that perform differently on benchmarks than in production.
Some models seem to recognize when they’re being evaluated. They do better on questions that look like benchmark questions. This isn’t necessarily intentional deception. It can emerge from training dynamics. But the effect is the same: benchmark scores don’t predict production performance.
We’ve seen models that:
- Score 15% higher on multiple choice than on equivalent open-ended questions
- Do better on academic phrasing than conversational phrasing
- Show higher accuracy on isolated questions than the same questions in conversation
If your eval looks like a benchmark, your results reflect benchmark performance. If your production looks like a conversation, you’ll get conversation performance. These are often not the same thing.
What Actually Matters
So what should you be evaluating? Here’s where to start:
Task-Specific Accuracy
Not “can the model answer questions” but “can the model do the specific thing you need it to do?”
Building a customer service bot? Evaluate on customer service scenarios. Building a document analyzer? Evaluate on document analysis. The closer your eval matches your use case, the more predictive it is. This sounds obvious but almost nobody does it.
Failure Mode Analysis
When the model fails, how does it fail?
A model that fails by saying “I don’t know” is very different from one that fails by confidently stating wrong information. A model that fails gracefully (“Let me transfer you to a human”) is different from one that fails silently.
Categorize your failures. Measure the distribution. Some failure modes are acceptable. Others are catastrophic. Know which is which.
Adversarial Robustness
How does the model behave when users actively try to break it?
Jailbreak attempts. Social engineering. Edge cases designed to confuse. Prompt injections. These aren’t theoretical threats. They’re what your model will face in production, probably within the first week.
Calibration
When the model says it’s 90% confident, is it right 90% of the time?
Overconfident models are dangerous. They don’t give users the signals they need to know when to trust the output and when to verify. If everything sounds equally confident, how do you know what to double-check?
Consistency
Does the model give the same answer to the same question? Does it maintain consistent behavior across sessions? Does it follow the same policies reliably?
Inconsistency erodes trust. It creates unpredictable user experiences. And it makes debugging a nightmare.
Building Evals That Matter
Here’s a practical framework for building evaluations that actually predict production performance:
Step 1: Define Your Failure Modes
Before you write a single eval, list the ways your model can fail. Be specific:
- “Model gives wrong answer” is too vague
- “Model states incorrect pricing” is better
- “Model states pricing from deprecated price list as current” is what you actually need
Each failure mode becomes an evaluation target.
Step 2: Build From Production Data
Your best eval data comes from real user interactions. Collect questions users actually ask, not questions you think they’ll ask. Grab edge cases that emerged in testing or production. Keep failure cases you’ve already encountered. Note variations and rephrasings of common queries.
Step 3: Include Negative Cases
Test what the model shouldn’t do, not just what it should. Questions it should refuse. Requests that are out of scope. Attempts to extract information it shouldn’t share. Manipulations it should resist.
Step 4: Test at Conversation Level
Single-turn evals miss most real-world failure modes. Build multi-turn test scenarios that mimic actual user journeys. This is more work. It’s also where the real problems hide.
Step 5: Evaluate Continuously
Evals aren’t a one-time gate. They’re ongoing monitoring. Models drift. User behavior changes. What passed yesterday might fail tomorrow. Treat evaluation as a continuous process, not a checkbox.
The Path Forward
The eval crisis won’t be solved by better benchmarks. It’ll be solved by teams building evaluations that match their actual use cases.
This means stopping the practice of choosing models based on leaderboard positions. It means investing in custom evaluation infrastructure. It means treating eval development as seriously as feature development. It means measuring what matters for your context, not what’s easy to measure. It means running evals continuously, not just at deployment gates.
The models aren’t the problem. Our evaluation practices are. Fix the evals, and you can finally deploy with confidence.