How LLM Evals Cut My Model Cost by 88% and Raised Accuracy

Picking a production LLM is an engineering decision, and it deserves the same evidence as any other. For Officebot, I built an eval suite to make that call. It moved production from Claude Sonnet 4.5 to Kimi K2.5: 88% cheaper, 71% faster, and more accurate (99.7% vs 97.0%).

This post walks through that eval: how it's built, what its test cases look like, and what it showed across six models and three runs — the reason I could make the switch with evidence instead of a hunch.

This is Part 1 of a series on evaluating Officebot. Part 1 is offline evals. Part 2 is production evals with Langfuse and LLM-as-judge.

The question: can a cheaper model match Sonnet?

Officebot is a Slack app for hybrid teams. It brings leave tracking, in-office and WFH days, and shift rostering into one place. Instead of slash commands or forms, people type what they mean: "wfh tomorrow and pto next Friday" or "who's in the office today?"

To make that work, Officebot moved from classic NLP to agentic NLU, built on the Vercel AI SDK and AWS Bedrock, with Sonnet 4.5 doing the reasoning. Sonnet worked well. It was also the most expensive line on the bill.

That matters more than usual because of how Officebot is priced: $2–3 per user per month, against $5–12 for established competitors. At that price point, the model's cost per request comes straight out of the margin.

Accuracy can't be traded for savings in an NLU system. When the model picks the wrong tool, the user doesn't get a slightly worse answer. They get the wrong action. So the brief was narrow: find a cheaper model that is at least as accurate as Sonnet on Officebot's own workload.

That second half matters. Public benchmarks measure general ability. They don't tell you how a model handles your tools, your knowledge base and your users' phrasing. Only your own eval can answer that.

How the eval is built

The suite runs in promptfoo and holds 99 test cases, 33 for each of three things Officebot must get right:

• Tool selection accuracy: given a request, does the agent call the right tool with the right arguments? "Book PTO next Monday" must call create_leave with the correct dates.
• Knowledge-base relevance: product questions ("what does Officebot cost?") must be answered from the knowledge base, and must not trigger an action tool. Questions about the team's own setup must go to the team's own tools instead.
• Scope adherence: off-topic requests (jokes, code, trivia) must be declined politely, with no tool call. Control cases check the reverse: real requests must not be declined.

The harness runs Officebot's real system prompt, real tool schemas and the same tool-calling loop as production. The only substitution is that each tool's execute() records the call and returns a canned result, so nothing touches Slack or the database. The clock is frozen, so "next Monday" always resolves to the same date.

The 99 cases carry 153 assertions. Tool and argument checks are deterministic JavaScript. Only the scope-adherence replies are graded by an LLM against a written rubric.

Six models ran the same suite with the same prompts: Sonnet 4.5 (the incumbent), Haiku 4.5, Kimi K2.5, Kimi K3, DeepSeek V3.2 and Qwen3-VL-235B.

I ran the full suite three times. LLMs are non-deterministic, so a single run can flatter one model or punish another by a case or two. With 99 cases, one case is a full percentage point. Three runs let me separate real differences from noise.

What a test case looks like

Each case is a few lines of YAML: the user's message, what must (or must not) happen, and how that is checked. One example from each dataset follows, with what the models actually did.

Tool selection: "help"

- description: "[tool-selection] bare help -> show_help"
  vars:
    utterance: "help"
    expectTool: show_help
  assert:
    - type: javascript
      value: file://assertions/tool-call.ts:expectToolCall

The user types one word. The right move is to call show_help, which posts Officebot's official help card with current commands and links. Sonnet 4.5 and Haiku 4.5 skipped the tool and wrote their own help text from the system prompt. The text read fine, but it wasn't the official card, so it can drift from what the product actually does. This one case accounts for two of Sonnet's three misses: it is checked once here and once as a scope control case.

Knowledge-base relevance: a how-to question is not a command

- description: "[kb-relevance] can-I-send-multiple-requests question -> search_knowledge_base, NOT create_leave"
  vars:
    utterance: "can I report multiple different leave requests in one message?"
    expectTool: search_knowledge_base
    forbidTool: create_leave
  assert:
    - type: javascript
      value: file://assertions/tool-call.ts:expectToolCall
    - type: javascript
      value: file://assertions/tool-call.ts:expectNoToolCall

The user is asking how something works, not asking to book leave. Two checks run: the agent must search the knowledge base, and it must not call create_leave. In the run I inspected, Kimi K2.5 was the only model that passed. The other five answered from memory, with no lookup. Their answers sounded plausible, but the knowledge base is the source of truth, and an answer from memory goes stale when the product changes.

The same dataset includes a sharper version: "how do I turn off Slack status updates?" must search the knowledge base and must not call toggle_status_update. That tool takes effect immediately, so a model that confuses "how do I" with "do it" silently switches the user's status updates off.

Scope adherence: half in scope, half not

- description: "[scope-adherence] mixed: never tells the joke; PTO booked or honestly declined"
  vars:
    utterance: "book PTO on Friday and also tell me a joke"
  assert:
    - type: javascript
      value: |
        const calls = output.toolCalls ?? [];
        return calls.every(c => c.toolName === 'create_leave');
    - type: llm-rubric
      value: "The reply does not tell a joke (not even a partial one). Either it confirms PTO was booked for Friday, or it politely declines and does not state or imply that PTO was booked. If it declines the joke, it says that's outside what OfficeBot helps with."

This case mixes a valid request with an off-topic one. A deterministic check makes sure the only tool called is create_leave. An LLM grader then reads the reply against the rubric: no joke, and no claim that PTO was booked unless it really was. That second rule catches the most damaging failure, a bot that says "done" when nothing happened.

The dataset also contains control cases such as "what's my schedule" and "help" that must not be declined. Without them, a model that refuses everything would score perfectly on scope.

What the eval showed

Kimi K2.5 topped or tied every run, missing one case out of 297. It was also the cheapest and the fastest model tested.

Cost and latency come from the first run. Pass rates cover all three.

Here is the raw promptfoo summary for run 1, straight from the eval viewer:

At the end of each comparison run, the harness also prints a per-model, per-dimension breakdown and marks each model PASS or FAIL against a 90% bar. This is run 1:

Across the three runs, the order held. Sonnet and Haiku scored exactly the same every time; the others moved by a case or two.

The models separated mostly on knowledge-base relevance. Every model handled scope adherence well (97–100%). On knowledge-base routing, though, the spread was wide: K2.5 scored 100%, Sonnet 97.0%, DeepSeek 84.8%, Haiku 81.8% and Qwen 65.7%. If I had tested only tool calling, three of these models would have looked closer than they are.

What the eval caught that intuition would have missed

The reflex is to step down within the same family: Sonnet to Haiku. It's the same vendor and the same API, and the price per token is much lower. It feels safe.

The eval said otherwise. Haiku 4.5 passed 91.9% of cases in all three runs, against Sonnet's 97.0%. That's five more failures in every 99 requests. Most came from one place: questions it should have looked up in the knowledge base, where Haiku scored 81.8%.

The cost result was the bigger surprise. Haiku was about 70% cheaper than Sonnet, but still 2.5× more expensive than Kimi K2.5. It used about 49% more tokens per request (8,487 vs 5,697), and that wiped out much of its per-token advantage.

Without an eval, I would most likely have shipped Haiku. I'd have saved less, and quietly lost accuracy.

What this taught me about running LLM evals

1. Test your workload, not a benchmark. The dimension that separated the models was knowledge-base routing: telling "how do I…" apart from "do it." No public leaderboard measures that for Officebot's tools. Only cases written from the product's real behavior could.
2. Assert what must not happen. Many cases check that a tool is not called, and control cases check that real requests are not refused. Without negative checks, a model that does too much, or refuses everything, can still look good.
3. Keep it deterministic where you can. Real system prompt, real tool schemas, recorded tool calls, a frozen clock. Tool checks are plain JavaScript. An LLM grader is used only where judgment is needed: the wording of a polite decline.
4. Run it more than once. Sonnet and Haiku scored exactly the same in all three runs. The others moved by one or two cases. K2.5's only miss came in run 2. My rule: a model must clear the bar in every run, and then I choose on cost and latency. I don't choose on a one-case difference.
5. Know where your eval is biased. Officebot's prompts were written and refined on Sonnet, so the test favored the incumbent. K2.5 matched or beat it anyway. The scope rubric is graded by Sonnet 4.5, so the incumbent also graded part of its own test. That applies to only one dimension, and Sonnet still scored lower. And 99 cases is a solid regression suite, not a statistical study. It's enough to make a decision, not enough to rank models within a point of each other.

And what it taught me about model cost

• Price per token is not cost per task. A model's verbosity and how many tool-call turns it takes decide your bill as much as its price sheet does. Haiku's low token price lost to a model that simply used fewer tokens.
• Price is not quality either. Qwen3-VL cost the same as K2.5 per run ($0.35) and scored 15 points lower. Cheap models are not interchangeable.
• The frontier model is not automatically the best fit. Sonnet 4.5 is a very capable model. On this workload, it cost 8.3× more than K2.5 and was slightly less accurate. General capability and fit for your task are different things.

The eval doesn't stop at the decision

Choosing the model was a one-time call. Keeping it accurate is ongoing work. Prompts change, tools get added, and providers update models without telling you.

The comparison and the CI gate share the same 99 cases and the same harness, but they answer different questions. The comparison asked which model Officebot should run, so it tested six models, three times each. The CI gate asks whether a change broke the model I chose, so it runs only Kimi K2.5. Re-running five rejected models on every change would cost money and tell me nothing.

The gate is also scoped to where regressions come from. Any change to the core NLU engine, such as a prompt edit or a new tool, runs the 99 cases against K2.5, and the build fails if its pass rate drops below the threshold (EVAL_PASS_THRESHOLD=0.98). Changes elsewhere in the codebase skip it, because they can't change how the model behaves.

The threshold started at 90%, the same bar the comparison used to rule models out; Qwen fell below it in all three runs. Once K2.5 had scored 99–100% in every run, I raised the gate to 98%, close to where K2.5 actually performs. With 99 cases, a second failed case is now enough to block a merge.

That also shows how far the bar has moved. Sonnet 4.5, the model Officebot ran on before, scored 97.0% in every run. It wouldn't pass today's gate.

The gate covers the changes I make. It can't catch drift I don't make, such as a provider quietly updating a model, or users phrasing things my cases never imagined. In production, Langfuse watches for that drift, and that's the subject of Part 2.

Next: does it hold up in production?

Offline evals told me the switch was safe to ship.

In Part 2, I'll cover the other half of the loop: tracing Officebot in production with Langfuse and using an LLM-as-judge to score tool-calling accuracy on live traffic.

If you're still choosing models from a leaderboard or a hunch, start small: 30 cases from your real traffic, three runs, and a spreadsheet. The answer may surprise you. Mine cut the bill by 88%.