33% to 90% — What Was Actually Broken
The benchmark said 33%.
Random guessing across six categories lands at 17%. Triage was barely above that.
I thought the prompts were the problem. They weren't — not entirely.
What triage does
Every message that comes into Pepper passes through triage first.
small_talk, recall, task, lookup, capability, cascade — six categories. One word out, and the rest of the system follows from it. Get it wrong, and everything downstream is wrong.
In EP 10, triage hit 94% on 50 hand-written test cases. That felt like a solid result.
Then I wrote 500 test cases. And built a proper benchmark runner.
33%.
Somewhere between 50 cases and 500, the illusion broke.
The black hole
The failure pattern was immediately obvious.
small_talk → 98% ✅ recall → 23% task → 13% lookup → 19% capability → 12–28%
small_talk was absorbing everything. 322 out of 499 cases that should have gone somewhere else — went to small_talk.
"What's the weather today?" → small_talk. "When is dad's birthday?" → small_talk. "Add dentist appointment Tuesday 3pm" → small_talk.
The model wasn't classifying. It was defaulting.
The first attempt: rewrite the prompt
The obvious move was to fix the prompt. I rewrote it completely — a proper decision tree, priorities from 1 to 5, explicit rules for every edge case, ⚠️ small_talk is the last resort at the top.
Before: 33%. After: 31%.
Worse.
The reason turned out to be the model. Gemini Flash-Lite gets worse with longer prompts. More structure, more rules — more confused. The prompt was fine. The model couldn't follow it.
So I switched the model
Flash-Lite → Flash. A meaningful step up.
Before: 31%. After: 32%.
One percent.
The model wasn't the problem either.
Payload
At this point, I stopped guessing and looked at what was actually being sent to the API.
model: gemini-2.5-flash thinkingBudget: 0 ← thinking completely disabled temperature: unset ← Gemini Flash default = 1.0
Two things.
thinkingBudget: 0 was left over from Flash-Lite, which has no thinking capability at all. The code never got updated when the model changed.
temperature: 1.0 is the default. For a classification task, 1.0 means the model is effectively picking at random. Every request, a different roll of the dice.
I'd been evaluating prompt quality with a broken thermometer.
temperature: 0
One change.
Before: 32% After: 49%
Seventeen percentage points. No prompt changes. No model changes. Just telling the model to stop guessing.
The benchmark confirmed what the pattern was already suggesting: recall went from 17% to 100% instantly. The prompts were fine. The model just wasn't being asked to be deterministic.
Korean prompts, English model
49% was better. Still not good enough.
The next thing I tried was changing the prompt language from Korean to English. Same rules, same examples, same structure — just translated.
Before: 49% After: 63%
task went from 33% to 100%. On a translation.
This one surprised me more than the temperature fix. The content didn't change. The logic didn't change. But the model's ability to follow the instructions changed dramatically.
Every LLM — including Gemini — is trained primarily on English data. Instruction-following, classification, decision trees — these capabilities were built in English. A Korean instruction set asks the model to translate before it can reason. Something gets lost in that step.
The principle that came out of this: model instructions go in English. User-facing output stays in Korean. Examples can be mixed. The language of the prompt and the language of the response are separate decisions.
After this, I went through every other LLM call in Pepper and translated all system prompts to English. slot-filler, extractor, toss.transfer, summary-cache, tools router — all of them.
Two pipelines
One thing the benchmark exposed: triage accuracy and capability matching are different numbers.
triage says "this is a capability request." matcher then determines which capability — calendar.create_event, naver.shopping, toss.transfer, and so on.
A 90% triage score means nothing if matcher is broken.
At this point, capability_cal_list was at 0%. capability_shopping was at 0%.
I assumed it was the matcher prompt. I rewrote the capability descriptions, added trigger keywords, expanded examples. Nothing moved.
Then I looked at the raw response.
model output: {"match":"naver.shopping"} code result: null
The model was right. The code was wrong.
Parsing bug
The matcher was stripping markdown code fences before parsing JSON.
// before const cleaned = result.text.replace(/^```json\s*|\s*```$/g, '').trim()
The model sometimes returns:
{"match":"naver.shopping"} The user is asking for the cheapest AirPods...
The regex strips the opening fence but leaves the trailing explanation. JSON.parse fails. Returns null. Silently.
// after const block = result.text.match(/```json\s*([\s\S]*?)```/) const cleaned = (block ? block[1] : result.text).trim()
Extract the code block content directly. Don't try to strip fences from the outside.
triage.ts had the same pattern. Fixed both.
"ALWAYS capability"
After the parsing fix, capability was still partially failing.
The prompt had the right rules. The model wasn't following them consistently. So I added one line to the capability section:
If the user message contains ANY action verb below, it is ALWAYS capability — never small_talk.
Explicit override. No wiggle room.
capability_toss: 50% → 100%.
capability_cal_create: 25% → 100%.
capability_shopping: 0% → 100%.
One sentence.
Temperature across the whole system
After fixing triage, I checked every other LLM call in the codebase.
The pattern that emerged:
generateWithFallback (single prompt) → gemini.call() → temperature: 0 generateMessagesWithFallback (messages) → gemini.callMessages() → temperature: unset claude.call() → temperature: unset
Two more fixes.
Classification and extraction tasks: temperature 0. Deterministic output for deterministic questions.
Response generation tasks: temperature 0.7. Pepper's replies need to feel natural, not robotic.
The triage temperature bug wasn't unique. It was everywhere.
The final numbers
TRIAGE ACCURACY 186/207 (90%) CAPABILITY MATCH 54/54 (100%)
Full breakdown:
small_talk 98% recall 90% task 100% lookup 73% ← still the weak point capability_* 83–100% cascade 75%
lookup at 73% is the remaining gap. Real-time information requests at the boundary — "is it raining?" reads like small_talk to the model even after all these fixes. That's the next session.
What actually happened
A clean version of the timeline:
33% → start 49% → temperature: 0 (+17pp) 63% → English prompt (+14pp) 81% → capability section + fixes (+18pp) 90% → parsing bug + policy fixes (+9pp)
But the honest version looks different.
The prompt rewrite made things worse. The model upgrade did almost nothing. "JSON only" added to the prompt caused a full regression — 63% back down to 32% — because that single sentence disrupted how the model was reasoning through the classification. Rolled it back. Then forward again.
The fixes that worked weren't prompt engineering in the way I expected. They were:
- Check the payload. Find out what's actually going out.
- Check the raw response. Find out what's actually coming back.
- Change one variable at a time. Know what each change did.
Payload, raw response, isolated changes. That's the actual work.
90% on 500 synthetic cases. The real number will reveal itself in use. And that "in use" was arriving sooner than I expected.