A runtime configuration I promoted, and the held-out run that took it back

Context

Reasoning modes make a local model better at hard questions and much slower at everything else, because the reasoning competes with the answer for one finite generation budget. Routing is the obvious fix: turn reasoning on only for the categories that need it. Measured against the cumulative evaluation ladder the suite had been developed on, the routed variant won outright — a clean sweep at roughly a quarter of the latency of always-on reasoning. That is a result worth promoting, and it was promoted.

Decision

Treat the ladder result as fitted rather than validated, and build the held-out stage as a separate artifact. The routing boundary and the cases that produced it were entangled — the category list had been tuned while looking at those exact cases — so testing the configuration there is fitting to the test set with extra steps.

Approach

• Ran the routed variant against a reasoning-disabled control on the cumulative ladder: 16 of 16 at about 29.8 seconds, against always-on reasoning at 15 of 16 and about 118 seconds. Better and roughly four times faster.
• Declined to validate the routing boundary on the suite that had produced it, and implemented a genuinely held-out 27-case challenge stage as its own harness version — deliberately not a longer version of the ladder.
• Held a standing rule against widening the routing categories because a new category sounded difficult. Every pre-emptive adjustment spends part of the only unfitted evidence available, and that evidence cannot be recovered once spent.
• Ran the routed variant first and the control second, preserving both runs regardless of outcome, so the comparison could not be quietly re-run until it agreed with the promotion that had already happened.
• Compared failures by category rather than by aggregate score, which is the only view that can distinguish a configuration that is slightly worse everywhere from one that is catastrophically wrong in one place.
• Reconciled the result against an earlier finding from my own work that pointed the other way, instead of leaving two contradictory conclusions on the record and letting the newer one win by recency.

Outcomes

• On the held-out suite the control won: 22 of 27 for reasoning-disabled against 21 of 27 for the routed variant — and it did so at 17.8 seconds against 160.4 seconds, roughly a ninth of the latency. The routed configuration lost on accuracy and cost nine times as much to run.
• The per-category view located the mechanism rather than just the loss: the routed variant scored 0 of 4 on long-context cases where the control scored 4 of 4. An aggregate one-point gap concealed a total failure in one category.
• The generalization was dropped and the narrower measured claim kept — the ladder result stands for labeled inputs on that suite, and nothing further.
• A latent defect in the router surfaced during the analysis: a keyword cue could switch reasoning on but never off, and it was evaluated disjunctively with the reliable declared-category signal, so the unreliable half silently wins any disagreement. Its measured impact on that run was zero, and it is recorded as latent rather than quietly presented as a near-miss.
• Reconciled against an earlier finding where routing was worse than running every stage unconditionally. Both results are correct, and the separating conditions are mechanical: whether the routing signal is declared or inferred, and whether the unconditional paths compete for one finite budget or simply add work. That reconciliation is more reusable than either result alone.
• A corrected variant exists and has not been promoted. Promotion requires a further separate held-out revision, because the correction was itself motivated by the held-out run.

Control boundary

Own hardware, own harness, public and synthetic evaluation content, no client or employer data. Every figure here is a single run on a small sample and is labelled as such — the point of the case study is the method and the reversal, not the precision of the numbers. The reversed claim remains published in my own records rather than removed, because a corrected result is only verifiable if the thing it corrected is still visible.

What this taught me

The ladder result was not wrong. It was over-generalized, which is a failure mode that feels nothing like being wrong — the number is real, reproducible, and still true of the thing it was measured on. What killed the promotion was noticing that serving this configuration on production traffic would require inferring the case category rather than being told it, which lands straight back in the brittle-signal regime that had already sunk an earlier routing attempt. The held-out suite did not reveal a mistake in the measurement; it revealed that the measurement had been answering an easier question than the one deployment would ask. That is worth more to me than the win would have been, and it is the second time in a year that building the fair comparison cost me the headline.