Compact retrieval as infrastructure repurposing — and the headline I had to retract

Context

Enterprise retrieval has to preserve the operating system around it — identity, authorization, evidence, citation lineage. Inside that boundary sat a narrower, more useful question: can a compact vector index hold a large evidence corpus inside a CPU and memory budget that a float32 index cannot fit, without giving up retrieval quality?

Decision

Treat retrieval quality and end-to-end speed as two separate claims that are routinely conflated, and refuse to publish either one without a control pipeline matched on every variable except the one under test.

Approach

• Discovered the first comparison was not apples-to-apples: the low-bit arm had keyword search, fusion, a precision recheck, and a reranker stacked on it while the float32 baseline had none of them. Rebuilt the baseline with the identical pipeline so vector storage format was the only remaining variable.
• Reproduced a published 2025 result showing an unrepaired quality cliff, then reintroduced each repair stage one at a time to establish which stage was actually doing the work — rather than attributing recovery to the compression format.
• Stress-tested the mechanism instead of the headline: 128 dimensions, four rerankers including a deliberately untrained keyword-only rescorer, four index formats, and compression levels past the one being advocated for.
• Scaled the quality claim to a 500,000-passage corpus with 6,980 judged queries, using paired statistical tests so small differences were not read as real ones.
• Re-derived every stated number against raw per-query data before it was allowed to become a finding.

Outcomes

• Quality held: compact indexing matched the matched-pipeline float32 comparator across every corpus tested, including 99.92% retention at the largest scale with a confidence interval crossing zero.
• The memory result was the real one: a roughly one-million-vector index that a float32 build could not fit inside an 8 GB CPU envelope at all, fit with over 7.8 GB free — changing which existing machines can host the service without new hardware.
• Retracted our own headline. Fixing the pipeline asymmetry moved the end-to-end speed claim from a 2.38× win to 1.04× — a statistical tie that missed our own threshold for a meaningful result. It was published as a tie.
• Named the mechanism honestly: the reranker consumes roughly 93% of request time, which makes it both the reason compact retrieval is safe to use and the reason the speed win disappears. No tested configuration decoupled the two.
• Narrowed the recommendation against our own interest: the marketed format is not the one carrying the result — a simpler, more established 8-bit format delivers nearly all of the same benefit, so the aggressive option is only warranted when the extra memory reduction is genuinely the binding constraint.
• Refused to let an adjacent claim borrow this one's credibility. The same program separately tested ternary and low-bit language models for bounded local work, and no tested configuration won on both cost and quality at once. That strand is reported as an open negative in the brief rather than folded into the retrieval headline — the two questions share a program, not a result.

Control boundary

Co-authored with Pawel Zaloga; the manuscript is unpublished and the underlying corpora, per-query data, and evaluation harness stay private. Limits stated in the brief are restated here rather than smoothed over: one vector engine, an emulated four-vCPU/8 GB shape rather than a real cloud instance, and no independent audit of the corrected pipeline yet. Nothing here is a claim about production deployment.

What this taught me

The compression ratio was not the finding. The finding was that a plausible-looking comparison had an asymmetry hiding inside it, and that catching it cost us the number we most wanted to report. A result that survives its own fair control is worth more than one that needs the framing to hold it up.