Reuse what you already have. A compact vector index lets a search service run on modest, existing CPU machines that a conventional float32 index simply can't fit on. That's the real value here — not speed, not a quality breakthrough, just infrastructure you don't have to buy.
Most AI systems that answer questions first search a pile of documents, then hand the best passages to a model as evidence. To search by meaning rather than exact wording, the documents get converted into embeddings — long lists of numbers. A large collection of ordinary 32-bit embeddings eats memory fast. Binary embeddings keep only the sign of each number, positive or negative, which shrinks the index dramatically.
The pitch writes itself: go binary, get the same answers, spend a fraction of the memory. That's the claim we set out to test. What we found is narrower and more useful than the pitch: the savings are real, but they come from memory and deployment flexibility, not from binary storage making search better on its own. Quality comes from the whole pipeline around it.
Three questions:
Then we tried to break it: noisy queries, a much larger corpus, five embedding models, four rerankers, and increasingly aggressive compression.
Our first comparison wasn't fair. The binary system had keyword search, a fusion step, a precision recheck, and a reranker running on top of it. The float32 baseline had none of that. Unsurprisingly, binary looked better for reasons that had nothing to do with binary storage — on NFCorpus, the early numbers even showed binary beating float32 outright (0.3715 vs 0.3435, a 108% reading). It wasn't a win. It was two systems doing different amounts of work.
We fixed it by giving float32 the identical pipeline — same embedder, same keyword search, same recheck, same fusion, same reranker. The only remaining variable was how vectors were stored. That single fix moved our headline speed number from a 2.38× end-to-end win down to 1.04× — a statistical tie.
One implementation note, so a re-measured number doesn't read as a mystery: the keyword-search backend was swapped mid-project (the original couldn't scale to our largest corpus), and every figure here was re-measured under the replacement. The reranker's share of total request time moved from roughly 70% to roughly 93% as a direct result of that fix — not from anything about binary vs. float32.
General lesson for anyone reading this as a methods example: when two pipelines differ in more than the one variable being tested, the result can't tell you which difference caused the outcome.
Four steps: cast a wide net with binary or float32 vector search; recheck the top candidates at higher precision; run keyword search in parallel and merge the two result lists; rerank the merged candidates.
We compared four index formats — binary, float32, int8, and product quantization (included specifically to check whether the result is unique to binary). Quality was tested on three corpora — SciFact (300 queries), NFCorpus (323 queries), FinanceBench (150 queries) — plus a 500,000-passage slice of MS MARCO with 6,980 judged queries, to see whether the result held at real scale. Every comparison used paired statistical tests so small differences weren't mistaken for real ones.
With everything else held constant, binary matched float32 on every corpus tested:
| Corpus | Binary | Matched float32 | Reading |
|---|---|---|---|
| SciFact | 0.7643 | 0.7643 | Bit-identical rankings |
| NFCorpus | 0.3699 | 0.3699 | 100.02% retention — no meaningful difference |
| FinanceBench | 0.7279 | 0.7153 | Binary read higher, but n=150 is too small to call it (CI crosses zero) |
The strongest number in the whole study lives outside that table. On the 500K-passage MS MARCO slice — 6,980 judged queries, roughly 100× the query volume of anything else here — binary hit nDCG@10 0.7962 against float32's 0.7968: 99.92% retention, with a confidence interval that includes zero. This is also the only place quality and scale were tested on the same embedding model rather than on two separately-measured setups — closing what had been the single largest open gap in the earlier drafts.
The headline finding here is that binary storage isn't carrying this alone.
We reproduced the setup from a 2025 published study (Huerga-Pérez et al.) — binary search, no repair stages — and got their result: quality fell 9.3–18.0% across four tests, a real and statistically clear cliff. Then we added the repair stages back one at a time. Full-precision queries recovered a big share of the loss. An 8-bit precision recheck recovered most of the rest. Keyword fusion plus reranking closed nearly all of what remained.
Binary search is a fast, cheap way to gather plausible candidates. The pipeline stages after it are what make those candidates trustworthy.
Three stress tests on that claim:
The one embedder that didn't fully recover — e5-base-v2, at 94.59% retention on NFCorpus — gets its own explanation in §10.
The memory result is simple. A roughly one-million-vector, 3072-dimension index, tested under an 8 GB CPU limit:
| Format | Memory | Fits in 8 GB? |
|---|---|---|
| Float32 | 11,707 MB | NO |
| Int8 | 2,927 MB | YES |
| Binary | 366 MB | YES |
| Product quantization (same 32×) | 366 MB | YES |
Float32 couldn't run at all. Binary and PQ leave 7.8+ GB free — enough to change which existing machines can host the service without a new high-memory purchase.
Timed in isolation, binary search was clearly faster — 559.2 searches/sec vs. 234.7 for float32, a 2.38× win at similar recall.
But nobody uses an isolated lookup. Once candidate rechecking, keyword search, fusion, and reranking are counted, the gap was 65.55ms for float32 vs. 63.08ms for binary — a 1.04× improvement, about 4%, and it missed our own 1.2× bar for a meaningful win. The reranker alone eats roughly 93% of request time; shaving the remaining 7% barely moves the total.
This is evidence about compact retrieval — not evidence that a true binary neural network can replace a general language model. The broader program keeps three separate questions apart: can compact vectors store and search enterprise knowledge efficiently (yes, under these conditions); can low-bit or ternary language models handle bounded tasks more cheaply than a strong hosted model (current tests show no win on both cost and quality at once); and can a true BNN — binary weights and activations, packed-bit runtime — run an enterprise workflow safely (untested here).
This is a careful retrieval study, not a universal proof. Latency numbers come from one vector engine (Qdrant) on an emulated four-vCPU/8GB shape — not a real cloud instance, not cross-validated on a second engine. FinanceBench has only 150 queries and uses extracted passages rather than full filings.
On the e5-base-v2 shortfall: we found a measured correlate, not a guess. Roughly 62% of its output dimensions are strongly one-signed across the corpus — about double the next-worst model tested, consistent on both corpora checked. A competing explanation (values simply collapsing toward zero) was tested directly and ruled out. That's a mechanism awaiting a fix, not an open mystery.
Before any formal submission: pin model versions, refresh the dependency lockfile, export figures in the venue's preferred format, and anonymize the repository if blind review is required.
Under the conditions actually tested — one vector engine, emulated four-vCPU/8GB hardware, the corpora in §4, and a fair pipeline on both sides — compact indexing (binary, and comparably int8) keeps a large index inside a memory budget float32 can't touch, without a measurable quality penalty on most tests. That's real, and it's scoped to what was measured. It is not yet a claim about a second engine, real cloud hardware, a bigger financial-domain set, or a fully-fixed e5-base-v2 — those stay open questions, not fine print.
The mechanism is the whole story. Quality belongs to the complete pipeline — broad retrieval, precision recheck, keyword evidence, reranking. Speed belongs to the same pipeline, and that's exactly why the gain is modest: reranking dominates the time budget, and reranking is also what makes the quality result trustworthy in the first place.
Compact retrieval is a practical way to repurpose infrastructure you already have. It's not a substitute for good ranking, good systems design, or careful evaluation — and it shouldn't be sold as one.
Every correction below was re-checked against the paperA/paperA_data archive before going in. §6 separated an unrelated embedder-vs-reranker attribution test (source of a 41/59 split wrongly cited there before) from this brief's own repair story, and added three previously-missing results: the 128-dim floor test, the four-reranker panel, and the float-format panel. §5 gave the 500K-passage scale test its own paragraph. §9/§12 disclosed N=15 behind the GPU-reranker comparison and corrected a stale figure it was quoted against (0.8607/0.7284 → 0.8753/0.7279). §10 restated the e5-base-v2 shortfall as a measured, partially-explained mechanism. New citations named the paper this study replicates (Huerga-Pérez et al.) plus three vendor sources. The debate-council cost line was moved to an appendix in v4; in this v5 readability pass it's been dropped entirely — a $0.523 internal-tooling cost note has no business in a client-facing brief, per Pawel's own review of v3, which flagged this and was only partially acted on.