Turbovec and the trick of not training a quantizer

A corpus of 10 million embeddings takes about 31 GB of RAM to hold in float32. Turbovec holds the same corpus in roughly 4 GB, and answers nearest-neighbor queries about as fast as FAISS does. Faster on ARM, if you believe the benchmarks. That is the pitch. The interesting part is how it gets there, because it skips a step that every product-quantization index I have used treats as mandatory.

Turbovec is a vector index written in Rust with Python bindings. Under the hood it implements TurboQuant, a quantization algorithm out of Google Research. TurboQuant was first framed for compressing the KV cache during LLM inference; turbovec applies the same math to stored embeddings. You use it the way you’d use FAISS or pgvector: it holds your embeddings and returns the closest ones to a query. The models and the application code around it don’t change. What changes is the memory footprint and the operational story.

The step it skips

Picture compressing a huge photo library. Instead of keeping every photo at full quality, you build a small set of reusable pieces and store each photo as a short recipe: use piece 12, piece 47, piece 3, piece 88. The recipe takes far less room than the photo.

A vector is just a long list of numbers that stands for a document, an image, or a sentence, and millions of them eat memory the way full-quality photos do. Product quantization compresses them the same way. It takes a sample of your vectors, runs k-means to find the cluster centers (the centroids), and writes those centers into a codebook, a dictionary of common pieces with a number on each. Every vector then gets stored as a handful of those numbers instead of the original list. [0.142, -0.883, 0.551, 0.019, ...] becomes something like [12, 47, 3, 88]. Much smaller.

It works well. The catch is that the codebook has to be learned first, and it is only as good as the sample it came from. Train it on school essays, then fill the database with medical articles, legal contracts, and code, and the old pieces stop describing the data. That is data drift. When it gets bad enough, recall degrades: the search starts missing matches it should return. The fix is to retrain the codebook, recompress every vector, and rebuild the index. For a corpus that grows all day, that is a recurring chore.

TurboQuant does not train a codebook. The authors call it “data-oblivious”: the quantizer is derived from math, not from your data. It works in four steps, and none of them looks at your data.

First, normalize. Each vector’s length is stripped off and stored as a single float, and only the direction gets quantized, a unit vector on a sphere.

Then rotate. Every vector is multiplied by the same random orthogonal matrix. The rotation is isometric, so it preserves every inner product and distance; geometrically nothing has moved. It doesn’t need a dense matrix multiply either: a structured transform like a randomized Hadamard gives the same effect in O(d log d), which keeps the encode step cheap. But after the rotation, each coordinate independently follows a shifted, scaled Beta distribution that converges to a Gaussian in high dimensions, whatever the input data looked like. Raw embeddings have skewed coordinates and outlier dimensions that eat up dynamic range. The rotation scrambles those away. (If these two steps feel dense, there is a plainer walkthrough in the appendix.)

That is the whole trick. Once you know the distribution of each coordinate ahead of time, you can precompute the optimal quantization buckets (a Lloyd-Max scalar quantizer) without ever looking at your data. For 2-bit precision, each coordinate goes into one of four buckets whose boundaries were worked out in advance. Then bit-pack: a 1536-dimensional float32 vector is 6,144 bytes, and at 2 bits per coordinate it comes out to 384 bytes. That 16× is the codes alone; the stored norm and the residual below add a little on top, so a real index compresses somewhat less.

Two refinements keep the compression from hurting accuracy. Scalar quantization makes reconstructed vectors slightly shorter, which biases inner products downward; turbovec computes a correction scalar at encode time to cancel the shrinkage. And the residual left after quantization gets a 1-bit QJL transform, which keeps the score ordering unbiased for about one extra bit per coordinate. (This step too has a plainer walkthrough in the appendix.) That is the TurboQuant-PROD variant, tuned for unbiased inner-product estimation at query time; a second variant, TurboQuant-MSE, minimizes mean-squared error and handles the pairwise scoring while the index is built. The result stays within about 2.7× of the Shannon lower bound for that bit rate, close to the theoretical best at this compression.

What you get for skipping training

No training means no retraining. You add vectors the moment they arrive and never stop to rebuild as the distribution moves. For an index that ingests continuously, that removes a whole category of operational work: no drift monitoring, no periodic reindex, no codebook that goes stale.

The speed comes from hand-written SIMD kernels: AVX-512BW on x86, NEON on ARM, with nibble-split lookup tables and runtime CPU feature detection so each machine takes its fastest instruction path. The ARM path is where turbovec claims to beat FAISS’s IndexPQFastScan, by something like 12 to 20 percent. I’d treat that number gently; it comes from the author’s own posts, not an independent benchmark. But the architecture explains why it’s plausible, and ARM is exactly where you’d want this: Apple Silicon laptops, ARM servers, edge boxes running semantic search without a GPU.

The library is local and nothing else. A Rust crate and a Python package, no managed service, no network call. Pair it with a local embedding model and a local LLM and the whole RAG pipeline stays on the machine, which matters when the documents can’t leave.

Using it

After pip install turbovec, you initialize an index with a dimension and a bit-width, add vectors as they come, and query for approximate nearest neighbors. Two index types ship: a plain TurboQuantIndex that hands out sequential integer IDs, and an IdMapIndex that maps to stable external uint64 IDs and survives deletes and updates with O(1) removal, which is the one you want in production. Filtering (id allowlists or bitmasks) happens inside the kernel rather than as a pass afterward. There are extras for the usual orchestration frameworks: turbovec[langchain], and the same for LlamaIndex and Haystack, so it drops into a normal Python stack without you writing an index abstraction.

What to check before you trust it

Near-optimal distortion is a statement about geometry, not about your retrieval task. You still have to measure recall on your own data, because how much the quantization hurts depends on your embedding model and your corpus, and the only way to know is to run it. Two bits maximizes compression and costs you the most recall; four bits buys fidelity back for twice the memory. Start at the bit-width your recall target allows, not the one with the best compression number.

It is still ordinary vector search underneath. Smaller and cheaper, but it finds neighbors by similarity. It won’t connect facts across documents or do multi-hop reasoning, and the compression doesn’t change that. If your retrieval needs structure, turbovec is the wrong layer to ask.

And it is young. The math behind TurboQuant is solid and the paper is real, but the library is new: thinner docs than FAISS, a smaller community, and the rough edges that come with both. If you need something proven across a thousand production deployments, it isn’t that yet. If you need to hold 10 million embeddings in 4 GB on a laptop with no GPU and no service to call, there isn’t much else that does it.

Appendix: normalize and rotate, explained simply

The two steps in the section above, without the math vocabulary.

Normalize: separate “how long” from “which way.” Think of each vector as an arrow. Every arrow has two properties: how long it is and which direction it points. The direction carries most of the meaning, so the trick is to split them apart. Write down the arrow’s length as one single number and set it aside, then shrink or stretch the arrow so it is exactly length 1. Now every arrow is the same length, and the tip of each one sits somewhere on the surface of a sphere, like points on a globe. Only the direction needs the heavy compression; the length is already saved cheaply as one number.

Rotate: spin everything the same way. Next, spin the entire globe by some random amount. Every arrow gets the same spin. The key is that spinning a globe changes nothing that matters: the distance from New York to Tokyo is the same whether the globe is spun or not, and the angle between any two arrows stays the same too. That is what “isometric” means; the rotation moves the labels around but preserves all the geometry, so search results don’t change at all. There is also a speed shortcut hiding in there. Doing the rotation the obvious way means a huge multiplication for every vector, but the Hadamard transform gets the same spinning effect much faster, like taking a highway instead of every back road.

So why bother spinning, if nothing changes? This is the actual magic. The geometry doesn’t change, but the individual numbers inside each vector do. A vector is a list of slots, each holding one number. Before the spin, those numbers are messy and unpredictable: one slot might always carry huge values, another tiny ones, with weird spikes, and you’d have no idea what to expect without studying your specific data.

The spin fixes that because of how rotation works under the hood: each slot’s new number is a blend of all the old ones, a little of slot 1, a little of slot 2, a pinch of every other slot, stirred together like a smoothie. Blends like that behave in a very predictable way. Roll one die and every face from 1 to 6 is equally likely. Roll twenty dice and add them up, and the totals pile up around the middle, because an extreme total needs every single die to land high at once, which almost never happens. Plot how often each total appears and you get a bell curve: tall in the middle, trailing off at the edges. Every slot after the spin is that kind of sum, so its values follow a bell curve too. And because the whole arrow has length 1 shared across hundreds of slots, each slot’s share is tiny, which is why most values sit near zero, the way most students in a class are near average height. This happens no matter what the data looked like before. It is like shuffling a deck of cards: whatever order the deck started in, after a good shuffle you know exactly what kind of randomness you’ve got.

The payoff. Compression works by sorting numbers into a few buckets. To pick good bucket boundaries you normally have to study your data first, which is the training step other systems need. But if a random spin guarantees the numbers always follow the same bell curve, you can work out the perfect buckets once, in advance, with pure math, and never look at anybody’s data at all. That is what “derived from math, not from your data” means.

Appendix: the 1-bit QJL step, explained simply

The residual refinement from the section above, without the math vocabulary.

Buckets are rounding, and rounding leaves leftovers. Sorting a coordinate into one of four buckets is rounding: a true value of 0.37 gets stored as the bucket whose center sits at, say, 0.41, so the stored number is off by a little. That gap between the real number and the bucket’s number is the residual, the rounding leftover. Each leftover is tiny, but a similarity score adds up contributions from every slot, and across 1,536 slots the leftovers pile into enough error to flip the order of results that were close.

Keep a sketch of the leftover, not the leftover itself. Storing the leftovers exactly would hand back all the memory the buckets just saved. So turbovec compresses the leftover even harder, with the same trick as before. Spin the leftover vector with a random rotation, then store one bit per slot: plus if that slot came out positive, minus if it came out negative. Nothing else. That is the 1-bit QJL transform (QJL is short for Quantized Johnson-Lindenstrauss, the piece of math it builds on), and it costs about one extra bit per coordinate.

Why a pile of signs is enough. A single sign tells you almost nothing. But after the spin, hundreds of signs read together still point in roughly the leftover’s overall direction, the way a crowd of compass needles that each only say “north-ish” or “south-ish” still reveal where north is once you count them up. The estimate each sign contributes is noisy, but the noise is fair: it overshoots and undershoots equally often instead of always leaning one way. That is what “unbiased” means.

Why fair noise is the right goal. Search doesn’t need scores to be exact; it needs their order to be trustworthy. An error that always pushes in one direction quietly rearranges close finishes, the way a referee who always leans toward one team changes outcomes. An error that is equally likely to push up or down mostly cancels out across many comparisons and leaves the ranking alone. The 1-bit sketch buys that fairness for one extra bit per slot.

Appendix: FAISS, explained simply

The yardstick this post keeps measuring against, for readers meeting it for the first time.

What it is. FAISS is a free, open-source library from Meta’s AI research team; the name is short for Facebook AI Similarity Search. It does one job: hold millions of vectors and, when handed a new one, return the stored vectors most similar to it, fast. Picture a librarian in a building with millions of shelves who has memorized where everything sits and can pull the ten books most like the one you brought in, without walking every aisle.

Why it is the yardstick. It has been around since 2017, it has been hammered on in production by thousands of teams, and it bundles most of the known ways to index vectors under one roof: exact search that checks everything, graph-based shortcuts, and compressed indexes built on the codebook idea from the step it skips. When a new vector index appears, “as fast as FAISS” is the claim that matters, the way a new sprinter gets measured against the record holder rather than against the school team.

The specific race in this post. The benchmark turbovec quotes is against IndexPQFastScan, FAISS’s compressed index: it uses the trained-codebook compression described earlier, with code tuned so the CPU can score many compressed vectors in a single instruction. That makes it the fair opponent, since both sides compress hard and both use hand-tuned CPU code. The difference this post cares about is that FAISS’s version needs the training step, and turbovec’s does not.