Nothing in the catalogue says "dripping"
A hardware store's search log has a page nobody enjoys reading: queries that led to no click. Near the top this month is something to stop the bathroom tap dripping. The search box sits on a keyword index, which found products whose text contains bathroom and tap. The shopper got basin mixer taps at €80, a bath tap, and a bathroom cabinet "sized to fit under any basin tap".
What stops a tap dripping is usually a new washer, and the store sells a €4 pack described as "replacement rubber washers for leaking taps". Apart from tap, it shares no word with the query, and to an index that compares words, dripping is not leaking. A synonym list could link them, but someone has to think of every pair. Keyword ranking, BM25 and the analyzers that turn text into terms, is Full-text search. This article is the other way to search: by what the text means.
Meaning as a position
An embedding is a vector, a fixed-length list of numbers, that a trained model produces from a piece of text. The store's model turns any text into 768 numbers, and only their geometry matters. Embedding models are trained on large numbers of texts that belong together, such as questions and their answers, to place related texts near each other, so dripping tap and leaking tap land close and dripping tap and bathroom cabinet land further apart.
The store embeds each product's title and description ahead of time and each query on arrival, with the same model, and searches for the product vectors nearest the query's. Nearness is usually measured with cosine similarity, the cosine of the angle between two vectors: 1 when they point the same way, lower as they diverge (some models are meant for the dot product instead; use the measure the model was trained for). Say the query scores 0.82 against the washers and 0.71 against the mixer tap: the washers come first, and nobody wrote a synonym.
Nearness is learned, not checked: tap washer and tap wrench sit close, and the search returns the nearest items it has even when none is what the shopper wants, unless a cut-off such as "ignore anything below 0.6" is set. Scores depend on the model, so that cut-off is tuned per model on real queries.
A gigabyte read per query
The catalogue has 400,000 products. The simplest search, exact or brute-force nearest-neighbour search, compares the query vector with every product vector and keeps the best 100. At 768 numbers of 4 bytes each, that is about 1.2 GB of vectors read for every query. For a few thousand vectors that is fine; at a few hundred searches a second over this catalogue it is hundreds of gigabytes read per second.
An approximate nearest-neighbour (ANN) index examines a small fraction of the vectors instead, and can miss some of the true nearest ones. The trade is measured as recall: of the true 100 nearest, the share the index returned. Every ANN index has a dial between recall, latency and memory. Recommenders make the same trade when they pull candidates (Recommendation systems: retrieval, ranking & cold start). Two designs cover most of what you will meet; PostgreSQL's pgvector extension offers both, as hnsw and ivfflat indexes.
HNSW (Hierarchical Navigable Small World) is a layered graph linking each vector to some of its near neighbours. A search keeps stepping to whichever neighbour is closer to the query, and the size of the candidate list it keeps, often called ef_search, is the dial. It reaches high recall at low latency, but the links cost memory, the graph is meant to sit in RAM, and building it is slow.
IVF (inverted file) clusters the vectors around, say, 1,000 centroids and scans only the lists of the few centroids nearest the query; how many, often called nprobe or probes, is the dial, and a neighbour filed just across an unprobed border is missed. It usually needs less memory and builds faster than HNSW, but its centroids come from the data it was built on, so as new vectors drift from that data, the lists can grow uneven and recall or latency can suffer: measure them, then rebuild when they slip. Where every match counts, measure recall against exact search on a sample of queries before setting the dial.
In stock at the local branch
The shopper has chosen a branch and ticked in stock, and that branch stocks about 4% of the catalogue. Each product vector is stored with metadata, plain fields such as category, price and stock per branch, and the search has to honour them. It can do so in two orders, and each has its own weak spot:
- Filter after the search. The index returns the 100 nearest products, and the service drops those not in stock at the branch. If the 4% is spread evenly among them, about four survive, sometimes none, and the shopper gets an empty page while a tap-repair kit, 140th nearest, sits on a shelf in that branch. Asking for more candidates helps, at the cost of latency, but any fixed cap below the whole catalogue can still leave the page short when the filter is narrow.
- Filter before the search. Search only the products in stock. With exact search over a small subset this works well. Inside an HNSW graph it is harder: matching vectors may be poorly linked to each other, so a search that steps only on matches can stop short and miss some, and one that also steps through non-matching vectors may walk through many to reach them.
Vector databases differ here, filtering during the graph walk, falling back to exact search when few vectors match, or keeping an index per tenant, so check what yours does. Price and stock stay in metadata, updated in place; the vector comes only from the text that describes the product.
The index belongs to one model
A year later a better embedding model comes out. Even at the same length, its vectors cannot be compared with the old ones: each model lays out its own space, and a new-model query against old-model products gets a ranking that means nothing. So every product is re-embedded into a new index that queries move to together with the new model, the build-and-switch that Typeahead & autocomplete walks through, and each vector records its model version. Running the model itself is Model serving: inference, versions & safe rollout.
The same holds for one product: corrected from "fits 15 mm pipe" to "fits 22 mm pipe", its vector encodes the old text until it is embedded again. Edited products are re-embedded, new ones added, discontinued ones deleted; a feed of row changes from the catalogue database can drive that pipeline; Change data capture explains it.
When the shopper types a part number
A plumber types WX-1250, the code on a tap cartridge's box. To an embedding model the code means little; its nearest vectors are often codes that look alike, and WX-1250 itself may miss the top 100, while a keyword search on the SKU field finds it first. Meaning-based ranking can miss an exact hit and keyword ranking can miss a paraphrase, so the store runs both. That is hybrid search: a keyword query and a vector query with the same filters, and the two lists merged into one.
The scores cannot simply be added: a BM25 score has no fixed upper bound, a cosine similarity lies between −1 and 1. One common merge ignores the scores and uses ranks. Reciprocal rank fusion (RRF) gives each product the sum, over the lists it appears in, of 1 ÷ (k + rank), with k usually set to 60. For the dripping tap, the washers, 1st in the vector list and 4th in the keyword list, score 1/61 + 1/64, about 0.0320; the mixer tap, 1st in the keyword list and 12th in the vector list, scores 1/61 + 1/72, about 0.0303. Here, ranking high in both lists beats leading only one.
That is also RRF's blind spot here: WX-1250, 1st in the keyword list and absent from the vector list, scores 1/61, no more than whatever leads the vector list. Stores usually add a rule that pins an exact identifier match at the top, above everything the merge and any later reranking produce. The other common merge adds normalised scores with tuned weights.
The search service embeds the query, asks both indexes with the same filters and merges the lists by rank. The catalogue pipeline re-embeds and reindexes each product that changes.
The merged list is still a first draft: a reranker, a slower model that reads the query and each product together, re-scores the top few dozen. It is the same stage a RAG system uses, and The RAG stack: embed, retrieve, rerank explains it.
That diagram is most of the Semantic Product Search Challenge. What it leaves out are the decisions this article turned on, so write them on your Grid: the embedding model's version in the vector store's title, the candidate count on each edge into the two indexes, and a note saying how the lists merge, what puts WX-1250 first, and whether the stock filter runs before or after the vector search.