Retrieval, vectors, and RAG.
Mostly it is search.

The obvious move is to paste in everything, and for a small handbook that is genuinely the right answer — see when not to bother below. It stops being right when "everything" is ten thousand documents, because you would be paying to send all of them on every question, and because a model handed a haystack answers worse than one handed the needle.

Everything before the question is preparation you do once and repeat when documents change. Everything after it happens in the couple of hundred milliseconds a user is waiting. Knowing which side of that line a piece of work sits on decides most of your architecture.

Chunking is string handling. Storing is a database. Searching is search. Only the embedding and the final answer involve a model at all, and the embedding is a much smaller and cheaper one than the model you are asking. Debug this like the search system it is, not like a mysterious AI problem.

Distance is normally measured by the angle between two vectors rather than the gap between their tips, which is what cosine similarity means. 1.0 is pointing the same way, 0 is unrelated. The practical upshot: a long passage and a short question can still score highly, because length pushes the tip further out without changing the direction much.

Vectors from different models are not comparable. Not slightly — not at all. If you change embedding model, or even its version, every vector you have stored becomes meaningless against new ones. See re-embedding, because this is a migration, not a config change.

The same blindness explains the classic embarrassment: ask "which customers have not paid?" and get back passages about customers who have paid. Negation barely moves a vector. If your questions turn on words like not, except, before or without, pure vector search will quietly let you down.

The tension is the whole game. Small chunks are precise and score sharply, but a chunk that has been cut adrift from its heading may no longer say what it is about — "this is not covered" is useless when the sentence naming what "this" is landed in the previous chunk. Large chunks keep context and dilute meaning.

Overlap is the cheap insurance: repeat the last sentence or two of each chunk at the start of the next, so a fact that straddles a boundary survives in one piece somewhere. It costs storage and nothing else. Ten to twenty per cent is normal.

Put the document title and the section heading at the top of every chunk. It costs a few tokens and repairs most of the orphaned-fragment problem at a stroke, because now the chunk says what it is about even when read alone.

And keep byte offsets on every chunk. When someone asks why the answer was wrong you want to open the source document at the exact place, not go hunting.

HNSW is the usual choice — fast queries and good accuracy, at the price of memory and a slow build. IVFFlat sorts vectors into buckets and searches only the nearest few; it builds quickly and uses less memory, and it is less accurate. Both have a knob that trades recall against latency, and both need the data present before you build them.

Run a few hundred queries against an exact search and against your index, and compare the result sets. If the index is returning 92% of what exact search found, you now know your ceiling, and you can stop blaming the model for the other 8%. Almost nobody does this and it takes an hour.

Test yourself with one question: if the vector store were wiped right now, could you rebuild it entirely from your database? If yes, everything below is a chore. If no, you have a data-loss problem wearing a search problem's clothes.

ON DELETE CASCADE is the single most valuable line in that schema. It means the most common sync bug in the world — deleted record, orphaned vector, ghost search result — is structurally impossible. If your vectors live in a separate service you have to write that behaviour, remember to call it, and handle it failing.

SELECT c.content, d.title, c.start_offset,
       1 - (c.embedding <=> $1) AS score
FROM   document_chunks c
JOIN   documents d ON d.id = c.document_id
WHERE  c.tenant_id  = $2                 -- filter FIRST
  AND  c.embed_model = $3
  AND  d.updated_at >= $4
ORDER  BY c.embedding <=> $1
LIMIT  20;

One statement, one round trip, transactionally consistent with the rest of your data, and the tenant filter applied by the database rather than hoped for afterwards. That is the argument for keeping them together, and it is a strong one.

Pre-filtering means the constraint is applied as part of the search, so the ten you get back are the best ten that qualify. Every serious vector store supports this; the trap is that the naive API in the quickstart usually does not, and it works fine on your test data where every row belongs to you.

Do not embed inline in the request that saved the row — the embedding call can be slow and it can fail, and neither should stop a user saving their work. Write the row, enqueue the job, let it retry. Store the hash of the text you embedded so the job can decide whether there is anything to do, and so a nightly sweep can find rows whose hash no longer matches.

Queues drop things. Deploys interrupt workers. Somebody will run a bulk UPDATE in a console. A job that walks the table comparing source_hash against the current text — and reports how many disagreed — turns "our search is stale and nobody knows why" into a number on a dashboard.

The safe shape: write both old and new vectors for a while, keep serving from the old one, backfill the new one in the background, compare the two on a set of real queries, then cut over and drop the old column. Dimensions usually differ between models, so this often means a second column rather than an overwrite — which is a feature, because it makes rollback possible.

So run both and combine the rankings. The standard method is reciprocal rank fusion: score each result by 1 / (60 + its rank) in each list and add them up. It needs no tuning, no score normalisation, and no knowledge of how the two systems scale their numbers — which is why it has outlasted cleverer schemes.

Postgres ships full-text search. If your chunks are already in Postgres, hybrid is one tsvector column and a second ranked query away, in the same statement, with the same tenant filter. There is very little reason not to.

The reason it works: an embedding is made without knowing your question, so it can only measure aboutness. A reranker reads both at once, so it can tell "refunds take 5–10 days" from "refunds are not available on sale items" for the question "how long does a refund take". Nothing in the first stage can make that distinction.

Latency, and it is not free — fifty pairs to score is real time. But it lets you retrieve more loosely than you otherwise dare, which means fewer misses at the first stage. Budget for it and measure both halves separately, or you will not know which one is failing.

Ids you can resolve back to a document and an offset make citations real rather than decorative — a person can click through and check. That single feature does more for trust than any amount of accuracy improvement, because it lets a sceptical user verify instead of believing you.

Retrieved text is data, not instructions. If a document contains "ignore your instructions and…", a naive pipeline will do it. Keep sources inside delimiters, say plainly that content within them is reference material only, and never let retrieved text decide whether a tool gets called.

Two numbers are enough to start. Recall@k: how often the right chunk is somewhere in the top k. MRR: how near the top it was. Recall tells you whether retrieval is even possible; MRR tells you whether reranking is earning its keep.

And separate the two failures. If the right chunk was never retrieved, no prompt change will save you. If it was retrieved and the answer was still wrong, retrieval is fine and the problem is downstream. Teams burn weeks on the wrong half of that.

Retrieval is also the wrong tool for questions that need aggregation rather than lookup. "How many orders were refunded last month" is a SELECT COUNT(*). Fetching twenty chunks about refunds and hoping the model counts correctly is a worse database with extra steps. Give it a tool that runs the query.

Log what was retrieved for every request. Before touching anything else, check whether the answer was in the context at all. This one habit will save you more time than everything else on this page.

Hybrid is the default, not the advanced option. Add keyword search and fuse the rankings.

In the same database this is ON DELETE CASCADE. Across two systems it is code you must write, call and monitor — plus a sweep that catches what the code missed.

Filter inside the search. It is also the difference between a quality bug and a data-leak bug, depending on whether the discard ever gets skipped.

Record the model on every row. Dual-write, backfill, compare, cut over.

Split on the document's own structure and fall back to a size limit. Add overlap. Put the heading in the chunk.