Build your own ChatGPT (4/9): the memory — why sending the whole PDF does not scale

"Chat with your documents" is the feature most people want to copy and the one most people implement wrongly. The mistake is always the same, and it is expensive.

First, the misunderstanding

When ChatGPT answers "according to the file you uploaded", there is no file stored inside the model. As we saw in part 2, the model remembers nothing between calls.

What exists is a search system: the document was chunked and indexed beforehand; on each question the most relevant chunks are retrieved and pasted into the prompt. The model receives the question and the passages, and answers. That arrangement has a name — RAG, retrieval-augmented generation — and it is your product's memory piece.

Why not just send the whole document

It is a fair question: if the context window holds a million tokens, why not throw the whole manual into every question?

Because you pay for it on every question. A 300-page manual is around 210,000 tokens. A support assistant handling a thousand questions a day:

Ninety-five times the difference — and cost is not even the worst problem:

• Quality drops. Every model loses precision when the answer is buried in an enormous context. Four correct passages beat 300 pages of haystack.
• Latency explodes. Processing 210,000 tokens takes seconds. The user stares at a frozen screen.
• It does not scale. It works for one manual. For the company archive it fits in no context window at all.

How retrieval works, without the mystery

1. Chunk

The document is cut into pieces of a few hundred words. It looks trivial and it is the stage that most affects final quality — more than the choice of model.

Cutting every N characters is the bad way: it splits tables in half and separates a clause heading from its text. The good way is cutting by structure — by section, by clause, by article — with a 10% to 15% overlap between neighbours so a sentence on the boundary is not lost.

Size that works in most cases: 300 to 800 tokens per chunk. Too small loses context; too large dilutes meaning and inflates input.

2. Turn into coordinates of meaning

Each chunk goes through an embedding model, which returns a list of numbers — a coordinate in a space where texts with similar meaning sit near each other.

That is what lets "holiday entitlement" find a passage about "incomplete accrual period" without a single word in common. For non-English content, use a modern multilingual model; it runs on a small card, and indexing a whole archive is an afternoon's work.

3. Store

The coordinates go into a vector database — Qdrant is the mature choice and runs as a one-click template. It answers "give me the 20 chunks closest to this question" in milliseconds, across millions of passages.

4. Retrieve and answer

The user's question becomes a coordinate with the same model, the database returns the neighbours, and the best chunks go into the prompt alongside the question. The brain does the rest.

The detail that separates a toy from a product

After hybrid search, the second-biggest win is reranking: take the top 20 candidates, pass them through a small model that reorders them by genuine relevance, and keep the top 4. It is cheap and noticeably improves answer precision.

Citing sources is not optional

Every answer must say where it came from: document, section, page. For three practical reasons:

1. Trust. An answer with a source can be verified in ten seconds; without one, the user has to take it on faith.
2. Debugging. When an answer is wrong, the citation shows whether the failure was retrieval (wrong chunk) or generation (right chunk, bad answer). Those are completely different fixes.
3. A brake on invention. Instructing "answer only from the passages; if it is not there, say you did not find it" sharply reduces hallucination. It does not eliminate it — but it reduces it.

Where each piece runs, and what it costs

Notice the pattern again: indexing is a burst — start the GPU, process the archive, shut it down. Answering is constant — which is why it is paid per token.

The no-code route

If you want the result before building the engineering, AnythingLLM deploys in one click and delivers the whole package: file upload, indexing, chat interface and user management. Point it at the token API as its brain and you have an assistant over your documents working the same afternoon.

It is the right way to validate whether people will actually use it before investing weeks in a bespoke pipeline.

When NOT to use retrieval

To be fair to the alternative: if your archive is small — a 30-page manual, one policy, an FAQ — sending everything on every question is simpler and works better. That is about 20,000 tokens, R$ 0.014 per question on the mid model. Building RAG for that is too much engineering for too little problem.

The practical rule: above roughly 100 pages, or when the archive grows every week, build retrieval. Below that, do not.

Next: the brush — image generation.

Keep reading: building a RAG system · embedding models compared · part 5: images