For a while now I’ve been learning languages from real textbooks I actually own — PDFs, not some app’s flashcard deck. Some time ago I built a tool that reads a whole book once, up front, and turns it into a structured knowledge base: every vocabulary word, every grammar point, every expression becomes its own typed entry. It works well, but it’s slow. On a real 723-page book, that extraction pass took around 40 hours of local processing before I could use the book for anything.
This time I wanted to try the opposite idea: don’t extract anything up front. Just index the book cheaply, and only make an LLM actually read something at the moment I ask a real question. This is the classic RAG (Retrieval-Augmented Generation) pattern, and I wanted to build one myself, end to end, fully local, and see honestly how it compares to the eager-extraction approach I already had.
This post is about that second project — what it does, how it’s actually built, and what building it taught me about the pieces underneath it.
The idea, in one diagram
Two phases that barely talk to each other, except through the vector store sitting between them:
Ingestion is boring on purpose — no generative model call anywhere in it, just text extraction, splitting, and embedding. All the “intelligence” happens later, at question time, and only over a handful of retrieved paragraphs instead of the whole book. That’s the whole bet this architecture makes: indexing should be cheap, and the model should only ever see a small, relevant slice of the book, never the entire thing.
Two different neural networks make that possible, doing two genuinely different jobs, and I think it’s worth actually explaining both before getting into the plumbing.
Embeddings and chat models are not the same kind of model
It’s easy to think of “the embedding model” and “the chat model” as two settings of the same thing, since Ollama serves both the exact same way. They’re not. They have different architectures, different training objectives, and different outputs.
The embedding model is closer to a classifier than a writer. It reads the whole input once and produces a single fixed-length vector — it never generates a token of text. nomic-embed-text is trained with a contrastive objective: during training, pairs of texts that are semantically related get pulled closer together in vector space, and unrelated pairs get pushed apart. Nothing about it is autoregressive; there’s no “next word” being predicted. The entire point of the training is that the geometry of the output space — distance and direction between vectors — ends up meaning something. That’s what makes “find the chunks whose meaning is closest to this question” a real, computable operation instead of just keyword overlap.
One genuinely non-obvious detail I learned building this: nomic-embed-text is actually trained to expect short task-instruction prefixes on its input — search_document: when embedding something that will be searched, search_query: when embedding the thing doing the searching — because it’s optimized for exactly this asymmetric retrieval setup (a short question finding long passages) rather than symmetric similarity (two similar-length texts compared to each other). I didn’t add these prefixes in this version — both chunks and questions go in raw. It still works, because the vectors are still meaningfully close for related meanings, but I’m fairly sure retrieval quality is leaving something on the table by skipping this. It’s on my list to try.
The chat model is the opposite shape. It’s autoregressive — at every step it looks at everything generated so far and predicts a probability distribution over the next token, then samples from it, one token at a time, until it decides to stop. temperature=0 in this app isn’t a stylistic choice, it’s a correctness one: temperature reshapes that probability distribution before sampling, and at 0 it collapses to always picking the single most likely next token — deterministic, repeatable output for the same input. For a system whose whole job is “answer faithfully from this fixed context,” I want the boring, consistent answer, not a creative one.
Different job, different math, same runtime serving both. That’s the whole embeddings-vs-chat distinction, and it’s the reason the pipeline has two separate model calls instead of one.
Vector search and why a normal database can’t do this
A regular database — even an indexed one — answers questions like WHERE page = 42 or WHERE id BETWEEN 10 AND 20 extremely well, because a B-tree index can jump straight to the right region of sorted data. It has nothing for “give me the 5 rows whose 768-number vector is closest to this other vector.” There’s no meaningful way to sort high-dimensional vectors along one axis, so a classic index doesn’t help — the honest answer would be comparing the query against every single stored vector, which is fine for a thousand rows and falls apart at scale.
This is the actual reason vector databases exist as a separate category of infrastructure. Qdrant (and similar systems) solve it with approximate nearest neighbor (ANN) search instead of brute-force comparison — specifically, Qdrant builds an HNSW graph (Hierarchical Navigable Small World): a multi-layer graph where the top layer has few points with long-range connections, and each layer below gets denser and more local.
A search starts at the sparse top layer, greedily walks toward whatever’s closest to the query, then drops down a layer and refines — narrowing in on the true neighborhood in roughly logarithmic steps instead of scanning everything. “Approximate” is the honest trade being made: it doesn’t guarantee the mathematically exact top-k closest vectors, in exchange for being dramatically faster at any real scale. For “find roughly the most relevant passages,” that trade is exactly right — a RAG system was never going to need mathematically perfect nearest neighbors, just good ones, fast.
The Qdrant concepts that actually show up in this app’s code:
- Collection — roughly a table. I use exactly one,
book_chunks, for every book. - Point — one stored item: an id, its vector, and a payload (arbitrary JSON metadata).
- Filter — a query can require the payload to match a condition in addition to being vector-close. Qdrant applies filters as part of the graph search itself rather than doing the ANN search first and throwing away results afterward — which matters, because naive post-filtering can starve you of results when the filter is selective (imagine searching a million-point collection but only 200 points belong to the one book you asked about).
That last point is exactly how one collection serves every book safely. Every point looks like this:
{
"id": "<uuid>",
"vector": [0.014, -0.22, 0.081, "... 768 numbers total"],
"payload": {
"page_content": "the chunk's raw text",
"metadata": { "book_id": "d892f07d-...", "page_number": 62, "chunk_index": 0 }
}
}
And the filter that scopes every search to one book, reused for both retrieval and for clearing out old chunks before a reindex:
def book_filter(book_id: str) -> Filter:
return Filter(must=[FieldCondition(
key="metadata.book_id", match=MatchValue(value=book_id)
)])
Distance metric is cosine similarity — the angle between two vectors, not their raw length. That matters because an embedding vector’s magnitude isn’t a meaningful signal here; two vectors can point in almost the same direction (same meaning) while having different lengths, and cosine similarity is the metric that ignores length and measures only direction.
I chose Qdrant specifically over the two obvious alternatives: pgvector (a Postgres extension — a fine choice if I already had a Postgres instance to lean on, but would’ve meant reusing my other project’s database rather than actually evaluating a purpose-built vector store), and Chroma (genuinely great for a five-minute local prototype, but light on the filtering and operational maturity I wanted once I actually cared about keeping books cleanly separated).
LangChain — the glue
Every piece above (loader, splitter, embedding model, vector store, chat model) has its own separate API if used directly. LangChain’s job is to give all of them a common shape — a Document, an Embeddings interface, a VectorStore interface, a ChatModel interface — so a pipeline built against those shapes doesn’t care which specific loader, embedding model, vector store, or chat model is plugged in underneath. Swapping Qdrant for a different vector store, or one Ollama model for another, becomes a one-line change instead of a rewrite.
LCEL (LangChain Expression Language) is the syntax for composing these pieces into a pipeline using the | operator, similar in spirit to a Unix pipe — each stage’s output becomes the next stage’s input:
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt_template
| chat_model
| StrOutputParser()
)
Reading that left to right: take the question, run it through the retriever to get chunks (and format them into plain text), feed both the formatted context and the original question into the prompt template, send the filled-in prompt to the chat model, parse its output into a plain string. A retriever, concretely, is just a thin wrapper around a vector store’s search call — retriever.invoke("some question") embeds the question and returns the top-k matching Documents, hiding the embed-then-search steps behind one call.
My actual chain in this app is a narrower version of exactly that shape — no separate retriever stage, because query.py calls Qdrant’s similarity_search_with_score directly instead. It needs the raw similarity scores back alongside the chunks, to show as real sources, and the standard retriever interface doesn’t expose those:
chain = PROMPT | ChatOllama(model=chat_model, temperature=0) | StrOutputParser()
answer = chain.invoke({"context": retrieved_passages, "question": question})
Worth naming honestly (this is also the comparison I keep coming back to against my other project): that project hand-built an LlmProvider/LlmRouter specifically to get “swap the model without touching application code.” LangChain’s ChatModel/Embeddings interfaces hand that same property over largely for free here, as long as the model has a LangChain integration already written for it — a real trade-off of picking an ecosystem-native framework (less code to write, more dependence on the framework’s abstractions) over a hand-rolled one (more code, but full control and no hidden behavior).
Tech stack, at a glance
| Layer | Choice |
|---|---|
| Language | Python 3.12 |
| RAG orchestration | LangChain (LCEL) |
| PDF loading | PyPDFLoader (wraps pypdf) — per-page text, which page-accurate citations need |
| Chunking | RecursiveCharacterTextSplitter, 1000 chars / 150 overlap |
| Embeddings | nomic-embed-text via Ollama |
| Chat model | Configurable via Ollama (llama3.2:3b by default) |
| Vector store | Qdrant, self-hosted |
| API | FastAPI |
| UI | Streamlit |
| Deployment | Docker Compose; Ollama native on host |
Running it: containers vs. the one thing that isn’t
Everything except Ollama runs in Docker Compose — three services, three responsibilities:
| Service | What it is | Port | Persists to |
|---|---|---|---|
qdrant | The vector database | 6333 | a named volume — survives restarts |
backend | FastAPI + the LangChain pipelines | 8000 | a named volume (book registry + original PDFs) |
frontend | Streamlit chat UI | 8501 | nothing — pure client |
The host.docker.internal line was the one real infra wrinkle: Docker Desktop on Mac doesn’t pass GPU access into containers, and both models need it to run at a reasonable speed. So Ollama has to stay outside Docker entirely, and the backend reaches it through a hostname Docker Desktop provides specifically for talking back to the host machine.
How the backend is organized
Five small modules, each doing one thing: main.py is pure HTTP plumbing, ingestion.py and query.py are the two pipelines from the first diagram, vectorstore.py is the only file that talks to Qdrant directly, and registry.py is a tiny JSON-file-backed book index — I didn’t want to justify a real database for “which books exist and what’s their content hash” in v1.
Ingestion, in more detail
Two details that look like they could be bugs but are deliberate:
- Deduplication is by content hash, not filename. Re-uploading the same PDF under a different name is recognized and skipped — no re-embedding, no duplicate book.
chunk_indexresets on every page. It’s “the Nth chunk of this page,” not a running total across the book — combined with the page number it’s still a fully reproducible identifier, just not a book-wide counter.
The collection’s vector dimension isn’t hardcoded either — the first time it’s created, the backend embeds a throwaway string with whatever embedding model is currently configured and reads the resulting vector’s length. Convenient, but it has a real gotcha: change the embedding model after a book is already indexed, and the collection keeps the old dimension — a same-dimension-but-different model will insert without error but silently return worse matches, since the new query vectors and the old stored vectors no longer mean the same thing in the same space.
A real example, with real numbers
I have a French textbook indexed (French All-in-One For Dummies, 723 pages, 1,738 chunks). I asked it this, through the actual UI, unedited:
“How date and time are told in french ?”
The five passages retrieval actually returned, in score order:
| Page | Score | What it was |
|---|---|---|
| 62 | 0.803 | “On the Clock: Telling Time” — the main section on this |
| 56 | 0.761 | “Using the Calendar and Dates” |
| 470 | 0.739 | An unrelated passage about the word quand (“when”) |
| 64 | 0.735 | Converting between 12-hour and 24-hour notation |
| 63 | 0.735 | Example sentences with il est … heures |
Notice page 470 — a genuinely weak, off-topic match. There’s no similarity-score floor in this app: similarity_search_with_score always returns its top-k results for a matching book, however weak the last one actually is. It didn’t derail the answer here; the other four carried it. But it’s a concrete example of a limitation I’d otherwise only be able to describe in the abstract.
The more interesting thing I learned from this exact run, honestly by accident: the system prompt tells the model to cite page numbers inline, like (p. 62). It didn’t — not once, in this answer or in most others I tried. If the app trusted the model’s own generated text as the source of the citation, this answer would have shipped with zero page references, despite being genuinely well-grounded in five real, correctly-page-numbered passages. So the app doesn’t do that. The “Sources” panel under every answer is built directly from what Qdrant returned — the (chunk, score) pairs — never parsed out of anything the model claims about itself.
That single design decision is probably the most important lesson from this whole build: the model is not a reliable narrator of its own reasoning, even when explicitly instructed to be. Compute the provenance yourself, from the actual retrieval step; don’t ask the model to hand it to you afterward.
Comparing it to the other approach
The actual point of building this second version was to compare it, honestly, against the eager-extraction one:
| Eager extraction (my other project) | This project (RAG) | |
|---|---|---|
| Getting a book ready | ~40 hours, one full generative pass over 146 sections | Minutes — chunk + embed, zero generative calls |
| What’s stored | Structured rows — ~4,200 typed vocabulary/grammar/expression items | Text chunks + vectors, no structure beyond page/chunk provenance |
| “List everything about X” | Reliable — it was all extracted up front | Not reliable — only what a question happens to retrieve |
| Open-ended question | Not really built for this — fixed extraction categories | Exactly what it’s for |
| Where the model’s unreliability shows up | Output-schema drift — an answer as an index instead of text, a flat list coming back nested | Grounding drift — answering fluently off a bad retrieval, or from general knowledge instead of the source (see above) |
| Model-swap abstraction | Hand-built interface + router | Comes mostly free from LangChain’s Runnable interfaces |
Neither one is a better version of the other — they’re answering different questions. If I want “give me every piece of restaurant vocabulary in chapter 4,” the structured version wins easily, because every piece of restaurant vocabulary really is sitting in a table already. If I want “what’s the actual difference between these two grammar forms,” this one wins, and wins by a lot, because it was never limited to a fixed extraction schema in the first place.
What I’d build next
Things deliberately left out of v1, because I wanted the simple version working and proven before reaching for anything fancier:
- Task-prefixed embeddings. Use
nomic-embed-text‘s intendedsearch_document:/search_query:prefixes instead of raw text, and measure whether retrieval quality actually improves. - A similarity floor on retrieval, so “the book doesn’t cover this” becomes a guarantee enforced by the retrieval step, not just something the model is asked nicely to say.
- A delete-book endpoint. The registry already supports removing an entry internally; nothing in the API exposes it yet.
- Multi-turn memory. Every question today is answered completely independently of the ones before it.
- Reranking or hybrid search, if plain cosine similarity ever actually shows up as insufficient on real questions — not before, on principle. I’d rather add complexity in response to an observed failure than speculatively.
None of these felt worth building before I had a working, honest, correctly-cited answer coming out the other end first. That part, I’m happy with.
Project Link: https://github.com/samsaydali7/llm-language-learning-rag