A Little Experiment with Non-Deterministic LLM Applications — A Language Learning App

I live in Hungary, and I am learning Hungarian from a real textbook. That is actually where this idea came from. Flashcard apps with someone else’s vocabulary list were not enough for me — I wanted something built around the specific book I was studying. So I built a tool that takes a real textbook PDF (plus its audio), reads it, and turns it into a structured knowledge base: every vocabulary word, grammar point, and expression, linked back to the exact page it came from. From that knowledge base, it generates exercises, flashcards, and listening drills for that specific book, using LLMs that run entirely on my own laptop.

One rule I set for myself from the start: nothing about my book leaves my machine. No uploading a textbook to a third-party API. That single constraint is what turned this into a real engineering project instead of “just call an API”: local models are slower, smaller, and much less predictable than a hosted one, and you have to design around that instead of ignoring it.

Below is a walkthrough of the parts I am most proud of: how I planned the build, how I structured the LLM integration so it was not just a pile of API calls glued together, how the pieces actually run, and — the part I think shows the most engineering judgment — how I found and fixed real performance problems by measuring instead of guessing.

The stack, and how a book becomes exercises

Quick orientation before going deeper. The backend is Java 21 / Spring Boot, with PostgreSQL for storage (Flyway for migrations, not auto-generated schemas), RabbitMQ for the async pipeline, and Apache PDFBox for reading the PDF itself. The frontend is Angular with Angular Material. The two language models run through Ollama. None of this is exotic — the interesting part is how the pieces are wired together, not any single technology choice.

Here is the actual flow, from a PDF landing on disk to a learner’s failures feeding back into what they review next. This follows the same shape as the architecture diagram in my original spec — knowledge base out to exercises/flashcards/listening, attempts back to a failure-driven review loop — just drawn with the real implementation detail filled in:

A few things the diagram does not show but matter in practice: the structure-extraction step does not just walk the detected chapters — it also diffs the book’s total page range against what the detected tree actually covers, and schedules extraction for any leftover gap (a chapter’s own intro text before its first subsection, for example), so nothing between two detected sections gets silently skipped. Each of the three queues in the diagram (structure.extraction.queue, knowledge.extraction.queue, exercise.generation.queue) has its own dead-letter queue, so a message that keeps failing gets parked somewhere inspectable instead of disappearing. Splitting knowledge extraction into one message per section, rather than one per book, is what makes a 700-page book survivable end to end: a single bad section fails on its own, and the other 145 keep going. And flashcards deliberately skip the LLM entirely — they’re built straight from the stored knowledge items, so a deck is instant even with no model loaded, at the cost of being less varied than a generated exercise.
The knowledge base model, and why it looks like this

The “knowledge base” box in the diagram above is really one small, deliberately boring data model. It is worth a closer look, because the shape of it is doing real work — it is the one piece every other feature (exercises, flashcards, listening, review) reads from, so getting it wrong would have meant redesigning everything downstream of it.

The knowledge base model, and why it looks like this

A few choices here were deliberate, not defaults I happened to land on:

Vocabulary, grammar, and expressions all share one base table (KnowledgeItem), with the three types as subclasses of it. The alternative — three separate, unrelated tables — would mean every place that browses or selects knowledge (the browsing screen, the scope selector, exercise generation) would need to query three tables and merge the results. With one shared table, a single query filtered by book, chapter, or topic returns vocabulary, grammar, and expressions together, and the type is just a column on the result. The three subtypes only add the one or two fields that are actually specific to them — a vocabulary item’s part of speech, a grammar point’s pattern text — everything else lives on the shared parent.
Every item carries a link back to the exact StructureNode (chapter/section) and page it came from, plus the original source excerpt. This is not just nice-to-have metadata — it is provenance, and the spec calls it out as a requirement on its own (“store source provenance”), separate from extracting the knowledge itself. In practice it means the UI can always show “this word came from page 42, in the section ‘At the Restaurant’,” and a wrong or strange extraction can be traced back to exactly what text produced it instead of being a mystery.
Topics are their own table, upserted by name as extraction discovers them — not a fixed list baked into the code. A hardcoded topic list would have to guess every subject a book might ever cover, in every language, before a single book was processed. Instead, whatever the extraction model names (“Greetings,” “Numbers,” “At the Doctor”) becomes a real Topic row the first time it comes up, scoped to that language. This is also what keeps the whole application language-agnostic, which the spec is explicit about: nothing in this table, or in the code that queries it, is aware that a given book happens to be French, Hungarian, or anything else.
Examples live in their own table (KnowledgeExample) instead of being a text field on the item. A single vocabulary word or grammar point can reasonably have more than one usage example, and keeping them as separate rows (rather than, say, joining them into one blob) means they stay individually queryable and orderable — which matters once flashcards and exercises need to pick a specific example for a specific item.
StructureNode is self-referencing (a node can be another node’s parent). Textbooks nest arbitrarily — a chapter has sections, a section has subsections — and a fixed number of levels would eventually be wrong for some book. A self-referencing tree handles any depth without a schema change.

None of these are unusual patterns on their own. What made them worth getting right up front is that this table sits at the center of the whole system — structure and knowledge extraction write into it, and browsing, exercises, flashcards, listening, and review all read out of it. A shortcut here (three separate tables, a fixed topic enum, no provenance) would not have broken anything on day one, but it would have made every feature built after it noticeably harder.

How I planned it: spec first, code second

Before writing any code, I wrote two documents by hand: a spec describing the architecture and how the pieces fit together, and a requirements document describing what the product actually needed to do for a learner. Only after those existed did the build start.

This might sound like process for its own sake, but it paid off in a specific way. I built this with an LLM coding assistant doing most of the implementation, and an assistant without a spec to check against will happily reinvent the architecture a little differently every session. Writing the important decisions down before implementation — how the LLM integration should be structured, how the PDF’s chapter/section structure should be detected — turned “what should this look like” into “does this match what we already decided.” That is a much more stable question to keep asking over a long build.

It also paid off long after the initial build was “done.” Every bug I found later — a frozen exercise screen, a parsing failure — got debugged against the documented pipeline, not against guesswork about how the code happened to behave. That is the real value of a spec: it outlives the session that wrote it.

The interesting engineering problem: making an unreliable LLM reliable

This is the part that is not just “call an LLM.” I use two different local models for two different jobs: a bigger, slower one for reading and extracting knowledge from the textbook (quality matters most here, since it happens once per book and every mistake carries downstream), and a smaller, faster one for generating exercises on demand (speed matters most here, since it happens constantly while someone is actually using the app). I wanted swapping either model to be a configuration change, not a rewrite, and I wanted a clean seam to plug in a hosted model like Claude later for the quality-critical path, without touching anything else.

So the whole LLM integration sits behind one interface. Application code depends only on this interface, never on a concrete model implementation:

public interface LlmProvider {

    /** Identifies this provider for routing/config purposes, e.g. "qwen", "llama", "claude". */
    String name();

    KnowledgeExtractionResult extractKnowledge(KnowledgeExtractionRequest request);
    ExerciseGenerationResult generateExercises(ExerciseGenerationRequest request);
    GrammarReviewResult generateGrammarReview(GrammarReviewRequest request);
}

QwenProvider and LlamaProvider are two small, separate implementations of it. Both just delegate to the same underlying Ollama HTTP client, and differ only in their name — that is what makes them selectable independently:

@Component
public class QwenProvider implements LlmProvider {
    private final OllamaClient ollamaClient;

    @Override public String name() { return "qwen"; }

    @Override
    public KnowledgeExtractionResult extractKnowledge(KnowledgeExtractionRequest request) {
        String prompt = KnowledgePrompts.build(request);
        return ollamaClient.generateStructured(request.model(), prompt, KnowledgeExtractionResult.class);
    }
    // generateExercises / generateGrammarReview follow the same shape
}

A router sits in front of both and decides which provider — and which specific model — handles a given job, purely from configuration:

llm:
  extraction:
    provider: ${EXTRACTION_PROVIDER:qwen}
    model: ${EXTRACTION_MODEL:qwen3:8b}
  exercises:
    provider: ${EXERCISE_PROVIDER:llama}
    model: ${EXERCISE_MODEL:llama3.2:3b}

public KnowledgeExtractionResult extractKnowledge(KnowledgeExtractionRequest request) {
    Workload workload = requireWorkload(properties.extraction(), "llm.extraction");
    return resolve(workload).extractKnowledge(request.withModel(workload.model()));
}

private LlmProvider resolve(Workload workload) {
    LlmProvider provider = providersByName.get(workload.provider().toLowerCase());
    if (provider == null) {
        throw new IllegalStateException("No LlmProvider registered for '" + workload.provider() + "'");
    }
    return provider;
}

That is the whole idea: changing which model handles extraction, or pointing exercise generation at a completely different provider, is a one-line environment-variable change. Never a code change, never a redeploy of application logic. It is also what makes the interface a real seam and not just decoration: adding a hosted provider later (say, Claude, for the quality-critical extraction path) just means writing one more class that implements the same three methods and registering its name. Nothing that already calls LlmRouter needs to change.

Worth showing what actually gets sent to the model, since “prompt engineering” is often talked about in the abstract. This is the real extraction prompt, trimmed slightly — it is a plain Java text block with a few placeholders filled in per section, ending in an explicit JSON shape the model is told to match exactly:

public static String build(KnowledgeExtractionRequest request) {
    return """
            You are a language-learning content analyst. You read a page from a %s textbook \
            and extract structured study material. Explanations, meanings, and notes must be \
            written in %s. Do not invent content that is not supported by the source text.

            Book: %s
            Section: %s
            %s

            Source text:
            ---
            %s
            ---

            Respond with ONLY a single JSON object matching exactly this shape (omit nothing, \
            use empty arrays [] when a category has nothing relevant):
            {
              "vocabulary": [
                {"headword": "", "partOfSpeech": "", "meaning": "", "notes": "",
                 "examples": [{"text": "", "translation": ""}]}
              ],
              "grammar": [...],
              "expressions": [...],
              "topics": ["short topic names such as Restaurant, Travel, Greetings - derive these \
            from the content, do not use a fixed list"]
            }
            """.formatted(/* learning language, explanation language, book title, section title, page, source text */);
}

Two things worth pointing out here, because they came from real mistakes, not from getting it right the first time. “Do not invent content that is not supported by the source text” is there because an early version without it produced fluent-sounding vocabulary that was not actually in the book — a small local model is happy to fill gaps with something plausible if you let it. And topics are explicitly derived from the content rather than picked from a fixed list, because a fixed list would not survive contact with a second book in a different language; deriving them keeps the same prompt working for any textbook, in any language, without a code change.

The interface, the router, and the prompt template were the “easy” architecture work. The genuinely hard part was something I did not fully appreciate until I hit it in practice: a 3-billion-parameter local model does not reliably follow instructions about output format, even when you ask it for strict JSON and describe the exact shape you want. I ran into this same category of problem three separate times, in three different disguises:

Asked to answer a multiple-choice question with the correct option’s text, it sometimes answered with the option’s number instead.
Asked for a flat list of strings, it sometimes nested them as pairs — which is actually a reasonable way to represent a matching exercise, just not the shape I asked for.
Given a wide-open scope, it silently built a request over 400,000 characters long, which got truncated by the runtime, and came back with nothing. No error, just an empty result that looked like the model had simply failed.

None of these are bugs in the usual sense — the code was doing exactly what I told it to do. The model simply does not honor a contract the way a typed API would. The fix, every time, was the same shift in thinking: treat the model’s output as something to recover a good answer from, not as something to trust. As one concrete example, when parsing an entire batch of generated exercises fails because one item in it has a malformed field, I no longer throw the whole batch away. Instead I fall back to parsing item by item and keep whatever is individually valid:

private static ExerciseGenerationResult lenientParseExercises(ObjectMapper mapper, String json) {
    JsonNode exercisesNode = mapper.readTree(json).get("exercises");
    List<GeneratedExercise> kept = new ArrayList<>();
    for (JsonNode element : exercisesNode) {
        try {
            kept.add(mapper.convertValue(element, GeneratedExercise.class));
        } catch (Exception e) {
            // one malformed exercise should not cost the whole batch — skip it, keep the rest
        }
    }
    return kept.isEmpty() ? null : new ExerciseGenerationResult(kept);
}

Losing one exercise out of five to a formatting quirk is an acceptable trade. Losing all five because one had a strange field is not, and that difference is what this whole layer is built around. Alongside this: a JSON extractor that strips the markdown wrapping models like to add before parsing anything, a normalizer for fields that can reasonably come back in more than one shape (an answer as an index versus its literal text), and a hard cap on how much book content ever goes into a single request, so “too much context” fails as a visibly bounded response instead of silently as nothing at all.

The lesson I took from this, one I would apply to any project using a small local model: you cannot prompt your way out of an unreliable model. You have to engineer the boundary around it.

How it actually runs

The deployment split comes down to one hardware fact: on a Mac, Docker does not get access to the chip’s hardware acceleration, so anything doing LLM inference needs to run natively on the host, not inside a container.

Everything in the flow above that talks to RabbitMQ runs as a background job rather than inside an HTTP request, which is what lets a multi-hour extraction run without anything up front timing out while it waits.

Two unglamorous decisions turned out to matter a lot once real data was involved: making sure the database and file storage survive restarts (easy to skip in a prototype, expensive to regret after a 40-hour extraction run), and — the topic of the next section — actually measuring how much work the queue could handle at once instead of assuming that more concurrent jobs meant more throughput.

Chasing performance: a few real stories

This part best shows how I actually work when something is wrong: turn “it feels slow” into a number before touching any code.

“The model is slow” turned out to be “the machine is out of memory.” When extraction was taking longer than it should, my first instinct was to blame the model. Instead I checked how much the OS was swapping to disk, and found several gigabytes of swap in active use on a 16GB machine. An idle app I had left running in the background was quietly using the RAM the model needed. Closing it roughly doubled extraction speed, measured on the same workload before and after, with no code changes at all. The lesson: on a resource-constrained machine, “the LLM is slow” and “the computer is out of memory” look identical from the outside, and only checking memory pressure directly tells you which one you are actually dealing with.

More parallelism made things slower, and I only found out because I timed it. I tried letting the job queue run two extraction jobs at once, expecting it to roughly double throughput. Instead, I measured both jobs individually, and both took longer than a single job running alone — long enough that one of them timed out entirely. The machine was not idle waiting for a second job; it was already at its limit with one. Adding concurrency is a reasonable first instinct, but it only helps once you have confirmed there is spare capacity for it to use, and the only way to confirm that is to time it, not assume it.

A guessed timeout was breaking real work. I had set a timeout based on “how slow could this reasonably be” — first 2 minutes, then 5. Both were wrong. Once I was running real chunks of a real book, legitimate calls were getting killed mid-answer. After I actually measured how long real calls take (anywhere from about 2 minutes to well over 5, depending on what else the machine was doing), the fix was to stop guessing at a ceiling and let a call run as long as it needs, while keeping a short timeout only to check that the server is reachable at all. A timeout based on a guess is eventually wrong in one direction or the other; a timeout based on a measurement can be exactly as strict as it needs to be.

A silent failure only got found by refusing to accept “it returned nothing” as an answer. A broad, unscoped request for exercises came back completely empty, with no error anywhere in the logs. The easy explanation is “the model just did not answer.” Instead I logged the actual size of what was being sent, and found a prompt over 400,000 characters long, cut off by the system before the model ever finished reading it. The real fix was a couple of lines: cap how much content ever goes into one request. Finding it meant not stopping at the first explanation that merely sounded plausible.

Run end to end against my real 723-page textbook, the finished pipeline extracted all 146 sections, produced roughly 4,200 distinct vocabulary, grammar, and expression items, and organized them into 616 topics, with only 20 isolated failures across the entire run. That number is a direct result of the measure-first habit above, not of any single clever piece of code.

What this project actually demonstrates

Designing around an unreliable dependency — building a resilient boundary layer around a local model that does not reliably follow instructions, instead of assuming it will.
System design under real constraints — a provider abstraction that made model choice a configuration change, and a deployment split driven by an actual hardware limitation, not a diagram drawn for its own sake.
Debugging discipline — every “it’s slow” or “it’s broken” claim in this project got turned into a measurement (vm.swapusage, timed concurrent calls, an actual prompt character count) before I changed a single line to fix it.
Planning that survives a long build — a spec written up front that kept a multi-week, LLM-assisted build pointed at one consistent architecture instead of drifting from session to session.