Why LLM-Generated Flashcards Are Usually Bad, and How to Screen Them

An engineer building LLM-generated flashcard pipelines identifies four common failure modes—compound questions, ambiguous prompts, weak cloze deletions, and pattern-matchable answers—and presents a two-stage extraction pipeline and deterministic screening checks to produce cards that survive real review queues.

I've watched a lot of teams bolt an LLM onto a document and call the output flashcards. It demos beautifully and it teaches badly. The generated cards look plausible — grammatical, on-topic, correctly formatted — and they fall apart the moment a real learner tries to review them six weeks later. The core problem is that "summarize this into Q&A pairs" optimizes for a different objective than "produce items that are cheap to recall and hard to fake." Those two things diverge fast. Here's what I've learned building generation pipelines that survive contact with a review queue. Naive generation produces four bad card shapes, over and over. Compound questions. "What is the TCP three-way handshake and why does it exist?" That's two cards wearing a trench coat. The learner half-recalls, grades it "Good," and the scheduler now believes both facts are known. You've corrupted the scheduling signal, which is worse than not having the card at all. Ambiguous prompts. "What year did it happen?" — extracted from a paragraph where "it" was obvious. On the card, out of context, there's no unique answer. The learner fails, marks it Again, and it returns tomorrow to fail identically. This is the single most common defect I see, because the model has the source context in its window and doesn't notice that the card doesn't. Cloze deletions with too little residual signal. Take "The mitochondrion is the powerhouse of the cell." Blank the subject and you get a reasonable card. Blank two spans and the second one is often recoverable from grammar alone. A cloze that can be solved by syntax trains nothing. Pattern-matchable answers. If every card in a German-nouns deck has an answer starting with "der," the learner learns the deck, not the material. This only shows up at deck level, so per-card evaluation never catches it. Telling the model "make cards atomic" gets you partial compliance and no guarantees. The reliable move is to make the pipeline structurally incapable of emitting a compound card: split extraction from card-writing. Stage one extracts claims — self-contained propositions with entities already resolved. Stage two turns one claim into one card. A card can't be compound if its input was a single claim. CLAIM SCHEMA = { "subject": str, resolved entity, no pronouns "predicate": str, exactly one assertion "qualifiers": list, conditions / scope "source span": str, provenance, for verification } def extract claims chunk - list Claim : one call, constrained decoding into the schema ... def claim to card claim: Claim - Card: template chosen deterministically from claim shape; the model only writes the natural-language surface ... The second win is that pronoun resolution happens at claim time, with the source paragraph still in view. By the time you're writing the card, the subject is already a proper noun, and the "what year did it happen" class of bug becomes unrepresentable rather than merely discouraged. For cloze specifically, I stopped letting the model choose the deletion span freely. Instead: extract the claim, identify which token carries the claim's information payload usually the object of the predicate, or a numeric qualifier , and delete that. If a sentence has no high-information span, it doesn't become a cloze — it becomes a Q&A card, or nothing. You can catch most defects with cheap deterministic checks before anything reaches a learner. These are the ones that earned their keep: python def screen card, deck : issues = compound: coordination joining two verb phrases in the prompt if re.search r"\b and|as well as \b", card.front, re.I and has two verb phrases card.front : issues.append "compound" unresolved reference if re.search r"\b it|this|they|these|the former|above \b", card.front, re.I : issues.append "ambiguous reference" answer leakage: content words shared between prompt and answer back words = content words card.back overlap = content words card.front & back words if back words and len overlap / len back words 0.5: issues.append "answer in prompt" trivial cloze: deleted span is a function word if card.type == "cloze" and is closed class card.deleted span : issues.append "trivial cloze" verbosity: long answers are usually compound in disguise if len card.back.split 20: issues.append "answer too long" return issues Then two model-based checks worth the tokens: Answerability without source. Give a fresh model only the card front — no document — and ask it to answer. If it can't produce the expected answer, the prompt is under-specified. If it produces the answer trivially for material the learner shouldn't already know, the prompt is leaking. Batch this per generation job; it's one call for many cards. Deck-level distinctness. Embed every front and flag pairs above a cosine-similarity threshold. Near-duplicate fronts with different backs are the nastiest failure in spaced repetition: the learner can't tell which card they're looking at and starts guessing from queue position. Anything very close in a decent sentence embedding gets merged or rewritten. The screening layer rejects a meaningful share of raw generations, and the rejection rate turns out to be the most useful signal we have about source material. A chapter with a high reject rate is usually narrative prose with few extractable claims — which means the fix is the chunking strategy, not the card prompt. That whole extract-screen-regenerate loop is what sits behind card generation in SmartRecall https://smartrecallai.com , and it's the part that took longest to get right; the initial "prompt an LLM for flashcards" version took an afternoon. I spent too long on a learned quality classifier. The deterministic screens plus one answerability check catch nearly everything it would, and they're debuggable — when a card is rejected I can point at the rule that rejected it. I'd also ship with cloze generation off by default. Q&A cards fail loudly: the learner can't answer. Bad clozes fail quietly: the learner answers from syntax and feels productive. Ship the loud failure mode first.