Training LLM model for asking questions

A developer advising on training an LLM for adaptive questioning recommends keeping difficulty adaptation in application code rather than fine-tuning, using explicit state rendering in system prompts, and verifying synthetic data correctness. The post cites Item Response Theory libraries py-irt and catsim as superior to vague difficulty tuning, and names MathDial, TutorChat, CIMA, GSM8K, and MATH as key datasets.

The thing I’d flag first: this probably isn’t a fine-tuning problem. Difficulty adaptation is control flow, not language generation. Keep the ladder in your application code and let the model do only the part it’s good at. python state = {"difficulty": 3, "topic": "roots", "streak": 0, "errors": } your code decides the next difficulty the LLM only generates a question at difficulty N, and grades the answer Two correct in a row, difficulty up. One wrong, explain and drop down. Twenty lines of Python, and it behaves far more reliably than a fine-tune — a fine-tuned model gives you no guarantee of monotonic difficulty and no way to inspect why it picked what it picked. On pre-context: no, the model won’t reliably “analyse” past interactions. It attends to what’s in the window, but over a long session it drifts, forgets which topics you already failed, and repeats questions. Render the state into the system prompt explicitly every turn — “learner is at difficulty 3, missed cube roots twice, hasn’t attempted logarithms.” That one change fixes most of what people reach for fine-tuning to solve. If you do fine-tune, mostly for tone and format consistency, it’s standard multi-turn messages format with loss masked to assistant turns: json {"messages": {"role": "system", "content": "Learner difficulty: 3. Recent errors: cube roots."}, {"role": "user", "content": "I think it's 3."}, {"role": "assistant", "content": "Not quite. 2³ = 8, so the cube root of 8 is 2. Let's step back..."} } One example per assistant turn with the full preceding conversation as context. A 10-turn dialogue becomes 5 training examples, not 1. You’d want a few thousand dialogues before behaviour shifts meaningfully, which you won’t hand-write, so generate with a stronger model. But here’s the step people skip: verify the mathematical correctness of every wrong answer and every explanation. Synthetic tutoring data is full of confidently wrong explanations, and training on those teaches the model to be wrong pedagogically as well as factually. That’s the one part you can’t automate away. Datasets worth reading before you generate anything: MathDial and TutorChat for tutor-student dialogue structure, CIMA for the pedagogical move taxonomy, GSM8K or MATH as a question bank you can attach difficulty labels to. Last thing, and it’s probably the highest-value point. “Slightly harder” is doing a lot of undefined work. Adaptive testing solved this decades ago with Item Response Theory — each question gets a calibrated difficulty parameter, each learner an estimated ability, and you select the next item to maximise information about that ability. Look at py-irt and catsim. IRT bolted onto a plain LLM will beat a fine-tune with a vague notion of harder, and you get an interpretable ability estimate you can actually show the user.