EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues

Researchers introduced EduBehaviors, an interpretable annotation framework that uses large language models to measure repeated observable behaviors in educational dialogues and then learns a classifier for a target construct, according to an arXiv paper (arXiv:2609.27043v1). Evaluated on the TalkMoves dataset for predicting Teacher TalkMoves labels, the best configuration reached a macro-F1 of 0.673 and 0.688 Cohen's kappa, which the authors report as competitive with direct prompting approaches. The team also released the EduBehaviors Toolkit, two tools for operationalizing the framework on researchers' own data.

arXiv:2609.27043v1 Announce Type: new Abstract: Large language models have allowed the rapid deployment of pedagogical annotations corresponding to constructs of interest, allowing a natural language interface for generating classifications on a conversational dataset. However due to the opaque nature of LLM reasoning, we have no verifiable, mechanistic insight into why a model chose a label for an utterance. We introduce the EduBehaviors framework, an interpretable, scalable approach to annotating educational data that uses LLMs to measure repeated observable behaviors relevant to many constructs of interest and then learns a classifier for the construct based on these observable behaviors. We evaluate the framework on the TalkMoves dataset, predicting the Teacher TalkMoves labels. Our best configuration results in a macro-F1 of 0.673 and 0.688 Cohen's kappa, proving competitive with direct prompting approaches. In addition, we release EduBehaviors Toolkit, two tools allowing researchers to operationalize the EduBehaviors framework in their own data.