A Cognitive Lab Study of Student AI Chatbot Use in Online Coursework

A cognitive lab study by Jinah Choi, Sonya Powers, Farzan Karimi-Malekabadi, and Michelle Barrett examined how students in Grades 7-12 interacted with an AI-enabled chatbot while completing online course activities, using chat logs, facilitator observations, post-session reflections, and survey responses. Published in the Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Works in Progress, pages 1-8, by the National Council on Measurement in Education (NCME), the study evaluated whether the chatbot supported its intended theory of action of helping students continue learning without simply completing work for them. The findings highlight typing fluency, answer-boundary handling, safety and privacy response patterns, and student-facing feedback mechanisms as key factors, illustrating how cognitive lab evidence can inform ongoing validation.

Abstract This cognitive lab study examined how secondary students in Grades 7-12 interacted with an AI-enabled chatbot while completing online course activities. Using chat logs, facilitator observations, post-session reflections, and survey responses, we analyzed spontaneous and lightly seeded interactions to evaluate whether the chatbot supported its intended theory of action: helping students continue learning without simply completing work for them. Findings highlight the importance of typing fluency, answer-boundary handling, safety and privacy response patterns, and student-facing feedback mechanisms. This study illustrates how cognitive lab evidence can inform ongoing validation. - Anthology ID: - 2026.aimecon-wip.1 - Volume: - Proceedings of the Artificial Intelligence in Measurement and Education Conference AIME-Con : Works in Progress https://aclanthology.org/volumes/2026.aimecon-wip/ - Month: - October - Year: - 2026 - Address: - Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States - Editors: - Joshua Wilson https://aclanthology.org/people/joshua-wilson/unverified/ , Christopher Ormerod https://aclanthology.org/people/christopher-ormerod/unverified/ , Magdalen Beiting-Parrish https://aclanthology.org/people/magdalen-beiting-parrish/unverified/ - Venue: - AIME-Con https://aclanthology.org/venues/aimecon/ - SIG: - Publisher: - National Council on Measurement in Education NCME - Note: - Pages: - 1–8 - Language: - URL: - https://aclanthology.org/2026.aimecon-wip.1/ https://aclanthology.org/2026.aimecon-wip.1/ - DOI: - Cite ACL : - Jinah Choi, Sonya Powers, Farzan Karimi-Malekabadi, and Michelle Barrett. 2026. A Cognitive Lab Study of Student AI Chatbot Use in Online Coursework https://aclanthology.org/2026.aimecon-wip.1/ . In Proceedings of the Artificial Intelligence in Measurement and Education Conference AIME-Con : Works in Progress , pages 1–8, Wyndham Grand Pittsburgh Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education NCME . - Cite Informal : - A Cognitive Lab Study of Student AI Chatbot Use in Online Coursework https://aclanthology.org/2026.aimecon-wip.1/ Choi et al., AIME-Con 2026 - PDF: - https://aclanthology.org/2026.aimecon-wip.1.pdf https://aclanthology.org/2026.aimecon-wip.1.pdf