I enjoyed this paper by Walton et al about the role of judgement in how students use AI. There’s currently a startling lack of rich data about the judgements students actually make in their use of chatbots chatbots, as they summarise on pg 2:
There remains little information beyond decontextualised self-reports about
what these judgements might be and how they influence what students learn (or fail to learn) from completing their assessments. Thus, educators and institutions continue to design assessments and set policy to take account of GenAI use without understanding how their choices will affect students and learning. Exploring how students make judgements with—and about—GenAI will therefore provide a much-needed perspective on how students are coming to learn with, rely on, and dissemble with GenAI.
Through a nicely designed walk through method they identify six categories of what they call judgement events: time bound occurrences where the student where a student evaluates AI and its outputs as they worked on an assessment. What I particularly like about this framing is how it enables us to distinguish between:
- The occurance and sequencing of judgement events.
- The (epistemically) better or worse judgement events which make up that sequence
It does what Milan and I describe in *The Platform Learns To Speak *as opening the blackbox of AI use in order to look at the process which underpins it. The obvious lesson to take from the notion of judgement events is to ask three questions of assessment design:
- What is the process? Where and when are judgements called for?
- What kind of judgement events are desirable for constructive alignment?
- How does the logic of the design ideally knit together these judgement events?
- How does the embedding of the assessment support or hinder this ambition?
A crucial point they make in this paper concerns student’s ability to distinguish their own epistemic contribution to the output. I’ve been prone in the last year to saying that* we need to help students understand what it feels like to be learning**. I stand by this but I realise it’s a precarious achievement rather than something we can rely on. This is why I like their two points on pg 13 so much:
This makes two points: firstly, GenAI use can enhance or hinder learning,
depending on circumstance. Secondly, students’ own views of what they learnt or how they worked with GenAI use does not distinguish between these cases.
*Thanks to David Meechan for setting me off on this, when he visited us.