Benchmarking Fine-tuning and Retrieval Strategies for a Multimodal Language Model on the NRC Reactor Operator Licensing Examination

A 31-billion-parameter open-weight multimodal model (Gemma 4 31B-IT) fine-tuned with supervised fine-tuning and retrieval-augmented generation using fixed-size chunking passed 8 of 14 U.S. Nuclear Regulatory Commission Reactor Operator licensing examinations, meeting the 80% human passing criterion, while no configuration without fine-tuning passed any exam. The study, published on arXiv, found that the preferred chunking strategy reverses depending on the model's training state and that retrieval-augmented fine-tuning underperforms standard supervised fine-tuning in matching search environments.

arXiv:2607.22067v1 Announce Type: new Abstract: The integration of large language models LLMs into the nuclear power industry requires outputs grounded in domain-specific knowledge. This study evaluates a 31-billion-parameter open-weight multimodal model Gemma 4 31B-IT on its capacity to apply nuclear knowledge by benchmarking eight model-retrieval configurations against the U.S. Nuclear Regulatory Commission NRC Reactor Operator licensing examination. We evaluate 14 Generic Fundamentals Examinations GFE from the 2015-2021 March sittings seven pressurized and seven boiling water reactor exams using the standard 80% human passing criterion. The base model is compared against configurations utilizing supervised fine-tuning SFT on Gemini-distilled chain-of-thought CoT rationales, retrieval-augmented generation RAG with BM25 sparse retrieval over the U.S. Department of Energy Fundamentals Handbook, and retrieval-augmented fine-tuning RAFT . Within the retrieval pipeline, we compare fixed-size sliding-window chunking against structure-aware chunking. The SFT configuration with fixed-size chunking RAG met the criterion on 8 of the 14 examinations, outperforming all alternatives, whereas no configuration without fine-tuning passed any. Aggregate accuracy reached 79.7%, with a confidence interval spanning the threshold, and 80.2% on PWR items specifically. Furthermore, two regularities emerged: the preferred chunking strategy reverses depending on the model's training state, and RAFT underperforms compared to standard SFT in matching search environments. These results demonstrate which combination of fine-tuning and search approaches achieves operator-level capabilities.