Is a +5.5% improvement on GPQA Diamond normal when full fine-tuning an 8B model on CPU with recycled hardware?

A hobbyist team reported a +5.5% improvement on GPQA Diamond Zeroshot after full fine-tuning an 8B model on CPU with recycled hardware, moving from 24.7% to 30.3% accuracy in the first epoch. The team, using Dolphin 3.0 Llama 3.1 8B Puro FP32 as the base model, asked the community whether the base score is accurate and whether the improvement is significant or indicates data leakage/overfitting.

Hi everyone We are a team of AI enthusiasts who started learning Linux and understanding how language models work at the beginning of this year. We are not experts or professionals; we do this purely as a hobby and out of a desire to learn and contribute. We have built a home server Z6 architecture using old, second-hand parts. Due to our limited resources, we run our Full Fine-Tuning tests on 8-billion parameter models 8B using strictly CPU and RAM. Our current goal is to understand and validate if our approach is moving in the right direction. For training, we used a dataset of 144,000 words structured into 6 independent blocks roughly ~42,000 real tokens per block . Processing these 6 blocks marks the completion of Epoch 1, which corresponds to our Checkpoint-6. To evaluate our progress, we used the lm-eval library with a task we believe is highly objective: GPQA Diamond Zeroshot graduate-level questions . 1. Base Model Evaluation Dolphin 3.0 Llama 3.1 8B Puro FP32 This is the exact command we executed on our machine: lm eval run \ --model hf \ --model args pretrained=/xxxx/Dolphin3.0-Llama3.1-8b-Puro-FP32 \ --tasks gpqa diamond zeroshot \ --num fewshot 0 \ --batch size 1 \ --device cpu Results obtained: Task: gpqa diamond zeroshot | Version: 2.2 Metric / Value: acc: 0.2475 24.7% | acc norm: 0.2475 Stderr: ±0.0307 2. Our Full Fine-Tuning Evaluation Checkpoint-6 / Epoch 1 lm eval run \ --model hf \ --model args pretrained=/xxxx/checkpoint-6 \ --tasks gpqa diamond zeroshot \ --num fewshot 0 \ --batch size 1 \ --device cpu Results obtained: Task: gpqa diamond zeroshot | Version: 2.2 Metric / Value: acc: 0.303 30.3% | acc norm: 0.303 Stderr: ±0.0327 Our questions for the community Could you help us understand? : Since we are learning step by step, we would truly appreciate it if you could help us confirm two things: Is the base model score accurate? We want to know if anyone running this exact command for Dolphin 3.0 on different hardware gets the same ~24.7% accuracy, or if hardware differences can influence the evaluation results. Is our improvement significant, or does it mean nothing? Going from 24.7% to 30.3% +5.5% on GPQA Diamond in the very first epoch while training purely on CPU… Is this expected behavior, a genuinely solid result, or could it be a symptom of an underlying issue like data leakage/overfitting that we are missing? We are doing this to master the art of Fine Tuning, and our ultimate goal is to share and gift all of this knowledge to the community down the road. These questions might seem obvious to many, but they are completely new to us. Thank you so much for your time and for helping us learn