Is a +5.5% improvement on GPQA Diamond normal when full fine-tuning an 8B model on CPU with recycled hardware?

Hi everyone!

We are a team of AI enthusiasts who started learning Linux and understanding how language models work at the beginning of this year. We are not experts or professionals; we do this purely as a hobby and out of a desire to learn and contribute.

We have built a home server (Z6 architecture) using old, second-hand parts. Due to our limited resources, we run our Full Fine-Tuning tests on 8-billion parameter models (8B) using strictly CPU and RAM.

Our current goal is to understand and validate if our approach is moving in the right direction. For training, we used a dataset of 144,000 words structured into 6 independent blocks (roughly ~42,000 real tokens per block). Processing these 6 blocks marks the completion of Epoch 1, which corresponds to our Checkpoint-6.

To evaluate our progress, we used the lm-eval

library with a task we believe is highly objective: GPQA Diamond Zeroshot (graduate-level questions).

1. Base Model Evaluation (Dolphin 3.0 Llama 3.1 8B Puro FP32)

This is the exact command we executed on our machine:

lm_eval run \
  --model hf \
  --model_args pretrained=/xxxx/Dolphin3.0-Llama3.1-8b-Puro-FP32 \
  --tasks gpqa_diamond_zeroshot \
  --num_fewshot 0 \
  --batch_size 1 \
  --device cpu

Results obtained:

Task: gpqa_diamond_zeroshot | Version: 2.2

Metric / Value: acc: 0.2475 (24.7%) | acc_norm: 0.2475

Stderr: ±0.0307

2. Our Full Fine-Tuning Evaluation (Checkpoint-6 / Epoch 1)

lm_eval run \
  --model hf \
  --model_args pretrained=/xxxx/checkpoint-6 \
  --tasks gpqa_diamond_zeroshot \
  --num_fewshot 0 \
  --batch_size 1 \
  --device cpu

Results obtained:

Task: gpqa_diamond_zeroshot | Version: 2.2

Metric / Value: acc: 0.303 (30.3%) | acc_norm: 0.303

Stderr: ±0.0327

Our questions for the community (Could you help us understand?):

Since we are learning step by step, we would truly appreciate it if you could help us confirm two things:

Is the base model score accurate? We want to know if anyone running this exact command for Dolphin 3.0 on different hardware gets the same ~24.7% accuracy, or if hardware differences can influence the evaluation results.

Is our improvement significant, or does it mean nothing? Going from 24.7% to 30.3% (+5.5%) on GPQA Diamond in the very first epoch while training purely on CPU… Is this expected behavior, a genuinely solid result, or could it be a symptom of an underlying issue (like data leakage/overfitting) that we are missing?

We are doing this to master the art of Fine Tuning, and our ultimate goal is to share and gift all of this knowledge to the community down the road.

These questions might seem obvious to many, but they are completely new to us.

Thank you so much for your time and for helping us learn!