{"slug": "is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on", "title": "Is a +5.5% improvement on GPQA Diamond normal when full fine-tuning an 8B model on CPU with recycled hardware?", "summary": "A hobbyist team reported a +5.5% improvement on GPQA Diamond Zeroshot after full fine-tuning an 8B model on CPU with recycled hardware, moving from 24.7% to 30.3% accuracy in the first epoch. The team, using Dolphin 3.0 Llama 3.1 8B Puro FP32 as the base model, asked the community whether the base score is accurate and whether the improvement is significant or indicates data leakage/overfitting.", "body_md": "Hi everyone!\n\nWe are a team of AI enthusiasts who started learning Linux and understanding how language models work at the beginning of this year. We are not experts or professionals; we do this purely as a hobby and out of a desire to learn and contribute.\n\nWe have built a home server (Z6 architecture) using old, second-hand parts. Due to our limited resources, we run our Full Fine-Tuning tests on 8-billion parameter models (8B) using strictly CPU and RAM.\n\nOur current goal is to understand and validate if our approach is moving in the right direction. For training, we used a dataset of 144,000 words structured into 6 independent blocks (roughly ~42,000 real tokens per block). Processing these 6 blocks marks the completion of Epoch 1, which corresponds to our Checkpoint-6.\n\nTo evaluate our progress, we used the `lm-eval`\n\nlibrary with a task we believe is highly objective: GPQA Diamond Zeroshot (graduate-level questions).\n\n**1. Base Model Evaluation (Dolphin 3.0 Llama 3.1 8B Puro FP32)**\n\nThis is the exact command we executed on our machine:\n\n```\nlm_eval run \\\n  --model hf \\\n  --model_args pretrained=/xxxx/Dolphin3.0-Llama3.1-8b-Puro-FP32 \\\n  --tasks gpqa_diamond_zeroshot \\\n  --num_fewshot 0 \\\n  --batch_size 1 \\\n  --device cpu\n```\n\nResults obtained:\n\nTask: gpqa_diamond_zeroshot | Version: 2.2\n\nMetric / Value: acc: 0.2475 (24.7%) | acc_norm: 0.2475\n\nStderr: ±0.0307\n\n**2. Our Full Fine-Tuning Evaluation (Checkpoint-6 / Epoch 1)**\n\n```\nlm_eval run \\\n  --model hf \\\n  --model_args pretrained=/xxxx/checkpoint-6 \\\n  --tasks gpqa_diamond_zeroshot \\\n  --num_fewshot 0 \\\n  --batch_size 1 \\\n  --device cpu\n```\n\nResults obtained:\n\nTask: gpqa_diamond_zeroshot | Version: 2.2\n\nMetric / Value: acc: 0.303 (30.3%) | acc_norm: 0.303\n\nStderr: ±0.0327\n\n**Our questions for the community (Could you help us understand?):**\n\nSince we are learning step by step, we would truly appreciate it if you could help us confirm two things:\n\n**Is the base model score accurate?** We want to know if anyone running this exact command for Dolphin 3.0 on different hardware gets the same ~24.7% accuracy, or if hardware differences can influence the evaluation results.\n\n**Is our improvement significant, or does it mean nothing?** Going from 24.7% to 30.3% (+5.5%) on GPQA Diamond in the very first epoch while training purely on CPU… Is this expected behavior, a genuinely solid result, or could it be a symptom of an underlying issue (like data leakage/overfitting) that we are missing?\n\nWe are doing this to master the art of Fine Tuning, and our ultimate goal is to share and gift all of this knowledge to the community down the road.\n\nThese questions might seem obvious to many, but they are completely new to us.\n\nThank you so much for your time and for helping us learn!", "url": "https://wpnews.pro/news/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on", "canonical_source": "https://discuss.huggingface.co/t/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on-cpu-with-recycled-hardware/179434#post_1", "published_at": "2026-08-29 19:43:41+00:00", "updated_at": "2026-08-29 19:49:19.978574+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Dolphin 3.0 Llama 3.1 8B Puro FP32", "GPQA Diamond Zeroshot", "lm-eval"], "alternates": {"html": "https://wpnews.pro/news/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on", "markdown": "https://wpnews.pro/news/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on.md", "text": "https://wpnews.pro/news/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on.txt", "jsonld": "https://wpnews.pro/news/is-a-5-5-improvement-on-gpqa-diamond-normal-when-full-fine-tuning-an-8b-model-on.jsonld"}}