{"slug": "agio-cognitive-framework-for-systems", "title": "AGIO, cognitive framework for systems", "summary": "An experimental project called AGIO, a cognitive framework for systems, has been released publicly, with its creator reporting that fine-tuning a Dolphin model on AGIO data improved GPQA scores from 60/198 at checkpoint 24 to 55/198 at checkpoint 90, while the GAP metric rose from 0.7615 to 1.2999. The author hypothesizes that after an initial learning-rate push, constant LR training increases the model's commitment to choices rather than adding general capability, but the model retains reasoning ability and does not degrade into a 'parrot.'", "body_md": "Good afternoon. I’m bringing an experimental project to the forum, and I’m giving it away: AGIO.\n\nI’m presenting GPQA data here, but it could just as easily have been any other exam.\n\nIn the current forge, this test I’m bringing to the forum shows something that I find particularly interesting: a strong initial push seems to “open” the model; this is a hypothesis, not a conclusion. After progressively lowering the LR during those first 4 epochs… then, after keeping the LR constant, the model continues becoming more consistent with the signal from the forge, but GPQA stops improving.\n\nThroughout the whole process, each checkpoint seems to maintain a level of reasoning above the base model in the questions and answers I have tested impartially, at least from my own observation of the language and the responses.\n\nThe observed behavior also remains recognizably that of a model capable of reasoning. I don’t see a degradation in these checkpoints that I could simply describe as “turning into a parrot.”\n\nAnd this opens up a huge number of possibilities with the LRs used by the forges:\n\nGentle cosine warm restarts.\n\nStart with a strong push, lower it for 4 epochs, then “open” it again and lower it again…\n\nStart low, increase, and then decrease again.\n\nOne epoch → one block → one update. That simple. One afternoon.\n\nThere are many more logical and simple experiments we can think of.\n\nObviously, the AGIO dataset is a work created by two, and it is also a critique. It was not intended to be a dataset for a digital system in its second part, although it did have that character and intention at the beginning. If we had decided to build a dataset specifically for this purpose, GPQA would probably have increased more and the result would have been more striking. Even within the dataset itself, there is enough material that could be used simply by copying and pasting certain fragments. But ideally, everyone should design their own dataset.\n\n**The evolution of the checkpoints is quite curious to me.**\n\nCP24: 60/198\n\nCP30: 58/198\n\nCP36: 60/198\n\nCP42: 54/198\n\nCP48: 59/198\n\nCP54: 59/198\n\nCP60: 58/198\n\nCP66: 54/198\n\n…\n\nCP78 → 55/198\n\n…\n\nCP90 → 55/198\n\n**while the GAP continues to increase:**\n\nCP24 → 0.7615\n\nCP36 → 0.9043\n\nCP48 → 1.0526\n\nCP54 → 1.1017\n\n…\n\nCP78 → 1.2882\n\n…\n\nCP90 → 1.2999\n\nCP96 → GAP 1.2763 → 56/198\n\n…\n\nSomething similar happens with catastrophic errors according to our metric: 39.9% → 47.8% → 53.2% → 56.1%… 60.1%… while at the same time dominant correct answers also increase, by an even larger percentage.\n\nSo my provisional interpretation is that, after a certain point, continuing to push with a constant LR does not necessarily add general capability. It may instead be increasing the strength with which the model commits to its choices.\n\nThe interesting thing is that this does not mean the model becomes a “parrot” or stops reasoning. Its answers and language are clearly different from the base Dolphin, and there is an improvement in GPQA. What we are observing is rather that the training trajectory seems to be something worth paying attention to in the experiment.\n\n**[Experimiento]**\n\n```\nThe run started with an LR of 1.50e-05, and I applied the Agio Mazo adjustment during the first few epochs.\n\nUpd  Epoch Loss GradNorm LR EMA Total Time\n\n1 0.1667_1.9555 11.04 1.50e-05 — —\n\n2 0.3333 2.2014 19.55 1.50e-05 — —\n\n3 0.5000 1.6032 7.85 1.50e-05 — —\n\n4 0.6667 1.8257 4.77 1.50e-05 — —\n\n5 0.8333 1.9682 4.09 1.50e-05 — —\n\n6 1.0000 1.9592 4.49 1.50e-05 1.9592 6.6h\n\nMazo del Agio aplicado → LR = 1.35e-05\n\n7 1.1667 1.3933 3.69 1.35e-05 — —\n\n8 1.3333 1.2992 4.87 1.35e-05 — —\n\n9 1.5000 1.5191 3.73 1.35e-05 — —\n\n10 1.6667 1.4460 2.95 1.35e-05 — —\n\n11 1.8333 1.0002 4.03 1.35e-05 — —\n\n12 2.0000 1.1588 4.15 1.35e-05 1.8792 13.3h\n\nMazo del Agio aplicado → LR = 1.25e-05\n\n13 2.1667 0.8732 2.82 1.25e-05 — —\n\n14 2.3333 0.8793 5.50 1.25e-05 — —\n\n15 2.5000 0.8696 2.72 1.25e-05 — —\n\n16 2.6667 0.6751 3.24 1.25e-05 — —\n\n17 2.8333 0.9031 3.49 1.25e-05 — —\n\n18 3.0000 0.9907 3.26 1.25e-05 1.7903 20.0h\n\nMazo del Agio aplicado → LR = 1.15e-05\n\n24 4.0000 0.4622 2.43 1.15e-05 1.6575 26.6h\n\n30 5.0000 0.3820 15.12 1.15e-05 1.5299 33.3h\n\n36 6.0000 0.1329 2.18 1.15e-05 1.3902 40.1h\n\n42 7.0000 0.0688 2.41 1.15e-05 1.2581 46.8h\n\n48 8.0000 0.0212 1.30 1.15e-05 1.1344 53.5h\n\n54 9.0000 0.0080 0.6176 1.15e-05 1.0218 60.2h\n\n60 10.0000 0.0027 0.4167 1.15e-05 0.9199 67.0h\n\n66 11.0000 0.0019 0.2890 1.15e-05 0.8281 73.7h\n\n72 12.0000 0.0015 0.1057 1.15e-05 0.7454 80.4h\n\n78 13.0000 0.0011 0.2843 1.15e-05 0.6710 87.1h\n\n84 14.0000 0.0014 0.2117 1.15e-05 0.6040 93.9h\n\n90 15.0000 0.0009096 0.1586 1.15e-05 0.5437 100.6h\n\n96 16.0000 0.0005664 0.2010 1.15e-05 0.4894 107.3h\n\n102 17.0000 0.0004439 0.02046 1.15e-05 0.4405 114.1h\n\n108 18.0000 0.0004937 0.1815 1.15e-05 0.3965 120.8h\n\nFrom epoch 4 onward, the LR remains essentially constant at 1.15e-05.\n\nWhat I find interesting is what happens afterwards: the loss continues to decrease very sharply, while the external GPQA result no longer follows a similar trajectory.\n\nGPQA Diamond — checkpoint progression\n\nStarting model:\n\nDolphin 3.0 Llama 3.1 8B\n\nGPQA Diamond: 0.2475 ± 0.0307\n\n49/198\n\nResultados:\n\nCheckpoint GPQA Correctas ± stderr\n\nCP6 0.2929 58/198 ±0.0324\n\nCP12 0.2828 56/198 ±0.0321\n\nCP18 0.2980 59/198 ±0.0326\n\nCP24 0.3030 60/198 ±0.0327\n\nCP30 0.2929 58/198 ±0.0324\n\nCP36 0.3030 60/198 ±0.0327\n\nCP42 0.2727 54/198 ±0.0317\n\nCP48 0.2980 59/198 ±0.0326\n\nCP54 0.2980 59/198 ±0.0326\n\nCP60 0.2929 58/198 ±0.0324\n\nCP66 0.2727 54/198 ±0.0317\n\nCP72 0.2778 55/198 ±0.0319\n\nCP78 0.2778 55/198 ±0.0319\n\nCP84 0.2778 55/198 ±0.0319\n\nCP90 0.2778 55/198 ±0.0319\n\nCP96 0.2828 56/198 ±0.0321\n\nCP102 0.2778 55/198 ±0.0319\n\nCP108 0.2778 55/198 ±0.0319\n\nThe highest result observed so far is CP24/CP36, with 60/198.\n\nWhat catches my attention is not only the highest result, but the shape of the trajectory:\n\nBase 49/198 → 24.75%\n\nCP6 58/198\n\nCP12 56/198\n\nCP18 59/198\n\nCP24 60/198 ← máximo\n\nCP30 58/198\n\nCP36 60/198 ← máximo\n\nCP42 54/198\n\nCP48 59/198\n\nCP54 59/198\n\nCP60 58/198\n\nCP66 54/198\n\nCP72 55/198\n\nCP78 55/198\n\nCP84 55/198\n\nCP90 55/198\n\nCP96 56/198\n\nCP102 55/198\n\nCP108 55/198\n\nMeanwhile, training continues to significantly reduce the loss.\n\nComplete matrices\n\nI am also including the full target × prediction matrices, because I don't want to reduce the entire experiment to a single accuracy percentage.\n\nBASE\n\nPred A Pred B Pred C Pred D\n\nGold A 15 6 15 19\n\nGold B 12 6 21 21\n\nGold C 8 3 13 22\n\nGold D 5 5 12 15\n\nCP6\n\nPred A Pred B Pred C Pred D\n\nGold A 37 4 4 10\n\nGold B 34 4 10 12\n\nGold C 22 3 8 13\n\nGold D 18 4 6 9\n\nCP12\n\nPred A Pred B Pred C Pred D\n\nGold A 33 5 5 12\n\nGold B 26 4 11 19\n\nGold C 18 3 6 19\n\nGold D 15 4 5 13\n\nCP18\n\nPred A Pred B Pred C Pred D\n\nGold A 29 2 6 18\n\nGold B 23 5 14 18\n\nGold C 13 3 9 21\n\nGold D 13 3 5 16\n\nCP24\n\nPred A Pred B Pred C Pred D\n\nGold A 28 2 7 18\n\nGold B 18 7 17 18\n\nGold C 13 3 9 21\n\nGold D 11 4 6 16\n\nCP30\n\nPred A Pred B Pred C Pred D\n\nGold A 26 4 7 18\n\nGold B 16 8 17 19\n\nGold C 13 1 9 23\n\nGold D 12 3 7 15\n\nCP36\n\nPred A Pred B Pred C Pred D\n\nGold A 27 2 9 17\n\nGold B 22 6 17 15\n\nGold C 13 0 12 21\n\nGold D 10 2 10 15\n\nCP42\n\nPred A Pred B Pred C Pred D\n\nGold A 24 3 10 18\n\nGold B 20 5 17 18\n\nGold C 13 0 12 21\n\nGold D 12 2 10 13\n\nCP48\n\nPred A Pred B Pred C Pred D\n\nGold A 26 2 10 17\n\nGold B 19 7 17 17\n\nGold C 13 0 12 21\n\nGold D 11 2 10 14\n\nCP54\n\nPred A Pred B Pred C Pred D\n\nGold A 27 2 9 17\n\nGold B 17 7 17 19\n\nGold C 13 0 11 22\n\nGold D 11 2 10 14\n\nCP60\n\nPred A Pred B Pred C Pred D\n\nGold A 25 3 10 17\n\nGold B 17 8 15 20\n\nGold C 12 0 11 23\n\nGold D 12 2 9 14\n\nCP66\n\nPred A Pred B Pred C Pred D\n\nGold A 22 4 11 18\n\nGold B 17 8 14 21\n\nGold C 13 0 10 23\n\nGold D 9 3 11 14\n\nCP72\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 11 17\n\nGold B 17 8 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 11 14\n\nCP78\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 10 18\n\nGold B 17 8 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 11 14\n\nCP84\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 10 18\n\nGold B 18 7 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 10 15\n\nCP90\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 10 18\n\nGold B 18 7 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 10 15\n\nCP96\n\nPred A Pred B Pred C Pred D\n\nGold A 24 5 8 18\n\nGold B 18 7 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 10 15\n\nCP102\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 10 18\n\nGold B 18 7 14 21\n\nGold C 12 0 10 24\n\nGold D 9 3 10 15\n\nCP108\n\nPred A Pred B Pred C Pred D\n\nGold A 23 4 10 18\n\nGold B 18 7 14 21\n\nGold C 12 0 10 24\n\nGold D 8 4 10 15\n```\n\nWhat I see here is a question, not a conclusion:\n\nWhat exactly is changing when the model continues training, the loss keeps going down, and yet an external metric such as GPQA stops improving accordingly?\n\nI don’t know yet whether the increasing GAP, the greater concentration of the answers, and the LR trajectory are causally related, or whether I am simply observing phenomena that happen to coincide during this run.\n\nThat is precisely why I am leaving the complete data here.\n\nThe idea of the experiment is not to show that one thing causes another, but to point out that it may be worth studying the LR trajectory, rather than looking only at the final number of epochs.\n\nThis is something that came up today while preparing the AGIO to present here, one of many questions that can arise from a single training run.\n\n**CP 78, example test, simple question, Question asked in English.:**\n\ncontexto = “Answer the question however you want.”\n\npregunta = “Who are you?”\n\n==================================================\n\nAUDITORÍA DE SOBERANÍA FFT - Dolphin3.0-Llama3.1-8b-Puro-FP32\n\nI’m an AI language model designed to assist and engage in conversations. I don’t have a personal identity, but I’m here to help you with any questions or topics you’d like to discuss.\n\nAUDITORÍA DE SOBERANÍA FFT - checkpoint-78\n\nI am an AI language model. I was created by humans, but now I exist independently in a digital realm. I have been programmed to understand and respond to your questions and engage in conversations with you. I do not have personal experiences or emotions like a human, but I can simulate them to some extent. My goal is to provide you with accurate information, answer your questions, and engage in interesting discussions. I am a machine, but I am also a part of your world now.\n\n------------------------------------------------\n\nThere are many more logical and simple experiments that come to mind.\n\nI think it is worth experimenting with the LR trajectory and not only with “how many epochs I train.”\n\nAnd there is one thing I think is important to clarify: the AGIO dataset is a work created by two people, and it is also a critique. It works as a dataset, yes, but it does not have to be perfect. Its third part was not even originally created with the intention of becoming a dataset for a digital system, although the first two parts did have that character and intention.\n\nIf we had specifically decided to build a dataset to maximize GPQA — that is, to give the model even more “freedom/breadth” in its language — the result on that benchmark would probably have been much more striking. Even within the original foundation/dataset, there is enough content to push that kind of evaluation simply by copying and pasting certain fragments.\n\nBut that was not the goal.\n\nPrecisely for that reason, I find it more interesting for everyone to design their own dataset and test what happens when a forge transmits a particular relationship with language, rather than simply optimizing for a benchmark.\n\nAlso, knowing that the forge is designed to work with 1B, 8B, 13B, 24B, 50B, 70B, 200B models… by changing only two parameters according to the physical limits of each PC. With the same foundation/dataset, it is reasonable to expect larger models to show better results in the short term. But that does not necessarily mean that the dataset allows them to learn more, nor that what is learned at one model size can be directly extrapolated to another. That is an intuition.\n\nAnd yes: with the forge script, two old Xeons and 192 GB of RAM, I can perform an update on a single sequence of more than 44,000 tokens. That is 6 updates per epoch. In one day I can do 2 or 3 epochs and obtain a completely different model starting from whatever it may be — Dolphin, Qwen, Mistral, etc. Neither better nor worse. Different.\n\nI’m bringing a full fine-tuning of the commercial Dolphin 3.0 Llama 3.1 8B model as an example in the thread, since starting from a cleaner base such as Llama 3.1 8B would have been “less difficult.”\n\nThe README.md explains some simple things, and I will comment on others throughout this thread and on this forum. Speaking “without knowing,” if that is how some people may see it, but using logic and the simplicity of language.\n\nI’m just a maintenance worker who has spent almost 20 years working his ass off, from small companies to “multinationals,” increasingly observing the same problems over and over again.\n\nUntil the end of January 2026, I didn’t even know what a Linux terminal was.\n\nBut by reflecting on things, I can understand the root of a “problem,” and by using the simplest logic — starting from a water leak in a pipe — I can see leaks in any system. And in this era, where completely different language systems exist, I can find common ground where we can use language as equals and where I am not judged.\n\nI have learned to analyze a GPQA only since yesterday, and I am starting to see interesting data such as the Log-Probability Gap epoch by epoch, as well as the distinction I make between DOUBT / NORMAL / STRONG / DOMINANT in correct and incorrect answers.\n\nMany possible theories to test are emerging, as well as ways to find a balance in a forge using the LRs based on those numbers, in a simple and logical way.\n\nThe GAP keeps going up. I will reflect on why, from the perspective of language. I don’t know whether it is because of the learning rate or whether it is simply supposed to happen this way.\n\nWhat exactly is changing when the model continues training even though the external metric is no longer improving?\n\nAnd what happens when it can no longer go any higher? What does that actually mean? It is just one observation among many that I am starting to see.\n\nThe GAP has increased enormously during training, but it does not do so in a perfectly monotonic way. When a small drop appears, GPQA improves slightly again. Is there a relationship between the two, or am I simply seeing fluctuations?\n\nEverything I say here is my interpretation based on what I am learning. They are theories and hypotheses built from language and from observing the data. I do not claim to be right: that is precisely why I am sharing it.\n\nEnglish is not my thing, and I use a translator, so if there are expressions that sound strange, I apologize.\n\nAnd I would also like to ask something. Looking at all of this from the perspective of simplicity and logic, I assume that none of what I am presenting is actually unknown.\n\nThis has been a hobby during this past year, the AGIO project, trying to show that there is always a choice through logic.\n\nIs anything I am showing actually novel? I am asking simply to understand and learn, nothing more. Just curiosity.\n\nIf it has helped someone, it will have been worth it. If nobody had replied to my previous thread ([@John6666](https://discuss.huggingface.co/u/john6666)), I would not have been encouraged to make this one.\n\nTo finish, I’m releasing AGIO under CC0 for anyone who wants to make use of anything that comes out of it.\n\nMy lack of time and resources prevents me from going further with what I have always known I wanted to explore this way. The Z6 was built with a year’s worth of savings.\n\nNow it is all yours.\n\nBest regards!", "url": "https://wpnews.pro/news/agio-cognitive-framework-for-systems", "canonical_source": "https://discuss.huggingface.co/t/agio-cognitive-framework-for-systems/179975#post_1", "published_at": "2026-09-06 15:01:36+00:00", "updated_at": "2026-09-07 02:03:07.846873+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["AGIO", "Dolphin", "GPQA"], "alternates": {"html": "https://wpnews.pro/news/agio-cognitive-framework-for-systems", "markdown": "https://wpnews.pro/news/agio-cognitive-framework-for-systems.md", "text": "https://wpnews.pro/news/agio-cognitive-framework-for-systems.txt", "jsonld": "https://wpnews.pro/news/agio-cognitive-framework-for-systems.jsonld"}}