{"slug": "need-feedback-on-my-research", "title": "Need feedback on my research", "summary": "A researcher evaluating small AI language models for local deployment found that Qwen2.5-3B performed worst in reconstruction memory, despite being the largest model tested. The study compared Qwen2.5 models of 0.5B, 1.5B, and 3B parameters on retrieval and reconstruction tasks, with the 3B model showing a significant decline in reconstruction as information increased. The researcher seeks feedback on whether the result is valid or due to experimental error.", "body_md": "Over the past few weeks, I have been conducting a study to evaluate small AI language models (100M ≤ x ≤ 3B parameters) in order to determine their memory limits for two specific tasks: retrieval and reconstruction. The goal is to identify models that are well suited for local deployment on low-spec laptops and PCs.\n\nTo achieve this, I designed a simple benchmark focused specifically on these two tasks. I then evaluated three Qwen2.5 models: 0.5B, 1.5B, and 3B.\n\nAfter completing the experiments and analyzing the results, I was not surprised to find that their retrieval memory performance was relatively similar across all three models.\n\nHowever, I was genuinely surprised by the reconstruction memory results. Unexpectedly, Qwen2.5-3B performed the worst in this benchmark, showing a much more significant decline in reconstruction performance as the amount of provided information increased compared with both the 1.5B and even the 0.5B models.\n\nBecause of this unexpected finding, I would appreciate any suggestions on additional analyses, experiments, or methodological checks that I should perform to determine whether this result is truly objective rather than the consequence of an experimental error or some other confounding factor.\n\nI am open to feedback of any kind and would greatly appreciate all type of insights.", "url": "https://wpnews.pro/news/need-feedback-on-my-research", "canonical_source": "https://discuss.huggingface.co/t/need-feedback-on-my-research/177694#post_1", "published_at": "2026-07-12 05:39:02+00:00", "updated_at": "2026-07-12 05:44:58.356316+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["Qwen2.5"], "alternates": {"html": "https://wpnews.pro/news/need-feedback-on-my-research", "markdown": "https://wpnews.pro/news/need-feedback-on-my-research.md", "text": "https://wpnews.pro/news/need-feedback-on-my-research.txt", "jsonld": "https://wpnews.pro/news/need-feedback-on-my-research.jsonld"}}