Over the past few weeks, I have been conducting a study to evaluate small AI language models (100M ≤ x ≤ 3B parameters) in order to determine their memory limits for two specific tasks: retrieval and reconstruction. The goal is to identify models that are well suited for local deployment on low-spec laptops and PCs.
To achieve this, I designed a simple benchmark focused specifically on these two tasks. I then evaluated three Qwen2.5 models: 0.5B, 1.5B, and 3B.
After completing the experiments and analyzing the results, I was not surprised to find that their retrieval memory performance was relatively similar across all three models.
However, I was genuinely surprised by the reconstruction memory results. Unexpectedly, Qwen2.5-3B performed the worst in this benchmark, showing a much more significant decline in reconstruction performance as the amount of provided information increased compared with both the 1.5B and even the 0.5B models.
Because of this unexpected finding, I would appreciate any suggestions on additional analyses, experiments, or methodological checks that I should perform to determine whether this result is truly objective rather than the consequence of an experimental error or some other confounding factor.
I am open to feedback of any kind and would greatly appreciate all type of insights.