AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining

AQL-Lite, a framework from researchers behind arXiv paper 2610.30344v1, raises a four-benchmark Arabic mean by 3.51 points over continual pretraining on raw web text at a matched 1B-token budget, while raw-web CPT yields no gain over the base model. In a controlled 5B-token comparison, AQL-Lite recovers 94% of full-parameter CPT's gain (mean 73.19 vs. 73.58) in 52% of the wall-clock time, and scaled to 40B Modern Standard Arabic tokens with instruction tuning on one million examples, the resulting 12B model scores 75.3 on the seven-benchmark Open Arabic LLM Leaderboard v2 suite. The method pairs a quality-first curation pipeline with partial-parameter CPT that trains RS-LoRA adapters, the last eight transformer blocks and the embedding matrix under layer-wise learning rates.

arXiv:2610.30344v1 Announce Type: new Abstract: Training Arabic large language models faces challenges from noisy web data, dialectal variation, and the high cost of continual pretraining CPT . We introduce AQL-Lite AQL: Arabic Quality Language , a time- and resource-efficient framework for adapting a multilingual LLM to Modern Standard Arabic MSA . AQL-Lite couples a quality-first curation pipeline structural cleaning, heuristic and model-based filtering, MSA selection, and deduplication with a partial-parameter CPT recipe that trains RS-LoRA adapters across the network together with the last eight transformer blocks and the embedding matrix, under layer-wise learning rates. At a matched 1B-token budget, CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text, which yields no gain over the base model. In a controlled 5B-token comparison, AQL-Lite recovers 94% of the gain of full-parameter CPT mean 73.19 vs. 73.58 in 52% of the wall-clock time and outperforms adapter-only baselines on every benchmark, and a component ablation shows that adapters, unfrozen blocks, and embedding training each contribute. Scaled to 40B MSA tokens and instruction-tuned on one million examples, the resulting 12B model scores 75.3 on the seven-benchmark Open Arabic LLM Leaderboard v2 OALL v2 suite. These results suggest that partial-parameter CPT on a carefully curated corpus is a practical route to efficient MSA adaptation.