{"slug": "unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on", "title": "Unsloth vs Axolotl vs TRL vs LLaMA-Factory: A Fine-Tuning Framework Comparison on Speed, VRAM, and Multi-GPU", "summary": "Unsloth achieves up to 7.3x training speedup over standard Transformers for MoE models like gpt-oss-20B on an NVIDIA B200, according to published benchmarks, while Axolotl delivers up to 1.45x speedup and 30% memory reduction via custom Triton kernels. TRL remains the reference baseline that Axolotl and LLaMA-Factory build upon, and LLaMA-Factory covers 100+ LLMs and VLMs with a zero-code web UI.", "body_md": "Four open source projects dominate LLM fine-tuning today. [Unsloth](https://github.com/unslothai/unsloth), [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), [TRL](https://github.com/huggingface/trl), and [LLaMA-Factory](https://github.com/hiyouga/LlamaFactory) all wrap the same underlying PyTorch and Hugging Face stack. They diverge on where they spend engineering effort.\n\nUnsloth rewrites kernels. Axolotl composes parallelism strategies. TRL defines the trainer APIs the others build on. LLaMA-Factory optimizes for breadth of model coverage and zero-code operation.\n\nThis comparison covers three axes engineers actually hit: training throughput, peak VRAM, and multi-GPU scaling.\n\n**Understand each framework **\n\n**TRL** is the reference implementation layer. It ships`SFTTrainer`\n\n,`DPOTrainer`\n\n,`GRPOTrainer`\n\n,`KTOTrainer`\n\n,`RewardTrainer`\n\n, and`RLOOTrainer`\n\n. Axolotl and LLaMA-Factory both call into it. The current stable release line is v1.8.0.**Unsloth** replaces parts of the modeling code with hand-written Triton kernels. Backpropagation steps are manually derived rather than autograd-generated.[Hugging Face’s own writeup](https://huggingface.co/blog/unsloth-trl)notes accuracy degradation is 0% versus standard QLoRA, because no approximations are introduced.**Axolotl** is a YAML-driven wrapper over Transformers, PEFT, TRL, Accelerate, and DeepSpeed. Its differentiator is composability of parallelism strategies, not kernel work.**LLaMA-Factory** is an[ACL 2024 system demonstration paper](https://arxiv.org/abs/2403.13372)with a Gradio web UI called LlamaBoard. The repository covers 100+ LLMs and VLMs.\n\n**Speed**\n\n**Unsloth: kernel-level gains on a single GPU**\n\nUnsloth's [published benchmarks](https://unsloth.ai/docs/basics/unsloth-benchmarks) show 2x training speed for Llama 3.1 8B and Llama 3.3 70B. The setup used the Alpaca dataset, batch size 2, and gradient accumulation 4. QLoRA ran at rank 32 on all linear layers.\n\nThe [MoE results](https://unsloth.ai/docs/basics/faster-moe) are larger. Unsloth fine-tuned `unsloth/gpt-oss-20b-BF16`\n\non an NVIDIA B200. It reports 712.33 ms per step at 8K context, versus 5,226.86 ms for Transformers v5. That is a 7.3x gap. At 4K the gap is 4.82x, and at 1K it is only 1.37x.\n\nThe trend direction is model-dependent, and Unsloth's docs scope this claim to gpt-oss. There, the speedup grows with sequence length, credited to Flex Attention and the MoE kernels.\n\nQwen3-30B-A3B on B200 runs the other way. Its reported speedup falls from 1.7x at 1K to 1.1x at 16K. Memory savings move the opposite direction, rising from about 2% to 15%.\n\nQwen3-30B-A3B on H100 reaches up to 1.77x. GLM-4.7-Flash on RTX PRO 6000 reaches 2.1x. A [collaboration with AMD](https://unsloth.ai/docs/basics/amd) measured Llama-3.1-8B LoRA SFT at 2.07 s/step. TRL plus FlashAttention-2 took 2.87 s/step, a 1.39x gap with matching loss curves.\n\n**Axolotl: kernels borrowed, parallelism native**\n\nAxolotl added [custom Triton kernels and autograd functions](https://axolotlai.substack.com/p/accelerating-lora-fine-tuning-with) for LoRA in February 2025, explicitly citing Unsloth as inspiration. They are opt-in through `lora_mlp_kernel`\n\n, `lora_qkv_kernel`\n\n, and `lora_o_kernel`\n\n.\n\nRecent [release notes](https://github.com/axolotl-ai-cloud/axolotl/releases) add SonicMoE LoRA support. It delivers up to 1.45x speedup and 30% memory reduction over a `grouped_mm`\n\nbaseline. That figure is for Qwen3.5-35B-A3B 8-bit LoRA on a single H100 SXM.\n\nAxolotl also ships FlashAttention 2/3/4, xFormers, Flex Attention, SageAttention, Liger Kernel, Cut Cross Entropy, and ScatterMoE.\n\n**TRL: the baseline everyone measures against**\n\nTRL is usually the reference point rather than the winner on raw single-GPU throughput. It compensates with breadth of memory and speed levers documented in [Reducing Memory Usage](https://huggingface.co/docs/trl/en/reducing_memory_usage) and [Speeding Up Training](https://huggingface.co/docs/trl/en/speeding_up_training).\n\nThose levers include packing, padding-free batching, truncation, [Liger Kernel](https://huggingface.co/docs/trl/en/liger_kernel_integration), and vLLM sleep mode for GRPO. TRL also has a [first-party Unsloth integration](https://huggingface.co/docs/trl/en/unsloth_integration), so the two are not mutually exclusive.\n\n**LLaMA-Factory: speed by delegation**\n\nLLaMA-Factory does not write its own kernels. It exposes other people's work through config flags.\n\nSetting `use_unsloth: true`\n\nactivates the Unsloth patch. The project's changelog reports 170% relative speed from that path. Unsloth's long-sequence training is listed at 117% speed and 50% memory. It also supports `enable_liger_kernel: true`\n\nand FlashAttention-2 via `flash_attn: fa2`\n\n.\n\n**VRAM**\n\n**Reported memory floors**\n\nUnsloth publishes a [VRAM requirements table](https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements) sorted by parameter count. It lists 6 GB for an 8B model in 4-bit QLoRA and 41 GB for 70B. LoRA at 16-bit costs 22 GB and 164 GB for the same models.\n\nLLaMA-Factory's [README hardware table](https://github.com/hiyouga/LlamaFactory) covers the same regime for 4-bit QLoRA. It lists 6 GB at 7B, 24 GB at 30B, and 48 GB at 70B. Full bf16 fine-tuning of 70B is listed at 600 GB.\n\nBoth tables describe minimums. Batch size, sequence length, and optimizer choice move the real number.\n\n**Context length is the sharper differentiator**\n\nPeak VRAM at a fixed context is less interesting than the maximum context a given VRAM budget allows. Unsloth's [context length benchmarks](https://unsloth.ai/docs/basics/unsloth-benchmarks) for Llama 3.1 8B QLoRA at rank 32 and batch size 1 are stark.\n\n| GPU VRAM | Unsloth context | Transformers + FA2 context |\n|---|---|---|\n| 8 GB | 2,972 | OOM |\n| 16 GB | 40,724 | 2,551 |\n| 24 GB | 78,475 | 5,789 |\n| 48 GB | 191,728 | 15,502 |\n| 80 GB | 342,733 | 28,454 |\n\nUnsloth attributes this to its gradient checkpointing algorithm combined with Apple's Cut Cross Entropy. For Llama 3.3 70B on an 80 GB A100, it reports 89,389 tokens. The FA2 baseline reaches 6,916.\n\n**The MoE memory story**\n\nMoE training is where memory behavior has shifted most in 2026. Unsloth reports `gpt-oss-20b`\n\nfine-tuning inside 12.8 GB, while Qwen3-30B-A3B at 16-bit LoRA needs 63 GB.\n\nIts B200 gpt-oss run used 47.43 GB at 8K context where Transformers v5 used 73.80 GB. At 16K, Transformers v5 went out of memory and Unsloth used 55.13 GB.\n\nThe mechanism is a split-LoRA formulation. PEFT materializes the LoRA delta across all experts before the MoE matmul. Unsloth reorders the operations instead, which is mathematically identical but avoids the materialization.\n\nAxolotl attacks the same problem differently. Its [MoE expert quantization](https://docs.axolotl.ai/docs/expert_quantization.html) quantizes expert weights during model loading, freeing the original bf16 tensor immediately.\n\nThe reason is a Transformers v5 change. Expert layers moved from `nn.Linear`\n\nto fused `nn.Parameter`\n\n3D tensors. bitsandbytes could no longer quantize them on load. Axolotl's docs report GLM-4.7-Flash QLoRA dropping from roughly 127 GiB to roughly 23 GiB reserved memory with `quantize_moe_experts: true`\n\n.\n\n**Multi-GPU**\n\nThis is where the ranking inverts. Unsloth's single-GPU lead does not carry over.\n\n**Axolotl: the deepest parallelism matrix**\n\nAxolotl's [multi-GPU guide](https://docs.axolotl.ai/docs/multi-gpu.html) offers three mutually exclusive sharding strategies: DeepSpeed ZeRO stages 1 through 3, FSDP, and DDP. FSDP2 is the recommended path, and FSDP1 is deprecated.\n\nOn top of those, its [N-D Parallelism guide](https://docs.axolotl.ai/docs/nd_parallelism.html) composes data, tensor, context, and expert parallelism through PyTorch's `DeviceMesh`\n\n. The documented support matrix confirms FSDP+TP, HSDP+TP, FSDP+CP, FSDP+TP+CP, and FSDP+EP.\n\nTwo combinations are explicitly unsupported. Expert parallelism cannot compose with TP or CP in v1. Pure DDP cannot compose with them either.\n\nAxolotl's [sequence parallelism](https://docs.axolotl.ai/docs/sequence_parallelism.html) uses the `ring-flash-attention`\n\nlibrary. Its [published H100 benchmark](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl) for Llama 3.1 8B QLoRA supports the tradeoff.\n\n| SP degree | Max context | Context scaling | Tokens/sec | Speedup / GPU count |\n|---|---|---|---|---|\n| 1 | 17,408 | 1.00x | 9,104 | 100.0% |\n| 2 | 34,816 | 2.00x | 15,806 | 86.8% |\n| 4 | 66,560 | 3.82x | 12,314 | 33.8% |\n| 8 | 129,024 | 7.41x | 11,096 | 15.2% |\n\nContext scales close to linearly. Throughput efficiency collapses. On 4090s at SP degree 8, the same benchmark records 0.88x speedup. Training got slower while context reached 32,768 tokens.\n\nAxolotl also supports multi-node training through torchrun and [Ray](https://docs.axolotl.ai/docs/ray-integration.html).\n\n**TRL: two sequence-splitting backends**\n\nTRL's [distributed training guide](https://huggingface.co/docs/trl/en/distributing_training) draws a clean distinction. Context Parallelism means Ring Attention on FSDP2. Sequence Parallelism means ALST/Ulysses on DeepSpeed.\n\nRing Attention needs Accelerate 1.11.0+, uses `cp_size`\n\nwith `cp_backend=\"torch\"`\n\n, and currently supports SDPA only. FlashAttention is not supported on that path. Sequences must divide by `cp_size * 2`\n\n.\n\nALST/Ulysses needs DeepSpeed 0.18.1+ and Accelerate 1.12.0+. It uses `sp_size`\n\nwith `sp_backend=\"deepspeed\"`\n\nand works with FlashAttention-2. It is bounded by attention head count, requiring `num_heads >= sp_size`\n\n.\n\nTRL's guidance is specific. Ring Attention suits 1M+ token sequences and limited network topology. Ulysses suits NVLink or InfiniBand interconnects and sequences up to roughly 500k tokens.\n\nTRL's own Ring Attention benchmark fine-tuned Qwen3-8B across 1, 2, 4, and 8 H100 GPUs. At 8 GPUs, context lengths above 300k tokens became trainable.\n\n**LLaMA-Factory: standard engines with Megatron**\n\nLLaMA-Factory's [distributed training docs](https://llamafactory.readthedocs.io/en/latest/advanced/distributed.html) cover DDP, DeepSpeed, and FSDP, including FSDP2 and Ray for single-node and multi-node runs. The docs also describe DeepSpeed AutoTP, which combines tensor parallelism with ZeRO.\n\nIts most consequential 2025 addition was a Megatron-core training backend through [mcore_adapter](https://github.com/alibaba/ROLL/tree/main/mcore_adapter). That opens a genuine large-scale pretraining path.\n\nThe FSDP+QLoRA path fine-tunes a 70B model on two 24 GB GPUs. That is the cheapest documented route to 70B in this comparison.\n\nThe friction point is the interface. Distributed configuration lives in YAML and CLI, not in LlamaBoard. Dev teams that adopted LLaMA-Factory for its zero-code UI lose that property when they scale past one GPU.\n\n**Unsloth: the open gap**\n\nUnsloth's [multi-GPU documentation](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) states that multi-GPU works through Accelerate and DeepSpeed, giving access to FSDP and DDP. It also states the process is complex and requires manual setup, with official support still being announced.\n\nThe practical route is `accelerate launch train.py`\n\nor `torchrun --nproc_per_node N_GPUS train.py`\n\n. For models too large for one GPU, `device_map = \"balanced\"`\n\nsplits the model across devices.\n\nUnsloth's [PyPI listing](https://pypi.org/project/unsloth/) marks multi-GPU as available with major improvements pending. The Studio [changelog](https://unsloth.ai/docs/new/changelog) describes preliminary automatic multi-GPU allocation for inference and training as of March 2026.\n\nRead together, the position is clear. Unsloth supports multi-GPU. It does not yet offer the composable parallelism matrix that Axolotl and TRL document.\n\n**Where each one breaks**\n\n**Unsloth** breaks when you need tensor, context, or expert parallelism as first-class config. It also breaks when your model is not in its supported list, since gains come from architecture-specific kernels.**Axolotl** breaks on the learning curve. You configure FSDP2 versus DeepSpeed, SP degree, and divisibility constraints. Those constraints span GPU count, sequence length, and attention heads.**TRL** breaks on defaults. It gives you correct primitives, not tuned ones. You supply the Accelerate config, the memory optimizations, and the parallelism plan.**LLaMA-Factory** breaks at the UI boundary. Its abstraction is excellent up to one node and thin above it.\n\n**Choosing**\n\n- Single consumer GPU, supported architecture, LoRA or QLoRA:\n**Unsloth**. The context-length headroom alone justifies it. - Two to eight GPUs, long context, full fine-tuning or RLHF pipelines:\n**Axolotl**. FSDP2 plus sequence parallelism is the best-documented path. - Custom training loops, novel post-training algorithms, tight Hugging Face coupling:\n**TRL**. You are building on the layer the others wrap. - Broadest model coverage, non-engineer operators, fastest first run:\n**LLaMA-Factory**. Then move to CLI when you scale. - These choices are not exclusive. LLaMA-Factory can run Unsloth as a backend. TRL ships an Unsloth integration. Axolotl calls TRL trainers internally.\n\n**Key Takeaways**\n\n- Unsloth wins single-GPU speed and context length; multi-GPU remains its documented weak point.\n- Axolotl ships the deepest parallelism matrix: FSDP2, DeepSpeed, TP, CP, and EP composable via DeviceMesh.\n- TRL is the primitive layer others wrap, now with Ring Attention and ALST/Ulysses sequence splitting.\n- LLaMA-Factory trades depth for breadth: 100+ models, zero-code UI, Megatron backend, CLI-only distributed setup.\n- Sequence parallelism scales context near-linearly but throughput efficiency drops sharply past four GPUs.\n\nAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.", "url": "https://wpnews.pro/news/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on", "canonical_source": "https://www.marktechpost.com/2026/07/22/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on-speed-vram-and-multi-gpu/", "published_at": "2026-07-22 09:16:40+00:00", "updated_at": "2026-07-22 09:28:08.895577+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Unsloth", "Axolotl", "TRL", "LLaMA-Factory", "Hugging Face", "NVIDIA B200", "H100", "AMD"], "alternates": {"html": "https://wpnews.pro/news/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on", "markdown": "https://wpnews.pro/news/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on.md", "text": "https://wpnews.pro/news/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on.txt", "jsonld": "https://wpnews.pro/news/unsloth-vs-axolotl-vs-trl-vs-llama-factory-a-fine-tuning-framework-comparison-on.jsonld"}}