# Amazon HOPE: Second-Order MoE Expert Pruning With Open awslabs Code > Source: > Published: 2026-11-09 00:00:00+00:00 Amazon published **HOPE** (Higher-Order Pruning of Experts) on November 9, 2026: a **second-order** way to delete idle experts from mixture-of-experts (MoE) language models while preserving clusters that routinely co-activate. The team ships a **three-step CLI** and Python API on [GitHub as awslabs/HOPE](https://github.com/awslabs/HOPE?ref=explainx), with the method explained on [Amazon's Hugging Face blog post](https://huggingface.co/blog/amazon/higher-order-pruning-experts?ref=explainx) and in [arXiv:2609.18916](https://arxiv.org/abs/2609.18916?ref=explainx). At **50% pruning**, HOPE reported up to **+6.1%** on agentic coding tasks versus the next-best first-order baseline and, on one Qwen3.5-35B run, a **42.3 SWE-Bench Pro** score that nearly matched the unpruned **41.8**. If you run large MoE models locally or on a fleet, most VRAM is **expert MLP weights you never touch on a given token**. HOPE is aimed exactly at that waste line — without asking you to retrain or rewrite inference servers. ## Why first-order expert pruning breaks cooperating experts MoE layers route each token to a small active set (often **top-8**) out of **dozens or hundreds** of experts per layer. Amazon's example: **Qwen3.5-122B** keeps **256 experts** in each of **48** MoE layers, and about **95%** of the model's on-disk parameter mass sits in those expert MLPs even when only eight fire per token. **Expert pruning** permanently deletes experts to shrink checkpoints and VRAM. That is attractive because it needs **no fine-tuning**, avoids dynamic expert load/unload latency, and stacks with **quantization** — themes we cover in [semantic caching for LLM apps](https://explainx.ai/blog/semantic-caching-llm-apps-cost-latency-guide-2026) and [vLLM versus SGLang](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026) when you are squeezing cost per request. The catch is **which** experts to cut. **First-order** methods (REAP, MAN, EAN, frequency heuristics) score experts **independently** on a calibration set and drop the lowest ranks **per layer**. That ignores a stable empirical fact Amazon highlights: experts form **cooperative groups** that get selected together, a pattern other MoE papers also observe early in training. HOPE MoE expert pruning compares first-order scores with pairwise expert cooperation *Image: [Amazon Science (Hugging Face blog)](https://huggingface.co/blog/amazon/higher-order-pruning-experts?ref=explainx)* Removing expert **A** can look safe in isolation while breaking expert **B** that relied on **A** whenever both routed. At **aggressive** prune rates, those broken links compound. HOPE treats that as a **quadratic optimization** problem instead of **N independent rankings**. For MoE basics and routing math, pair this with our [KV cache memory explainer](https://explainx.ai/blog/what-is-kv-cache-llm-inference-explained-guide-2026) — MoE shrinks **static** weights, while KV cache still scales with context at runtime. ## How HOPE builds the F-matrix and picks a prune set HOPE's derivation (Appendix A in the paper) starts from **pruning error**: deleting experts changes which backup experts receive traffic, which makes exact minimization intractable. Amazon introduces a surrogate **Z** that upper-bounds the dominant error term and expands into **pairwise products** of gate weights and expert output norms. Calibration is one forward pass per layer, recording which experts activate, gate weights, output norms, and co-activation counts. Those statistics assemble an **F-matrix** where: - **Diagonal** entries approximate**solo importance** (and nearly recover REAP's ranking). - **Off-diagonal** entries score**cooperative contribution** when two experts share a token's active set. HOPE then solves a **binary quadratic program** per layer: pick exactly **P** experts to delete minimizing **p^T F p**, using a continuous relaxation and rounding. Solving takes on the order of **one to two seconds per layer** — negligible versus calibration, per Amazon's post. GLM-4.5-Air HOPE interaction matrix showing cooperating MoE expert clusters *Image: [Amazon Science (Hugging Face blog)](https://huggingface.co/blog/amazon/higher-order-pruning-experts?ref=explainx)* The blog visualizes **GLM-4.5-Air** layers where bright blocks along clustered axes show **tight cooperation bands**. That picture is why HOPE sometimes keeps a slightly lower solo-scored expert: it protects a **cluster** first-order cuts would dismantle. **Limitations** Amazon states plainly: HOPE is layer-local (cross-layer HOPE is preliminary in the paper), focuses on **pairwise** not triplet interactions, and its lead over REAP **shrinks below ~40%** pruning. For giant frontier MoE stacks like [Tencent HY4](https://explainx.ai/blog/tencent-hy4-official-release-770b-vision-computer-use-terminal-bench-44-4-weights-planned-2026), expert pruning is complementary to architectural choices (latent attention, grouped routing) already baked into the base model. ## Benchmarks: where HOPE wins and ties the unpruned model Amazon tested **Qwen3.5-122B**, **Qwen3.5-35B**, and **GLM-4.5-Air** with calibration from **Evol-CodeAlpaca** and **SWE-Bench Verified** trajectories, then evaluated on **Tulu3** suites plus **SWE-Bench Pro** agentic coding. HOPE MoE pruning benchmark ranks versus REAP at aggressive 50 percent expert cuts *Image: [Amazon Science (Hugging Face blog)](https://huggingface.co/blog/amazon/higher-order-pruning-experts?ref=explainx)* Headline numbers from the post: | Setting | HOPE takeaway | |---|---| | Average rank across conditions | **2.07** of five methods (best) | | Top-1 finish rate | **39%** of conditions | | 50% pruning average rank | **1.58** vs**2.42** for REAP | | Qwen3.5-35B @ 50%, SWE-Bench Pro | **42.3** HOPE vs**41.8** unpruned; REAP**36.2** | **SWE-Bench Pro** matters for practitioners because it stresses **multi-turn tool use** — the regime where diverse expert combinations show up. That lines up with Amazon's story: agentic workloads benefit most when cooperative structure survives pruning. Amazon also notes an occasional **pruned-beats-full** effect (noise experts removed), not unique to HOPE. Code-focused calibration did not catastrophically harm non-code Tulu3 tasks in their runs — useful if you plan to specialize a pruned checkpoint for [coding agents](https://explainx.ai/blog/agent-sandboxes-coding-agents-compared-e2b-modal-daytona-guide-2026) without maintaining two full-size MoE copies. Compare this compression axis to kernel-side work like [SGLang Muse mega-kernels](https://explainx.ai/blog/sgl-muse-mega-meta-megakernel-muse-spark-open-gb300-1109-tokens-per-second-sglang-plugin-2026): HOPE shrinks **which experts exist**; Mega kernels speed **how matmuls run** on whatever weights remain. ## How to run HOPE on your own MoE checkpoint Amazon documents a CLI in three commands (also exposed as Python helpers): ``` hope calibrate \ --model-path Qwen/Qwen3.5-35B-A3B \ --prompts calibration_prompts.txt \ --out-path observations.h5 hope solve \ --obs-path observations.h5 \ --prune-frac 0.50 \ --out-path pruneset.json hope prune \ --model-path Qwen/Qwen3.5-35B-A3B \ --pruneset-path pruneset.json \ --out-path ./Qwen3.5-35B-A3B-pruned-50 ``` **Calibration prompts** should resemble production traffic if you care about a niche (support bots versus codegen). The HDF5 stores the **F-matrix**; **solve** emits **pruneset.json**; **prune** writes a standard Hugging Face folder you can point vLLM or SGLang at — no custom expert-loader patches per Amazon. Practical checklist before you run it on a shared GPU box: 1. **Disk and RAM** to hold the dense MoE source once. 2. A **calibration file** long enough to cover routing modes you care about (Amazon used code-heavy sets for agentic gains). 3. A **prune fraction** decision:**50%** is where HOPE's memory win is largest but also where routing distortions hurt if calibration lied about your traffic. If you are already experimenting with [StepFun P2 distillation](https://explainx.ai/blog/stepfun-p2-mopd-multi-teacher-distillation-parameter-priors-qwen3-2026) or other Qwen-family tweaks, treat HOPE as an **orthogonal** lever — it does not change tokenizer or attention code, only which expert weights exist. Hybrid-architecture follow-ups live in a separate Amazon repository linked from the blog for readers pruning **next-gen** MoE hybrids; check that README if your model mixes dense and sparse blocks. ## What this means for builders and operators HOPE does not replace quantization, speculative decoding ([our guide](https://explainx.ai/blog/what-is-speculative-decoding-llm-inference-guide-2026)), or KV compression. It attacks a different bill item: **the enormous expert parameter tensor** that makes MoE attractive at quality but painful at deploy time. Operators running **Qwen3.5** or **GLM-4.5** class models should read HOPE as a **cheap experiment**: a day of calibration and solve passes can tell you whether you can halve expert VRAM on agentic workloads before you commit to buying more GPUs. Researchers comparing pruning papers now have an open **second-order** baseline with reproducible CLI steps — rare in a space full of leaderboard-only claims. Watch the usual caveats: pruning is **irreversible** without keeping the original checkpoint; calibration mismatch still kills quality; and layer-local pruning cannot fix a router that was broken before you started. ## Choosing a calibration set without fooling yourself Amazon's README pattern uses a plain-text **`calibration_prompts.txt`**. Quality of that file is the hidden hyperparameter: - **Match your traffic shape.** If production is 80% single-turn FAQ and 20% multi-tool coding, a file that is 100% SWE trajectories will over-prune experts that matter on short turns. - **Length mix matters.** Expert routing differs on 500-token prompts versus 32K RAG dumps. Include both if both exist in prod. - **Re-run solve when the model updates.** Router statistics drift across checkpoint revisions even within Qwen3.5 patch releases. HOPE's diagonal **almost equals REAP** (Spearman **0.988** in the blog). When HOPE beats REAP, the win comes from **off-diagonal** terms — cooperators REAP would split. If your calibration never activates those pairs, HOPE collapses toward REAP behavior and you paid QP time for little gain. For deployment math: at **50%** pruning on a **122B-class MoE**, you are not saving 50% of end-to-end latency — you shrink **static weights** and improve **batch size headroom**. Pair HOPE with **FP8 weights** and **[PagedAttention](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026)** tuning rather than expecting a magic throughput doubling. ## REAP, MAN, and when first-order methods are enough Amazon benchmarks against **REAP**, **MAN**, **EAN**, and **frequency** cuts. At **10–20%** pruning HOPE's average rank advantage shrinks; first-order cuts are **simpler** (no QP) and may be fine when you only need a **modest VRAM shave**. Use HOPE when: 1. You are trying to **fit a model that already OOMs** at your context target. 2. Your eval includes **agentic multi-step** workloads (tool calls, repo edits). 3. You can afford **one offline calibration job** per model revision. Stick with REAP or quant-only when you need a **quick A/B** on a stable chat model with short prompts. ## Governance and licensing notes HOPE ships under **Apache 2.0** on **awslabs/HOPE** per the GitHub link in Amazon's post. That is distinct from **model licenses** on Qwen or GLM weights — pruning does not change their terms, but **redistributing pruned checkpoints** still inherits the base model's license. Inside enterprises, treat pruned artifacts like any other **derived model**: log which **pruneset.json** produced which checkpoint, store calibration prompts for audit, and keep an unpruned **rollback** copy until production metrics clear.