Shoe string ablation or pruning

For now, I gave it a try, and it turns out the pruning itself can be done on CPU alone:

#

For your particular goal—a smaller version of the Ultra-Heretic model you already like, without paying for a large GPU job first—I would start by pruning the already-ablated Ultra-Heretic checkpoint, using the expert indices published with REAP-48.

That is not a claim that pruning after ablation is always better than ablation after pruning. Those operations can interact, and neither order is guaranteed to preserve the behavior you care about. It is simply the lower-cost starting point here because both ingredients already exist: the Ultra-Heretic weights and REAP-48’s exact per-layer expert keep-list.

I tried that construction on the actual published weights. The CPU pruning succeeded, and a subsequent conversion of the retained checkpoint to GGUF also succeeded. It did not require running Heretic again or renting a GPU.

What completed:

| Check | Observed result | | Source | Ultra-Heretic gpt-oss-120b safetensors | | Expert selection | Published REAP-48 indices, 48 of 128 experts per layer | | Pruned checkpoint | 36 layers; 27,141,018,624 tensor bytes (~27.14 GB) | | GGUF conversion | 10 GGUF parts ,27,164,999,840 file bytes (~27.16 GB) | | GGUF structure | GGUF v3, gpt-oss, 36 blocks, 48 experts, 687 unique tensors; MXFP4 expert weights represented | | Model / text generation | Not tested | | Similarity to the original Ultra-Heretic / actual RAM savings / speed | Not tested |

So this is a successful checkpoint transformation and GGUF structural verification, not yet a claim that the new model runs correctly or behaves like your current Heretic GGUF. The two processing stages were completed across a full-pruning run and a later conversion-only run, not as one uninterrupted execution of the public scripts below.

I put the text-only scripts and a README here, without up any model weights:

Ultra-Heretic + REAP-48 CPU pruning / GGUF recipe · README and commands

The published, cleaned-up scripts have passed small synthetic tests, but that exact public four-file version has not yet been run end-to-end against the full 65 GB source model. The real-model result above comes from the preceding Windows CPU workflow using the same pinned source revisions and pruning method.

On your follow-up six-step plan: keeping your existing Heretic GGUF as a baseline is exactly what I would do. Your proposed route—run Heretic on REAP-48 in the cloud, then convert and compare—is still a reasonable alternative. I would just put this CPU-only construction in front of it, and avoid using precisely the same prompts both to tune Heretic and to claim independent validation. A handful of prompts reserved for checking the result would already help.

Service and cost: for this CPU-pruning/GGUF route, a rented GPU is not required. You still need enough system RAM, substantial temporary disk space, a ~65 GB source download, and CPU time. If you decide to rerun Heretic on the pruned network, then a suitable cloud machine may be useful; the required hardware and final cost need to be checked for that particular model and runtime, rather than inferred from the GGUF file size alone.

#

Why this order? Three practical paths

The two operations do different things:

• Heretic / ablation changes model weights to affect refusal-related behavior. The published Ultra-Heretic variant says it was made withHeretic v1.2.0 and Arbitrary-Rank Ablation (ARA) ; itsmodel card describes the original settings. TheHeretic repository explains the broader optimization approach.
• REAP-style expert pruning removes selected MoE experts and the corresponding router rows. That reduces the number of stored experts, without performing a new behavioral ablation. See theREAP-48 model card andpruning code and write-up .

Consequently, these are different experiments:

A. Reuse work that is already published (my recommended first try)

   published Ultra-Heretic safetensors
       + REAP-48's exact kept expert indices
       -> CPU expert slicing
       -> GGUF conversion
       -> short runtime/behavior checks

   If acceptable: stop; no new Heretic optimization needed.
   If not: B or C remains open.

B. Optimize ablation for the smaller network (your proposed cloud path)

   published REAP-48 checkpoint
       -> check Heretic's support for this 48-expert MXFP4 model
       -> run Heretic / calibrate its ablation on that model
       -> export GGUF
       -> compare against your original Heretic GGUF and plain REAP-48

C. Reconsider which experts to keep

   published Ultra-Heretic checkpoint
       -> collect representative calibration inputs and generated continuations
       -> recompute expert saliency on that checkpoint
       -> prune with the new keep-list
       -> export and compare

Why A first? The existing REAP-48 pruning_manifest.json identifies the exact original expert indices retained in all 36 layers. We can apply those indices directly to the Heretic weights, rather than recomputing expert scores or trying to reproduce the original Heretic optimization. This is a mechanical transformation we can test separately from model behavior.

Why might B differ? Ablating a model after removing most of its experts could produce different internal activations, residual directions, or optimal ablation settings. That makes a fresh Heretic run on REAP-48 a legitimate comparison, provided that specific pruned architecture is supported by the version of Heretic and its model- stack you choose. I have not verified a successful Heretic optimization on REAP-48, so I would check a small load/forward/calibration step before paying for a long optimization run.

Why might C differ? The published REAP-48 experts were selected using 60 resume/job-scoring prompts together with the original model’s generated continuations, and evaluated on a frozen 311-prompt replay of that particular task. This was a meaningful workload-specific result, but not a demonstration that those same 48 experts are globally optimal, or ideal for the changes made by Ultra-Heretic. A new Heretic-specific or target-workload keep-list might help, but that is a more expensive hypothesis to test—not a guaranteed improvement.

So I would not treat either order as a universal winner. A is the cheapest construction to try with the assets already available; B and C become useful if a real behavioral comparison shows something missing.

#

What was actually checked in the CPU experiment

For reproducibility, the source revisions in this experiment were:

The checkpoint transformation used the existing 128-expert tensor layout. For each layer, it selected the 48 retained experts in the manifest’s ascending original-index order. It sliced all the relevant expert blocks, scales, biases, and router rows along the expert axis; non-expert tensors were carried over without intentionally changing their values. The output preserved the Heretic checkpoint’s tensor dtypes, including its FP32 expert biases, and changed num_local_experts to 48.

The test did not calculate new REAP scores, retrain anything, or rerun Heretic. In particular, it did not modify the selected experts to mimic those deleted from the original model.

The actual workflow checked that the pruned checkpoint had the expected set of 687 tensors and a consistent safetensors index. Its tensor payload summed to 27,141,018,624 bytes. A later run converted that retained checkpoint to a 10-part GGUF totaling 27,164,999,840 bytes, and a GGUF reader confirmed the expected architecture, layer/expert counts, and unique tensor count. The extra file bytes are container/representation overhead, not a second measured compression ratio.

The initial GGUF attempt stopped on a tokenizer-class compatibility problem: the published Ultra-Heretic tokenizer metadata names TokenizersBackend (Transformers v5-era), while this pinned conversion setup uses Transformers 4.57.6. A narrowly scoped conversion-time metadata adjustment to PreTrainedTokenizerFast, with serialized tokenizer/vocabulary/token-ID checks and exact restoration of the original configuration, allowed the later conversion to complete. That workaround was for this checkpoint and converter combination, not a general statement that all v4/v5 tokenizer implementations are interchangeable.

A few boundaries matter:

• Confirmed: physical slicing of the real published weights and GGUF structural export with the pinned revisions.
• Confirmed separately: synthetic unit tests for the cleaned-up public scripts, covering expert selection and stored-value/dtype/index consistency on small artificial weights.
• Not confirmed: this newly generated GGUF in a model runtime, producing correct text, maintaining the original Heretic refusal behavior, preserving general reasoning/code quality, or a particular throughput on an RTX 4060/96 GB machine.
• Not claimed: full 65 GB end-to-end execution of the publicly refactored four-file script release. That is a reproducible recipe based on the successful construction, not an independently measured identical run.

This distinction is worth keeping: “the GGUF was written and its structure checked” is not the same test as “the model runs and gives the intended answers.”

#

Reproducing it, or just checking the existing idea

The public folder has four text files:

These are scripts only, no redistributed weights. The instructions below describe the pinned Windows/Python 3.12 path used for the public example; your OS was not specified, so this is an example environment rather than an assumption about yours. The upstream conversion code is Python and does not require compiling llama-cpp-python or building llama.cpp’s inference binaries.

With uv and Git installed, from the folder containing the three Python files:

uv venv --python 3.12 .venv
uv pip install --python .venv/Scripts/python.exe "torch==2.11.0" "transformers==4.57.6" "huggingface_hub==0.36.0" "hf_xet==1.7.0" "safetensors==0.7.0" "numpy==2.3.2" "sentencepiece>=0.2,<0.3" "protobuf>=4.21,<5" "PyYAML>=5.1" "requests>=2.25" "tqdm>=4.27"

.\.venv\Scripts\python.exe test_synthetic.py
.\.venv\Scripts\python.exe heretic_reap48_to_gguf.py --precheck --remove-downloaded-shards

.\.venv\Scripts\python.exe heretic_reap48_to_gguf.py --mode full --remove-downloaded-shards

The precheck checks small files and disk capacity first. The full path uses hf_hub_download(local_dir=...) and downloads the source’s large safetensors one shard at a time, applying the published expert keep-indices on CPU. hf_xet is included for the Xet-backed Hugging Face weights. This avoids the default shared Hub blob-cache layout for those downloads, although temporary transfer data and local download metadata still exist; HF’s local_dir documentation explains the distinction.

For this specific public script, the --remove-downloaded-shards flag permits cleanup of newly downloaded source shards, after their pruned outputs have been verified. It does not mean “delete my existing model files,” and it does not delete the new pruned checkpoint or GGUF output. Without that flag, considerably more disk space is needed. The README recommends roughly 85 GB free with shard cleanup as a conservative starting check; it is not an absolute guarantee across filesystems, transfer intermediates, or library versions. The full source download is about 65 GB; the pruned checkpoint and GGUF are each about 27 GB.

The outputs are separate:

work/pruned_checkpoint/   # retained pruned safetensors
work/run_report.json      # result/failure information
gguf_output/              # retained split GGUF files

If you already have an appropriate complete pruned safetensors checkpoint with the pinned manifest, you can skip the download and pruning:

.\.venv\Scripts\python.exe heretic_reap48_to_gguf.py --mode convert --checkpoint "D:\path\to\pruned_checkpoint"

This path checks the checkpoint’s expected metadata first. On retries, use a fresh/empty gguf_output/ directory (or --gguf-dir pointing to one), rather than mixing old and new split files. Also be aware that the converter temporarily adjusts the tokenizer configuration: it restores it after normal completion or a Python exception, but after a force-kill/power loss an .original-backup file may need to be restored manually before rerunning. The README documents that recovery path.

Finally, converting to GGUF does not automatically prove it will load or run with your installed version of llama.cpp. A useful next check, after you have all ten parts together, is to point a compatible llama.cpp runtime at the first split GGUF file; the runtime discovers the others. The llama.cpp GGUF splitting discussion explains this. Start with a very short generation before trying an elaborate benchmark. I have not yet run that test for the newly generated model.

#

What a service might cost, and how I would compare the outcomes

A service is possible, but there are different jobs hiding behind that question.

| Route | What you pay for | When it makes sense | | Local CPU slicing + GGUF | Your disk, RAM, bandwidth, electricity and time; no GPU rental | First choice if you can spare the workspace | | Rented CPU VM with enough RAM/disk | CPU time, scratch disk, storage and perhaps data transfer | If your local disk/CPU is the limiting factor | | Cloud GPU + Heretic on REAP-48 | GPU runtime, setup/calibration/optimization, storage and transfers | If route A loads but does not give the behavior you want | | Fresh REAP calibration on Ultra-Heretic | More substantial compute plus representative calibration data | If it appears the expert selection is the bottleneck |

For a scale reference, Runpod’s published GPU rental guide lists Secure Cloud rates verified on August 10, 2026 such as $0.53/hour for an RTX A6000 48 GB, $1.09/hour for an L40S 48 GB, and $1.59/hour for an A100 80 GB. See current pricing for the actual tier and availability when booking.

Those prices are examples of rental rates, not quotations for this task. I have not measured the Heretic optimization runtime or established that any listed single GPU has enough VRAM and compatible support for your exact 48-expert model. System RAM, model quantization, scratch storage, repeated optimization trials and any idle-volume charges can determine the real bill. A cheap hourly GPU can still make a costly experiment if the model cannot load efficiently or the job needs many attempts.

If you follow your proposed cloud route, I would keep your steps 1, 4 and 5, but slightly rearrange the middle:

1. Keep your known-good Ultra-Heretic GGUF and the plain REAP-48 result you have already tried. They are useful controls.
2. Try the CPU-sliced Heretic → REAP-48 construction first if disk/RAM permit; it involves no new calibration or cloud GPU optimization.
3. If that candidate loads but misses an important part of your intended behavior, try REAP-48 → new Heretic ablation on a suitable service. First verify model load/forward and Heretic compatibility with the pruned architecture before paying for a larger search.
4. For either candidate, convert/export to the same kind of GGUF and compare with the controls under roughly the same runtime/chat-template/sampling setup.
5. If the remaining problem appears to be which experts survived ,then investigate new REAP calibration on the Heretic model, rather than assuming the resume/job-scoring keep-list is optimal for your use.

I would make the actual comparison fairly small at first. You already have previous prompts that matter to you, which is a better starting point than a giant generic benchmark. One or two samples can check that the new model loads and produces normal output; a few more can check the behaviors you care about. Include at least one untouched prompt if any prompts were used to tune Heretic. If a difference matters, repeat that prompt a couple of times or use fixed generation settings so sampling noise does not masquerade as a model change.

A practical set of checks could cover:

• A few of your existing Heretic prompts , compared side by side with the original Heretic and unmodified REAP-48.
• Ordinary instruction-following and formatting , including short structured output if you use it.
• One or two reasoning/code or longer-answer examples relevant to your own use, not just refusal-related prompts.
• Actual host RAM/VRAM and tokens per second* after* a real runtime load, because a 27 GB file size by itself is not a measured serving-memory or speed figure.

If those small comparisons already answer the practical question, there is no particular reason to turn it into an expensive full-model evaluation. If they reveal a real tradeoff, the same controls can guide a more focused experiment.

One broader conversion note: llama.cpp issue #29078 discusses memory/disk pressure with other very large checkpoints. That is useful background for capacity planning, not a failure observed in this completed 27 GB GGUF conversion or a reason to assume identical resource figures.

For me the main takeaway is that “prune the ablated model” is an inexpensive, structurally demonstrated option in this particular case. It is worth checking its real runtime behavior before spending money on a new ablation or REAP calibration. Your original cloud proposal remains a sensible second branch if that first candidate does not meet your needs.

source & further reading

discuss.huggingface.co — original article Rakarrack-0.6.1 port making progress! ( AI assisted ) Cloud Storage Poll Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)