Hey everyone,
Setup: GMKtec EVO-X3, AMD Ryzen AI MAX+ 395 (Strix Halo), Radeon 8060S, 117.55 GiB unified memory, Ubuntu 24.04, Vulkan/RADV, llama.cpp built with GGML_VULKAN=ON. Running Qwen3-32B-heretic-Q8_0 on llama-server (127.0.0.1:8081).
Goal: build an uncensored/abliterated Greek-native LLM for local use. Best candidate: ilsp/Llama-Krikri-8B-Instruct (Llama 3.1 8B Greek fine-tune). Plan: abliterate with Heretic (heretic-llm), convert to GGUF Q4_K_M, run on Vulkan.
Questions:
- Has anyone abliterated Llama-Krikri-8B or any Greek fine-tune? Does Heretic support it?
- Best workflow: abliterate safetensors then convert to GGUF, or use abliterate.cpp on GGUF? Any pitfalls?
- Best quantization for Greek on Strix Halo: Q4_K_M, Q5_K_M, Q6_K? Need quality/speed balance.
- Recommended llama.cpp Vulkan flags for Strix Halo: -ngl 99, -c 32768, flash attention, q8_0 KV cache, speculative decoding? Any known issues?
- Any existing uncensored Greek models I missed (Krikri, Meltemi, Sophea, etc.)?
Thanks!