Open-Source E2E FHE Implementation for Privacy-Preserving Llama 3 8B Inference

Researchers Yuhang Fan and colleagues posted Odin, described as the first open-source end-to-end GPU CKKS implementation of Llama-3, to arXiv on 11 Sep 2026 (revised 16 Sep 2026), reporting server-side fully homomorphic encryption inference over all 32 Transformer layers of Llama-3-8B on a single NVIDIA H100 80 GB GPU. With a 128-token input, Odin completes end-to-end FHE evaluation in 366.4 s at 58.9 GiB peak device memory, a 4.51x speedup over the THOR-style baseline's 1651.9 s under identical model, input, CKKS parameters and hardware. Odin's gains come from a feature-major cross-layer ciphertext packing layout, transient intra-operator layouts for linear projections and attention, and minimax polynomial approximation with input-range control for nonlinear operations.

Computer Science Cryptography and Security Submitted on 11 Sep 2026 v1 https://arxiv.org/abs/2609.12378v1 , last revised 16 Sep 2026 this version, v2 Title:An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference View PDF https://arxiv.org/pdf/2609.12378 HTML experimental https://arxiv.org/html/2609.12378v2 Abstract:Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption FHE lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots. It therefore determines the ciphertext count and the homomorphic cost of linear layers, and it constrains how data pass between linear layers, attention, and nonlinear computation. As models and sequences grow, inefficient layouts accumulate encoding, compute, and layout-conversion overhead. We present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama. Starting from a THOR-style baseline whose bottleneck is weight encoding, Odin uses a feature-major cross-layer layout to unify residual connections and layer interfaces, and builds transient intra-operator layouts for linear projections and attention. This reduces redundant plaintext encoding of weights in wide projections. Within attention, QK^T produces scores that Softmax can consume directly, and PV consumes the resulting probabilities, avoiding intermediate repacking. For nonlinear ops, we use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, reducing polynomial degree and multiplicative depth. To our knowledge, Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3. With Llama-3-8B weights and a 128-token input, Odin evaluates all 32 Transformer layers on a single NVIDIA H100 80 GB GPU. Server-side end-to-end FHE evaluation takes 366.4 s and 58.9 GiB peak device memory. Under the same model, input, CKKS parameters, and hardware, THOR takes 1651.9 s, a 4.51x speedup. Submission history From: Yuhang Fan view email https://arxiv.org/show-email/b3af3443/2609.12378 Fri, 11 Sep 2026 02:51:44 UTC 74 KB \ v1\ https://arxiv.org/abs/2609.12378v1 v2 Wed, 16 Sep 2026 07:01:10 UTC 74 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .