{"slug": "open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference", "title": "Open-Source E2E FHE Implementation for Privacy-Preserving Llama 3 8B Inference", "summary": "Researchers Yuhang Fan and colleagues posted Odin, described as the first open-source end-to-end GPU CKKS implementation of Llama-3, to arXiv on 11 Sep 2026 (revised 16 Sep 2026), reporting server-side fully homomorphic encryption inference over all 32 Transformer layers of Llama-3-8B on a single NVIDIA H100 80 GB GPU. With a 128-token input, Odin completes end-to-end FHE evaluation in 366.4 s at 58.9 GiB peak device memory, a 4.51x speedup over the THOR-style baseline's 1651.9 s under identical model, input, CKKS parameters and hardware. Odin's gains come from a feature-major cross-layer ciphertext packing layout, transient intra-operator layouts for linear projections and attention, and minimax polynomial approximation with input-range control for nonlinear operations.", "body_md": "# Computer Science > Cryptography and Security\n\n  [Submitted on 11 Sep 2026 (\n\n[v1](https://arxiv.org/abs/2609.12378v1)), last revised 16 Sep 2026 (this version, v2)]\n# Title:An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference\n\n[View PDF](https://arxiv.org/pdf/2609.12378)\n\n[HTML (experimental)](https://arxiv.org/html/2609.12378v2)\n\nAbstract:Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots. It therefore determines the ciphertext count and the homomorphic cost of linear layers, and it constrains how data pass between linear layers, attention, and nonlinear computation. As models and sequences grow, inefficient layouts accumulate encoding, compute, and layout-conversion overhead.\n\nWe present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama. Starting from a THOR-style baseline whose bottleneck is weight encoding, Odin uses a feature-major cross-layer layout to unify residual connections and layer interfaces, and builds transient intra-operator layouts for linear projections and attention. This reduces redundant plaintext encoding of weights in wide projections. Within attention, QK^T produces scores that Softmax can consume directly, and PV consumes the resulting probabilities, avoiding intermediate repacking. For nonlinear ops, we use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, reducing polynomial degree and multiplicative depth. To our knowledge, Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3.\n\nWith Llama-3-8B weights and a 128-token input, Odin evaluates all 32 Transformer layers on a single NVIDIA H100 80 GB GPU. Server-side end-to-end FHE evaluation takes 366.4 s and 58.9 GiB peak device memory. Under the same model, input, CKKS parameters, and hardware, THOR takes 1651.9 s, a 4.51x speedup.\n\n## Submission history\n\nFrom: Yuhang Fan [\n[view email](https://arxiv.org/show-email/b3af3443/2609.12378)]\n\n**Fri, 11 Sep 2026 02:51:44 UTC (74 KB)**\n\n[\\[v1\\]](https://arxiv.org/abs/2609.12378v1)\n**[v2]** Wed, 16 Sep 2026 07:01:10 UTC (74 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference", "canonical_source": "https://arxiv.org/abs/2609.12378", "published_at": "2026-09-27 20:19:36+00:00", "updated_at": "2026-09-27 20:31:30.669730+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety", "ai-infrastructure"], "entities": ["Odin", "Llama-3-8B", "Yuhang Fan", "arXiv", "NVIDIA H100 80 GB GPU", "THOR", "CKKS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference", "markdown": "https://wpnews.pro/news/open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference.md", "text": "https://wpnews.pro/news/open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference.txt", "jsonld": "https://wpnews.pro/news/open-source-e2e-fhe-implementation-for-privacy-preserving-llama-3-8b-inference.jsonld"}}