HomomorphicEncryption.org's FHE Benchmarking Suite added a BERT workload whose current reference submission follows DESILO's THOR method, the suite's public repository showed on July 24. The workload runs BERT-Base on MRPC under encrypted inference and supplies a common harness for comparing implementations. It is a reference workload, not evidence that all generative or decoder-only language models have comparable encrypted-inference performance.
HomomorphicEncryption.org's FHE Benchmarking Suite has published a BERT workload whose current reference submission follows DESILO's THOR method and uses the DESILO FHE library. The suite's public repository was updated on July 24, while DESILO and VentureSquare highlighted the reference implementation later in the month.
The workload targets BERT-Base, a 110-million-parameter encoder model, on the Microsoft Research Paraphrase Corpus task from the GLUE benchmark. It provides a shared harness for key generation, input encryption, encrypted computation, decryption and result verification.
What the benchmark standardizes
The benchmark project is intended to make fully homomorphic encryption results easier to compare across libraries and hardware. Its organizers specify workloads, measurement stages and quality checks, while submitters document their implementation and report latency, memory and communication costs.
That structure matters because FHE performance figures can change sharply with the model, cryptographic parameters, hardware, batch size and which setup costs are included. A shared harness makes those assumptions visible instead of reducing an implementation to a single headline number.
The repository's current single-input example reports 5,777 seconds for encrypted computation and about 9,401 seconds end to end, including key generation and preprocessing. VentureSquare separately reported a roughly two-minute THOR inference figure and a matrix-multiplication improvement of up to 9.7 times. Those figures should not be treated as directly comparable without matching the workload, hardware, parameter set and timing boundaries.
What the reference does not establish
DESILO describes THOR as an encrypted language-model inference framework, but the public benchmark workload is BERT rather than a generative, decoder-only model. The reference implementation therefore provides a reproducible baseline for one Transformer task; it does not establish production latency for arbitrary chat, retrieval or long-context workloads.
For practitioners, the useful development is the benchmark artifact itself. Teams evaluating private inference can inspect the full execution path and compare accuracy, latency, memory, key size and communication under a common task. Broader conclusions will require additional submissions and workloads that reflect the architectures and operating constraints teams actually plan to deploy.
Key Points #
- 1The official FHE Benchmarking Suite repository uses DESILO's THOR method as the current reference submission for its BERT workload.
- 2The harness measures the full encrypted-inference path, making hardware, parameter and setup assumptions more visible.
- 3The public workload covers BERT-Base on MRPC and should not be generalized to all generative or decoder-only language models.
Scoring Rationale #
The official benchmark adds a reproducible reference for encrypted BERT inference, but its current scope is one encoder-model task and its timing cannot be generalized to production generative-AI workloads.
Sources #
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.