{"slug": "totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance", "title": "ToTra – open-source LLM gateway with GDPR/EU AI Act compliance", "summary": "ToTra, an open-source AI gateway written in Go, now provides GDPR and EU AI Act compliance for any large language model with a single line of code change. The platform enforces per-user quota limits, blocks personally identifiable information across 18 languages at the network edge, and generates immutable audit logs before any data reaches providers like OpenAI or Anthropic. Organizations can self-host the gateway to maintain full control over their API keys and infrastructure while adding cost tracking and chargeback reporting.", "body_md": "**AI Gateway & Governance Platform**\n\nOpen-source LLM proxy written in Go. Add quota enforcement, PII blocking, cost tracking, and compliance to any LLM in one line of code.\n\n[Quick Start](#get-started-in-5-minutes) ·\n[Integration Guide](#connect-your-apps) ·\n[Features](#features) ·\n[Architecture](#architecture) ·\n[Gateway Docs](/SugaC-275/ToTra/blob/main/docs/gateway.md) ·\n[Admin API](/SugaC-275/ToTra/blob/main/docs/admin.md) ·\n[Discussions](https://github.com/SugaC-275/ToTra/discussions)\n\nToTra is an open-source AI gateway and governance platform that sits in front of any LLM provider.\n\nPoint your existing apps at ToTra instead of OpenAI, Anthropic, or any other provider — and instantly get:\n\n**Quota enforcement**— per-user and per-team hard budget caps** PII blocking**— 18 language groups scanned at the edge before any data leaves your network** Cost tracking**— per-user, per-team, per-model token and USD spend with chargeback reports** Compliance**— GDPR workflows, EU AI Act checklist, hash-chained immutable audit log** Zero code changes**— 100% OpenAI-compatible; swap one line in your config\n\n``` php\nflowchart LR\n    A[\"🖥️ Your App\\n(OpenAI SDK / curl\\n/ LangChain)\"] -->|\"1 · API request\"| B\n\n    subgraph B[\"ToTra Gateway  :8080\"]\n        direction TB\n        B1[\"🔑 Auth & API Key\"]\n        B2[\"📊 Quota Check\\n(per user / team)\"]\n        B3[\"🔒 PII Scan\\n(18 languages)\"]\n        B4[\"⚡ Semantic Cache\"]\n        B5[\"🔀 Route & Load Balance\"]\n        B1 --> B2 --> B3 --> B4 --> B5\n    end\n\n    B -->|\"2 · forward request\"| C[\"☁️ LLM Providers\\nOpenAI · Anthropic\\nGemini · Mistral · Azure\\nBedrock · Ollama\"]\n    C -->|\"3 · response\"| A\n\n    B -->|\"4 · usage events\"| D\n\n    subgraph D[\"ToTra Admin  :8081\"]\n        direction TB\n        D1[\"💸 Cost Tracking\"]\n        D2[\"📋 Compliance & Audit\"]\n        D3[\"🔔 Budget Alerts\"]\n    end\n\n    D --> E[\"📊 Dashboard  :3000\\nAdmin Console · Reports\\nEmployee Self-Service\"]\n```\n\n- 🚀\n**Written in Go**— < 2 ms p95 overhead. Native binary, no Python runtime, no warm-up. - 🔒\n**PII blocked at the edge**— email, IDs, credit cards, health records across 18 language groups. Sensitive data is redacted before it ever reaches an LLM. - 💸\n**Hard budget caps**— requests over limit get`429`\n\nbefore touching any provider. Real-time Slack / webhook alerts. - 📋\n**Compliance out of the box**— GDPR data-subject workflows, EU AI Act checklist, and an immutable hash-chained audit log on every request. - 📊\n**Finance-ready reporting**— department chargeback CSV, budget forecasts, spend anomaly detection. - 🏠\n**Self-hosted**— your keys, your infrastructure, your data. No external dependency.\n\n**Prerequisites:** Docker + Docker Compose\n\n```\ngit clone https://github.com/SugaC-275/ToTra.git\ncd ToTra\ncp .env.example .env          # fill in your provider API keys\ndocker-compose --profile app up -d --wait\n```\n\nOpen ** http://localhost:3000** and sign in:\n\n| Field | Value |\n|---|---|\n`admin@acme.com` |\n|\n| Password | `totra123` |\n\nChange default credentials immediately after first login via\n\nSettings → Security.\n\nOne line change. Every other line of code stays the same.\n\n**Python (OpenAI SDK)**\n\n``` python\nimport openai\n\n# Before — calls OpenAI directly\nclient = openai.OpenAI(api_key=\"sk-...\")\n\n# After — routes through ToTra (zero other changes)\nclient = openai.OpenAI(\n    api_key=\"your-totra-api-key\",      # issued from the ToTra admin panel\n    base_url=\"http://your-totra-host:8080/v1\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}]\n)\nprint(response.choices[0].message.content)\n```\n\n**Node.js / TypeScript (OpenAI SDK)**\n\n``` python\nimport OpenAI from \"openai\";\n\nconst client = new OpenAI({\n  apiKey: \"your-totra-api-key\",\n  baseURL: \"http://your-totra-host:8080/v1\",\n});\n\nconst response = await client.chat.completions.create({\n  model=\"gpt-4o\",\n  messages: [{ role: \"user\", content: \"Hello!\" }],\n});\nconsole.log(response.choices[0].message.content);\n```\n\n**curl**\n\n```\ncurl http://your-totra-host:8080/v1/chat/completions \\\n  -H \"Authorization: Bearer your-totra-api-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"gpt-4o\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n```\n\n**LangChain**\n\n``` python\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(\n    model=\"gpt-4o\",\n    openai_api_key=\"your-totra-api-key\",\n    openai_api_base=\"http://your-totra-host:8080/v1\",\n)\n\nresponse = llm.invoke(\"Hello!\")\nprint(response.content)\n```\n\nOnce connected, every request is automatically routed through quota enforcement, PII scanning, semantic caching, and cost tracking.\n\n**🔒 PII Protection — 18 Language Groups**\n\nEvery request body is scanned in real time before it reaches any LLM. Detected PII is redacted and the event is logged. Blocked requests return `422`\n\n.\n\n| Language Group | Detected Types |\n|---|---|\n| Universal | Email, credit card, IBAN, SWIFT/BIC, ICD medical codes |\n| Chinese | National ID, phone, bank account, unified credit code, securities account |\n| English | US SSN, phone, NI number, passport, driver's license, medical record number |\n| Japanese | My Number (個人番号), phone, postal code, health insurance number |\n| Korean | RRN (주민등록번호), phone, passport, business registration number |\n| EU (14 countries) | National IDs, tax numbers, social security — DE/FR/ES/IT/NL/PL/SE/PT/BE/CH/DK/FI/NO/AT |\n| Arabic (GCC + MENA) | National ID, Iqama, Emirates ID, QID, CIN, NIN, phone |\n\nConfigure rules per team, per model, or globally in the admin panel.\n\n**💸 Cost & Spend Management**\n\n- Per-user, per-team, per-model token and USD cost tracking\n**Hard budget caps**— requests over limit get`429`\n\nbefore touching the provider- Configurable alert thresholds with Slack / Feishu / webhook notifications\n- Monthly budget forecasts based on current burn rate\n**Department chargeback reports** with CSV export for finance- Procurement analytics and ROI dashboards\n- Spend anomaly detection with automatic alerts\n\n```\nDashboard → Cost → Reports → Export CSV\n```\n\n**📋 Compliance & Audit**\n\n**GDPR**— data-subject export and deletion request workflows, configurable retention policies** EU AI Act**— compliance checklist with per-model status tracking** Immutable audit chain**— every request is hash-chained; the log cannot be tampered with** SIEM integration**— configurable webhook targets for security event forwarding- Data residency controls — keep all data on-premises or in a specific region\n\n**⚡ Gateway & Routing**\n\n**OpenAI-compatible**— drop-in replacement for the OpenAI API (`/v1/chat/completions`\n\n,`/v1/embeddings`\n\n, streaming)**Anthropic-compatible**— native Anthropic messages API support- Multi-provider routing — automatic fallback across providers and models\n**Semantic cache**— SimHash LSH deduplication; repeated prompts skip the LLM entirely- Prompt compression — reduce token spend on long context\n- Streaming proxy — full\n`text/event-stream`\n\nsupport **File pipeline**— upload PDF / DOCX / PPTX → parse → chat in one API call- Rate limiting, IP allowlist, API-key authentication\n\n**🔐 Administration**\n\n- JWT authentication + OIDC / SSO integration\n- Role-based access control (admin / employee)\n- User and team management with quota request / approval workflow\n- Model catalogue — enable, disable, and configure providers per team\n- Bot notifications — Slack, Feishu, custom webhooks\n- HR sync connector (CSV import)\n**Agent session tracking**— detects and terminates dead-loop agent sessions automatically\n\n| Provider | Chat | Embeddings | Streaming | Files |\n|---|---|---|---|---|\n| OpenAI (GPT-4o, o1, o3, o4) | ✅ | ✅ | ✅ | ✅ |\n| Anthropic (Claude 3.5, 4) | ✅ | — | ✅ | ✅ |\n| Google Gemini | ✅ | ✅ | ✅ | — |\n| Mistral AI | ✅ | ✅ | ✅ | — |\n| Meta Llama (via Ollama) | ✅ | ✅ | ✅ | — |\n| Cohere Command | ✅ | ✅ | ✅ | — |\n| Azure OpenAI | ✅ | ✅ | ✅ | ✅ |\n| AWS Bedrock | ✅ | ✅ | ✅ | — |\n| Local / Ollama | ✅ | ✅ | ✅ | — |\n| Any OpenAI-compatible endpoint | ✅ | ✅ | ✅ | — |\n\nToTra is written entirely in Go. The gateway adds **< 2 ms** overhead at p95 under production load.\n\n| Concurrency | p50 | p95 | p99 |\n|---|---|---|---|\n| 10 VUs | < 1 ms | 2 ms | 4 ms |\n| 50 VUs | 1 ms | 3 ms | 8 ms |\n| 200 VUs | 2 ms | 6 ms | 15 ms |\n\nMeasured against a 100 ms mock upstream.\n\n[Reproduce the benchmark →]\n\n```\nYour Apps  (OpenAI SDK / curl / LangChain / any HTTP client)\n    │\n    ▼\nToTra Gateway  :8080\n    auth · quota · PII scan · policy · semantic cache · routing\n    │\n    ▼\nOpenAI · Anthropic · Gemini · Mistral · Local Models\n    │\n    │ (usage events)\n    ▼\nToTra Admin  :8081\n    cost · compliance · budgets · audit trail · notifications\n    │\n    ▼\nDashboard  :3000\n    admin console · department reports · employee self-service\n```\n\n| Service | Stack | Port |\n|---|---|---|\n`gateway` |\nGo 1.25 / Fiber | 8080 |\n`admin` |\nGo 1.25 / Fiber | 8081 |\n`parser` |\nPython 3.12 / FastAPI | 8090 |\n`dashboard` |\nReact 19 / Vite | 3000 |\n`postgres` |\nPostgreSQL 16 | 5432 |\n`redis` |\nRedis 7 | 6379 |\n\n| Cost Dashboard | Department Reports |\n|---|---|\n\n| User Management | Employee Self-Service |\n|---|---|\n\n```\n# 1. Start databases\ndocker-compose up -d postgres redis\n\n# 2. Run each service in its own terminal\ncd gateway   && go run .\ncd admin     && go run .\ncd parser    && uvicorn main:app --port 8090\ncd dashboard && npm install && npm run dev\n\n# 3. Seed dev credentials (first time only)\ncd scripts/set-dev-passwords\nPOSTGRES_HOST=localhost POSTGRES_DB=totra \\\nPOSTGRES_USER=totra POSTGRES_PASSWORD=totra_secret go run .\n```\n\n**Default dev credentials:** `admin@acme.com`\n\n/ `totra123`\n\nCopy `.env.example`\n\nto `.env`\n\n. Key variables:\n\n| Variable | Description |\n|---|---|\n`POSTGRES_HOST/PORT/DB/USER/PASSWORD` |\nPostgreSQL connection |\n`JWT_SECRET` |\nShared secret for JWT signing |\n`ENCRYPTION_KEY` |\n32-byte hex key — admin credential store |\n`GATEWAY_ENCRYPTION_KEY` |\n32-byte hex key — gateway credential store |\n`OPENAI_API_KEY` |\nYour OpenAI key (set per provider) |\n`ANTHROPIC_API_KEY` |\nYour Anthropic key |\n\nSee [ .env.example](/SugaC-275/ToTra/blob/main/.env.example) for the full list including Redis, SMTP, and notification settings.\n\n```\nmake test\n\n# Per service\ncd gateway   && go test ./...\ncd admin     && go test ./...\ncd dashboard && npm run test:run\ncd parser    && pytest\n```\n\nWe welcome contributions — bug fixes, new provider integrations, docs improvements, and feature requests.\n\n```\ngit clone https://github.com/SugaC-275/ToTra.git\ncd ToTra\n\n# Run tests before submitting\nmake test\n```\n\n- Fork the repo and create a branch from\n`main`\n\n- Make your change and add tests where relevant\n- Ensure\n`make test`\n\npasses - Open a pull request\n\nFor larger features, open a [Discussion](https://github.com/SugaC-275/ToTra/discussions) first to align on direction.\n\n- 💬\n[GitHub Discussions](https://github.com/SugaC-275/ToTra/discussions)— questions, ideas, show & tell - 🐛\n[GitHub Issues](https://github.com/SugaC-275/ToTra/issues)— bug reports\n\n[MIT](/SugaC-275/ToTra/blob/main/LICENSE) — free to use, self-host, fork, and modify.", "url": "https://wpnews.pro/news/totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance", "canonical_source": "https://github.com/SugaC-275/ToTra", "published_at": "2026-06-06 00:16:10+00:00", "updated_at": "2026-06-06 00:47:29.687324+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "ai-policy", "large-language-models"], "entities": ["ToTra", "OpenAI", "Anthropic", "GDPR", "EU AI Act", "Go", "SugaC-275"], "alternates": {"html": "https://wpnews.pro/news/totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance", "markdown": "https://wpnews.pro/news/totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance.md", "text": "https://wpnews.pro/news/totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance.txt", "jsonld": "https://wpnews.pro/news/totra-open-source-llm-gateway-with-gdpr-eu-ai-act-compliance.jsonld"}}