{"slug": "show-hn-goethe-the-infrastructure-for-self-hosted-ai", "title": "Show HN: Goethe – The infrastructure for self-hosted AI", "summary": "A project called Goethe, published on GitHub by meshapplied, turns a Linux machine with an NVIDIA GPU into a self-hosted inference server that exposes an API compatible with OpenAI, Anthropic and LiteLLM clients, so prompts, data and models stay on hardware the user controls. Goethe installs via a curl script and `goethe start`, serves every loaded model at one address with per-app keys, and supports Blackwell (compute 12.0/12.1 and 10.0/10.3), Hopper (9.0), Ada (8.9) and Ampere (8.0/8.6) GPUs on x86_64 or ARM64 Linux with glibc 2.30 or newer, requiring NVIDIA driver 535 or newer. On a 32 GB Blackwell GPU, `goethe recommend` lists meshapplied/Qwen3.8-27B-NVFP4-NInfer first, and the tool ships its own CUDA 13 runtime so no CUDA toolkit is needed on the host.", "body_md": "**Run and share your models on your own infra in 3 steps, and run them fast.**\n\nIf Goethe is useful to you, a ⭐ on [GitHub](https://github.com/meshapplied/goethe) helps others find it.\n\nGoethe turns a Linux machine into your own API, compatible with OpenAI, Anthropic and LiteLLM clients. Install it, add a model, and point your apps, your team or your other machines at one address. Your prompts, your data and your models stay on hardware you control.\n\n```\ncurl -fsSL https://meshapplied.com/install.sh | bash\ngoethe start\n```\n\n`goethe start` starts Goethe and walks you through three steps: install a model, serve it through the API, and configure who can reach it. If you prefer plain commands, these do the same:\n\n```\n# 1. Start Goethe and load a model\ngoethe up\ngoethe install --model nvidia/Llama-3.1-8B-Instruct-NVFP4 --load\n\n# 2. Share it\ngoethe configure api                 # this machine only, your VPN (recommended), or a public address\ngoethe key --create my-laptop        # one key per app or person\ngoethe api                           # the address to give your apps\n```\n\nOn a 32 GB Blackwell GPU, `goethe recommend` lists `meshapplied/Qwen3.8-27B-NVFP4-NInfer` first.\n\nAnything that speaks the OpenAI, Anthropic or LiteLLM API can use it right away, all at the same address: the OpenAI and Anthropic SDKs, LiteLLM, LangChain, LlamaIndex, Open WebUI, coding assistants, or plain `curl`.\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"https://<your-goethe-address>:8443/v1\", api_key=\"goethe_sk_...\")\nreply = client.chat.completions.create(\n    model=\"nvidia/Llama-3.1-8B-Instruct-NVFP4\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\nprint(reply.choices[0].message.content)\n```\n\n- **Easy to serve.** Every model you load is served at the same URL; a request picks its model by name.\n- **Fast.** Use the custom kernel attachments to accelerate and optimise model capability.\n- **Private by default.** Nothing reaches the API until you allow it. Rate-limit, expire, scope or revoke keys at any time.\n- **Fits your network.** This machine only, Tailscale / WireGuard / ZeroTier, or a public IP.`goethe configure api` walks you through the choice.\n- **Many models, many GPUs.** Run several models side by side, split large models across GPUs, and keep other GPU software running next to Goethe.\n- **Always on.** Loaded models come back after a reboot or a crash.\n\n| Environment | Examples | \n|---|---|\n| Your own hardware | Workstations and servers with one or more NVIDIA GPUs, including DGX Spark | \n| Cloud GPU virtual machines | AWS, Google Cloud, Azure, Oracle Cloud, Lambda, CoreWeave, Paperspace | \n| GPU containers | RunPod, Vast.ai, and Docker with the NVIDIA Container Toolkit | \n\nLinux on x86_64 or ARM64 (glibc 2.30 or newer, for example Ubuntu 20.04+ or Debian 11+). Goethe recognises the cloud provider or container it runs in and shows the address to use from outside.\n\n| Architecture | Compute capability | GPUs | \n|---|---|---|\n| Blackwell | 12.0, 12.1 | GeForce RTX 5090, RTX PRO 6000 Blackwell, DGX Spark (GB10) | \n| Blackwell data center | 10.0, 10.3 | B200, GB200, B300, GB300 | \n| Hopper | 9.0 | H100, H200, GH200 | \n| Ada | 8.9 | GeForce RTX 4090, RTX 6000 Ada, L40S, L40, L4 | \n| Ampere | 8.0, 8.6 | A100, A30, A40, A10, RTX A6000, GeForce RTX 3090 | \n\nOlder GPUs (Volta, Turing) are not supported. A machine can mix GPU generations; each model runs on GPUs of one generation, and large models can be split across several GPUs.\n\nGoethe ships its own CUDA 13 runtime and compiler, so no CUDA toolkit is needed on the machine. What it needs is the NVIDIA driver:\n\n| NVIDIA driver | CUDA it offers | Goethe runs on | \n|---|---|---|\n| 580 or newer | 13.0 and newer | Every supported GPU | \n| 535 to 579 | 12.2 to 12.9 | Data-center GPUs (A100, H100, H200, B200, L40S, L4 and similar), through NVIDIA's CUDA forward compatibility | \n| Older than 535 |  | Not supported; the installer offers to update the driver | \n\nGeForce and TITAN cards need driver 580 or newer. Enough GPU memory for the model is the only other requirement; `goethe recommend` lists the models that fit.\n\nSupported [model families and quantization formats](https://github.com/meshapplied/goethe/blob/main/docs/user-guide.md#supported-models) are in the user guide.\n\n- **[User guide](https://github.com/meshapplied/goethe/blob/main/docs/user-guide.md):** install, add models, share the API, and troubleshooting.\n- **[Command reference](https://github.com/meshapplied/goethe/blob/main/docs/cli.md):** every`goethe` command and option.\n- **[Our models](https://github.com/meshapplied/goethe/blob/main/docs/research-models.md):** the models we publish, and how fast they run.\n- **Full documentation:**[meshapplied.com/docs](https://meshapplied.com/docs) .\n\n| Path | What it is | \n|---|---|\n| [`engine/`](https://github.com/meshapplied/goethe/blob/main/engine) | Goethe: the `goethe` command, the background service, the gateway and the installer | \n| [`nvfp4-kernels/`](https://github.com/meshapplied/goethe/blob/main/nvfp4-kernels) ,[` ternary-kernels/`](https://github.com/meshapplied/goethe/blob/main/ternary-kernels) | GPU kernels behind our models | \n| [`docs/`](https://github.com/meshapplied/goethe/blob/main/docs) | User guide, command reference and model notes | \n\nOur models on [Hugging Face](https://huggingface.co/meshapplied) and at [meshapplied.com/models](https://meshapplied.com/models).\n\nGoethe is licensed under the [Apache License 2.0](https://github.com/meshapplied/goethe/blob/main/LICENSE). The license of each model you serve still applies.\n\nGoethe is built on third-party software, among them SGLang, PyTorch, FlashInfer, Triton, Hugging Face Transformers, nginx and NVIDIA's CUDA libraries. [NOTICE](https://github.com/meshapplied/goethe/blob/main/NOTICE) lists them with their licenses.", "url": "https://wpnews.pro/news/show-hn-goethe-the-infrastructure-for-self-hosted-ai", "canonical_source": "https://github.com/meshapplied/Goethe", "published_at": "2026-11-07 00:26:21+00:00", "updated_at": "2026-11-07 00:50:06.098325+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "large-language-models", "ai-products", "developer-tools"], "entities": ["Goethe", "meshapplied", "NVIDIA", "Llama-3.1-8B-Instruct-NVFP4", "Qwen3.8-27B-NVFP4-NInfer", "OpenAI", "Anthropic", "LiteLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-goethe-the-infrastructure-for-self-hosted-ai", "markdown": "https://wpnews.pro/news/show-hn-goethe-the-infrastructure-for-self-hosted-ai.md", "text": "https://wpnews.pro/news/show-hn-goethe-the-infrastructure-for-self-hosted-ai.txt", "jsonld": "https://wpnews.pro/news/show-hn-goethe-the-infrastructure-for-self-hosted-ai.jsonld"}}