Show HN: Serve open models on your own infra in 3 steps

Goethe launched a tool that turns a Linux machine with NVIDIA GPUs into a self-hosted model-serving API compatible with OpenAI, Anthropic and LiteLLM clients, set up in three steps via `goethe start` or the `goethe up`, `goethe install --model Qwen/Qwen3-8B --load` and `goethe configure api` commands. Goethe runs models on SGLang with continuous batching, prefix caching and CUDA graphs, and adds its own NVFP4 kernels for Blackwell, Hopper, Ada and Ampere GPUs so 4-bit models use about a quarter of the memory of full precision. The tool ships its own CUDA 13 runtime, requires NVIDIA driver 580 or newer for all supported GPUs (535 to 579 works only on data-center GPUs via forward compatibility), and supports Linux x86_64 or ARM64 with glibc 2.30 or newer.

Run and share your models on your own infra in 3 steps, and run them fast. Goethe turns a Linux machine into your own API, compatible with OpenAI, Anthropic and LiteLLM clients. Install it, add a model, and point your apps, your team or your other machines at one address. Your prompts, your data and your models stay on hardware you control. curl -fsSL https://meshapplied.com/install.sh | bash goethe start goethe start starts Goethe and walks you through three steps: install a model, serve it through the API, and configure who can reach it. If you prefer plain commands, these do the same: 1. Start Goethe and load a model goethe up goethe install --model Qwen/Qwen3-8B --load 2. Share it goethe configure api this machine only, your VPN recommended , or a public address goethe key --create my-laptop one key per app or person goethe api the address to give your apps Anything that speaks the OpenAI, Anthropic or LiteLLM API can use it right away, all at the same address: the OpenAI and Anthropic SDKs, LiteLLM, LangChain, LlamaIndex, Open WebUI, coding assistants, or plain curl . python from openai import OpenAI client = OpenAI base url="https://<your-goethe-address :8443/v1", api key="goethe sk ..." reply = client.chat.completions.create model="Qwen/Qwen3-8B", messages= {"role": "user", "content": "Hello "} , print reply.choices 0 .message.content - Three steps, one address. Every model you load is served at the same URL; a request picks its model by name. - Fast. Models run on SGLang, with continuous batching, prefix caching and CUDA graphs. Goethe adds its own NVFP4 kernels for Blackwell, Hopper, Ada and Ampere GPUs, so 4-bit models use about a quarter of the memory of the full-precision version and stay quick. - Private by default. Nothing can reach the API until you allow it, you can rate-limit, expire, scope or revoke keys at any time. - Fits your network. Serve on your own infra only, over Tailscale, WireGuard or ZeroTier, or on a public IP. goethe configure api walks you through the choice. - Many models, many GPUs. Run several models side by side, split large models across GPUs, and keep other GPU software running next to Goethe. - Always on. Loaded models come back by themselves after a reboot or a crash. | Environment | Examples | |---|---| | Your own hardware | Workstations and servers with one or more NVIDIA GPUs, including DGX Spark | | Cloud GPU virtual machines | AWS, Google Cloud, Azure, Oracle Cloud, Lambda, CoreWeave, Paperspace | | GPU containers | RunPod, Vast.ai, and Docker with the NVIDIA Container Toolkit | Linux on x86 64 or ARM64 glibc 2.30 or newer, for example Ubuntu 20.04+ or Debian 11+ . Goethe recognises the cloud provider or container it runs in and shows the address to use from outside. | Architecture | Compute capability | GPUs | |---|---|---| | Blackwell | 12.0, 12.1 | GeForce RTX 5090, RTX PRO 6000 Blackwell, DGX Spark GB10 | | Blackwell data center | 10.0, 10.3 | B200, GB200, B300, GB300 | | Hopper | 9.0 | H100, H200, GH200 | | Ada | 8.9 | GeForce RTX 4090, RTX 6000 Ada, L40S, L40, L4 | | Ampere | 8.0, 8.6 | A100, A30, A40, A10, RTX A6000, GeForce RTX 3090 | Older GPUs Volta, Turing are not supported. A machine can mix GPU generations; each model runs on GPUs of one generation, and large models can be split across several GPUs. Goethe ships its own CUDA 13 runtime and compiler, so no CUDA toolkit is needed on the machine. What it needs is the NVIDIA driver: | NVIDIA driver | CUDA it offers | Goethe runs on | |---|---|---| | 580 or newer | 13.0 and newer | Every supported GPU | | 535 to 579 | 12.2 to 12.9 | Data-center GPUs A100, H100, H200, B200, L40S, L4 and similar , through NVIDIA's CUDA forward compatibility | | Older than 535 | | Not supported; the installer offers to update the driver | GeForce and TITAN cards need driver 580 or newer. Enough GPU memory for the model is the only other requirement; goethe recommend lists the models that fit. Supported model families and quantization formats https://github.com/meshapplied/goethe/blob/main/docs/user-guide.md what-goethe-supports are in the user guide. - User guide https://github.com/meshapplied/goethe/blob/main/docs/user-guide.md : install, add models, share the API, and troubleshooting. - Command reference https://github.com/meshapplied/goethe/blob/main/docs/cli.md : every goethe command and option. - Our models https://github.com/meshapplied/goethe/blob/main/docs/research-models.md : the ternary and NVFP4 models we publish, and how fast they run. - Full documentation: meshapplied.com/docs https://meshapplied.com/docs . | Path | What it is | |---|---| | engine/ https://github.com/meshapplied/goethe/blob/main/engine | Goethe: the goethe command, the background service, the gateway and the installer | | nvfp4-kernels/ https://github.com/meshapplied/goethe/blob/main/nvfp4-kernels , ternary-kernels/ https://github.com/meshapplied/goethe/blob/main/ternary-kernels | GPU kernels behind our NVFP4 and ternary models | | docs/ https://github.com/meshapplied/goethe/blob/main/docs | User guide, command reference and model notes | Our models on Hugging Face https://huggingface.co/meshapplied and at meshapplied.com/models https://meshapplied.com/models . Goethe is licensed under the Apache License 2.0 https://github.com/meshapplied/goethe/blob/main/LICENSE . The license of each model you serve still applies. Goethe is built on third-party software, among them SGLang, PyTorch, FlashInfer, Triton, Hugging Face Transformers, nginx and NVIDIA's CUDA libraries. NOTICE https://github.com/meshapplied/goethe/blob/main/NOTICE lists them with their licenses.