Show HN: TensorSharp: Open-Source Local LLM Inference Engine

TensorSharp, an open-source local LLM inference engine for GGUF models, has been released by developer Zhongkai Fu. The .NET-based engine runs on personal hardware, supports NVIDIA, AMD, Intel, and Apple Silicon, and offers Ollama- and OpenAI-compatible APIs, providing a private, cost-free alternative to cloud-based AI services.

TensorSharp A native .NET LLM inference engine for GGUF models — with a command-line tool, a browser chat server, and Ollama- & OpenAI-compatible APIs for programmatic access. Everything runs on your own hardware : your laptop, workstation, or server. No data leaves the machine, there are no per-token fees, and the same engine powers a quick command-line test, a shared internal chatbot, and a production REST endpoint. This wiki is the complete guide — pick a starting point below or use / to search. Explore the wiki 🚀 getting-started.html Getting Started Prerequisites, build, download a model, and stream your first reply. ⌨️ cli.html Command Line Run prompts, images, audio, batches, and benchmarks from the CLI. 🌐 server.html Server & Web UI Host a browser chatbot and HTTP endpoints on localhost. 🔌 http-api.html HTTP API Call it from curl, Python, or any Ollama/OpenAI client. 🧩 code-api.html C Library Embed the engine directly in your .NET application. 📚 api-reference.html API Reference Searchable tables of flags, env vars, endpoints, and types. 🧠 models.html Models Supported architectures, downloads, multimodal, and reasoning. 📖 glossary.html Glossary & FAQ New to LLMs? Plain-language definitions and common questions. Quick start in ~30 seconds After installing the .NET 10 SDK getting-started.html prerequisites , you are four commands away from a streaming reply model download aside . - Clone & build The native GGML library compiles automatically on the first build. git clone https://github.com/zhongkaifu/TensorSharp.git cd TensorSharp dotnet build TensorSharp.slnx -c Release - Download a model A small, well-tested starting point is Gemma-4-E4B Q8 0 from Hugging Face. More in Model downloads models.html downloads . - Run it Pick the for your hardware. --backend echo "Explain mixture-of-experts in one sentence." prompt.txt macOS Apple Silicon ./TensorSharp.Cli --model gemma-4-E4B-it-Q8 0.gguf --input prompt.txt --backend ggml metal Windows / Linux + NVIDIA ./TensorSharp.Cli --model gemma-4-E4B-it-Q8 0.gguf --input prompt.txt --backend ggml cuda - Prefer a UI + API? Start the server and open the browser chat — it also serves the compatibility endpoints. ./TensorSharp.Server --model gemma-4-E4B-it-Q8 0.gguf --backend ggml metal open http://localhost:5000 Why TensorSharp? Private by default Inference happens on your hardware. Prompts, documents, and images never leave the machine. No per-token bill Run as much as your hardware allows — predictable cost, no metered API. Drop-in compatible Speaks the Ollama and OpenAI wire formats, so existing tools and SDKs just work. Runs anywhere NVIDIA CUDA , AMD / Intel / NVIDIA Vulkan , Apple Silicon Metal/MLX , or pure CPU — with automatic fallbacks. Modern model support Gemma, Qwen, GPT-OSS, Nemotron-H, Mistral, plus vision, audio, reasoning & tools. Built in .NET A native C engine you can embed in your apps, not just a black-box binary. 🏁 benchmarks.html head-to-head Benchmarked vs llama.cpp On identical GGUF files and the same GPU it trades wins with the C++ engine: the 26B-A4B MoE prefills 1.32× faster with first tokens 1.30× sooner, 12B wins or ties every decode scenario 1.17× , and JSON-mode decode streams 7.7× faster on E4B. Who is this for? TensorSharp serves a wide range of visitors. Here is the fastest path for each. Beginners & students Start with the Glossary & FAQ glossary.html , then Getting Started getting-started.html . Developers Jump to the HTTP API http-api.html , C Library code-api.html , and API Reference api-reference.html . Senior / principal engineers Read Advanced Features advanced.html — paged KV, continuous batching, speculative decoding. Managers, CTOs & CEOs See the business value overview.html business and capability matrix overview.html status . Sales & marketing Use the feature catalog features.html and benchmarks benchmarks.html for positioning. Researchers & professors Explore model architectures models.html and the head-to-head benchmarks benchmarks.html head-to-head .