{"slug": "show-hn-tensorsharp-open-source-local-llm-inference-engine", "title": "Show HN: TensorSharp: Open-Source Local LLM Inference Engine", "summary": "TensorSharp, an open-source local LLM inference engine for GGUF models, has been released by developer Zhongkai Fu. The .NET-based engine runs on personal hardware, supports NVIDIA, AMD, Intel, and Apple Silicon, and offers Ollama- and OpenAI-compatible APIs, providing a private, cost-free alternative to cloud-based AI services.", "body_md": "# TensorSharp\n\nA **native .NET LLM inference engine for GGUF models** — with a command-line tool, a browser chat server, and **Ollama- & OpenAI-compatible APIs** for programmatic access.\n\nEverything runs on **your own hardware**: your laptop, workstation, or server. No data leaves the machine, there are no per-token fees, and the same engine powers a quick command-line test, a shared internal chatbot, and a production REST endpoint. This wiki is the complete guide — pick a starting point below or use `/` to search.\n\n## Explore the wiki\n\n[🚀](getting-started.html)\n\n### Getting Started\n\nPrerequisites, build, download a model, and stream your first reply.\n\n[⌨️](cli.html)\n\n### Command Line\n\nRun prompts, images, audio, batches, and benchmarks from the CLI.\n\n[🌐](server.html)\n\n### Server & Web UI\n\nHost a browser chatbot and HTTP endpoints on localhost.\n\n[🔌](http-api.html)\n\n### HTTP API\n\nCall it from curl, Python, or any Ollama/OpenAI client.\n\n[🧩](code-api.html)\n\n### C# Library\n\nEmbed the engine directly in your .NET application.\n\n[📚](api-reference.html)\n\n### API Reference\n\nSearchable tables of flags, env vars, endpoints, and types.\n\n[🧠](models.html)\n\n### Models\n\nSupported architectures, downloads, multimodal, and reasoning.\n\n[📖](glossary.html)\n\n### Glossary & FAQ\n\nNew to LLMs? Plain-language definitions and common questions.\n\n## Quick start in ~30 seconds\n\nAfter installing the [.NET 10 SDK](getting-started.html#prerequisites), you are four commands away from a streaming reply (model download aside).\n\n-\n### Clone & build\n\nThe native GGML library compiles automatically on the first build.\n\n```\ngit clone https://github.com/zhongkaifu/TensorSharp.git\ncd TensorSharp\ndotnet build TensorSharp.slnx -c Release\n```\n\n-\n### Download a model\n\nA small, well-tested starting point is\n\n**Gemma-4-E4B (Q8_0)** from Hugging Face. More in[Model downloads](models.html#downloads). -\n### Run it\n\nPick the\n\nfor your hardware.`--backend`\n\n```\necho \"Explain mixture-of-experts in one sentence.\" > prompt.txt\n\n# macOS (Apple Silicon)\n./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --backend ggml_metal\n\n# Windows / Linux + NVIDIA\n./TensorSharp.Cli --model gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --backend ggml_cuda\n```\n\n-\n### Prefer a UI + API?\n\nStart the server and open the browser chat — it also serves the compatibility endpoints.\n\n```\n./TensorSharp.Server --model gemma-4-E4B-it-Q8_0.gguf --backend ggml_metal\n# open http://localhost:5000\n```\n\n## Why TensorSharp?\n\n### Private by default\n\nInference happens on your hardware. Prompts, documents, and images never leave the machine.\n\n### No per-token bill\n\nRun as much as your hardware allows — predictable cost, no metered API.\n\n### Drop-in compatible\n\nSpeaks the Ollama and OpenAI wire formats, so existing tools and SDKs just work.\n\n### Runs anywhere\n\nNVIDIA (CUDA), AMD / Intel / NVIDIA (Vulkan), Apple Silicon (Metal/MLX), or pure CPU — with automatic fallbacks.\n\n### Modern model support\n\nGemma, Qwen, GPT-OSS, Nemotron-H, Mistral, plus vision, audio, reasoning & tools.\n\n### Built in .NET\n\nA native C# engine you can embed in your apps, not just a black-box binary.\n\n[🏁](benchmarks.html#head-to-head)\n\n### Benchmarked vs llama.cpp\n\nOn identical GGUF files and the same GPU it trades wins with the C++ engine: the 26B-A4B MoE prefills 1.32× faster with first tokens 1.30× sooner, 12B wins or ties every decode scenario (1.17×), and JSON-mode decode streams 7.7× faster on E4B.\n\n## Who is this for?\n\nTensorSharp serves a wide range of visitors. Here is the fastest path for each.\n\n#### Beginners & students\n\nStart with the [Glossary & FAQ](glossary.html), then [Getting Started](getting-started.html).\n\n#### Developers\n\nJump to the [HTTP API](http-api.html), [C# Library](code-api.html), and [API Reference](api-reference.html).\n\n#### Senior / principal engineers\n\nRead [Advanced Features](advanced.html) — paged KV, continuous batching, speculative decoding.\n\n#### Managers, CTOs & CEOs\n\nSee the [business value](overview.html#business) and [capability matrix](overview.html#status).\n\n#### Sales & marketing\n\nUse the [feature catalog](features.html) and [benchmarks](benchmarks.html) for positioning.\n\n#### Researchers & professors\n\nExplore [model architectures](models.html) and the [head-to-head benchmarks](benchmarks.html#head-to-head).", "url": "https://wpnews.pro/news/show-hn-tensorsharp-open-source-local-llm-inference-engine", "canonical_source": "https://tensorsharp.ai/", "published_at": "2026-07-10 02:42:07+00:00", "updated_at": "2026-07-10 03:06:05.966634+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "ai-products"], "entities": ["TensorSharp", "Zhongkai Fu", "GGUF", "Ollama", "OpenAI", "Hugging Face", "NVIDIA", "Apple"], "alternates": {"html": "https://wpnews.pro/news/show-hn-tensorsharp-open-source-local-llm-inference-engine", "markdown": "https://wpnews.pro/news/show-hn-tensorsharp-open-source-local-llm-inference-engine.md", "text": "https://wpnews.pro/news/show-hn-tensorsharp-open-source-local-llm-inference-engine.txt", "jsonld": "https://wpnews.pro/news/show-hn-tensorsharp-open-source-local-llm-inference-engine.jsonld"}}