{"slug": "show-hn-self-hosted-inference-for-agents", "title": "Show HN: Self-Hosted Inference for Agents", "summary": "Superlinked released SIE, an open-source inference engine that serves 100+ models for agent tasks through one OpenAI-compatible API, replacing separate model servers per task. The engine supports search, document-to-markdown conversion, structured output, content safety, and agent loops, with on-demand model loading and integrations with LangChain, LlamaIndex, and other frameworks.", "body_md": "**Self-hosted inference for agents. Every open model your agents call, served from one cluster in your cloud.**\n\n[Docs](https://superlinked.com/docs/) |\n[Quickstart](https://superlinked.com/docs/quickstart/) |\n[API Reference](https://superlinked.com/docs/reference/api/) |\n[Models](https://superlinked.com/models)\n\n⭐ *Help us reach more developers and grow the SIE community. Star this repo!*\n\nSIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, and the agent loop itself. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.\n\n- OpenAI-compatible API for drop-in migration:\n`/v1/embeddings`\n\n,`/v1/chat/completions`\n\n,`/v1/completions`\n\n,`/v1/responses`\n\n- Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB\n- Serves multiple models simultaneously with on-demand loading and LRU eviction\n- Ships the full production stack: load-balancing gateway, KEDA autoscaling, Grafana dashboards, Terraform for GKE, EKS, and AKS\n- Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB\n\nThe repository root is a virtual Python workspace. From the repository root, install and verify every workspace member with the committed lock (the audio-prep member requires its documented native build prerequisites):\n\n```\nuv python install 3.12\nuv lock --check\nuv sync --frozen --all-packages\nuv run --frozen --project . --no-sync pytest -c pyproject.toml\n```\n\nPackage membership is explicit in the root `pyproject.toml`\n\n; a package joins\nthe workspace only in the same change that adds its complete source.\n\nOne SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; browse [ packages/sie_server/models/](https://github.com/superlinked/sie/tree/main/packages/sie_server/models) for the full set.\n\n| Task | What it does | Models |\n|---|---|---|\nSearch |\nEmbed, match, and rerank to retrieve the right context. | `bge-m3` , `splade-v3` , `colbertv2` , `qwen3-reranker` |\nDocument to markdown |\nPDFs, Office files, and scans become clean markdown. | `lightonocr` , `glm-ocr` , `mineru` , `paddleocr-vl` , `docling` |\nStructured output |\nSchema-valid JSON, extracted or generated. | `gliner2` , `nuner-zero` , `qwen3.6-27b` |\nGuard content |\nA safety verdict with a probability you threshold. | `granite-guardian-2b` |\nRun the agent loop |\nPlan steps and call tools with an open LLM, streaming included. | `qwen3.6-27b` |\n\nPrefer a notebook? [ examples/quickstart.ipynb](/superlinked/sie/blob/main/examples/quickstart.ipynb) runs this same flow, on your machine or a free Colab GPU.\n\n**1. Start the server**\n\n```\n# macOS (Apple Silicon) or Linux, native (requires Python 3.12)\npip install \"sie-server[local]\" && sie-server serve\n\n# Linux, NVIDIA GPU\ndocker run --gpus all -p 8080:8080 \\\n  -v sie-hf-cache:/app/.cache/huggingface \\\n  ghcr.io/superlinked/sie-server:latest-cuda12-default\n\n# Linux, NVIDIA GPU — Transformers 5 OCR models (LightOnOCR and GLM-OCR)\ndocker run --gpus all -p 8080:8080 \\\n  -v sie-hf-cache:/app/.cache/huggingface \\\n  ghcr.io/superlinked/sie-server:latest-cuda12-transformers5\n\n# Linux, CPU\ndocker run -p 8080:8080 \\\n  -v sie-hf-cache:/app/.cache/huggingface \\\n  ghcr.io/superlinked/sie-server:latest-cpu-default\n```\n\nDocker images are bundle-specific so dependency-incompatible model families stay isolated. Use the\n`transformers5`\n\nimage for LightOnOCR or GLM-OCR; the `default`\n\nimage intentionally does not advertise them.\n\n```\n# in a second terminal\ncurl http://localhost:8080/readyz   # expect: ok\n```\n\nThe server speaks the OpenAI API out of the box, embeddings and generation alike (the cluster gateway serves `/v1/chat/completions`\n\n, `/v1/completions`\n\n, and `/v1/responses`\n\n). Your first call needs nothing but curl:\n\n```\ncurl http://localhost:8080/v1/embeddings \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\": \"sentence-transformers/all-MiniLM-L6-v2\", \"input\": \"Hello world\"}'\n# {\"object\": \"list\", \"data\": [{\"object\": \"embedding\", \"embedding\": [-0.0344, 0.0310, ...\n```\n\nEach model's first call downloads its weights (progress appears in the server terminal). Later calls skip the download; inference latency depends on the model, task, hardware, and batch size.\n\n**2. Install the SDK**\n\n```\npip install sie-sdk                # Python\nnpm install @superlinked/sie-sdk   # TypeScript (pnpm and yarn work too)\n```\n\n**3. Generate embeddings, rerank, and extract entities**\n\n``` python\nfrom sie_sdk import SIEClient\nfrom sie_sdk.types import Item\n\nclient = SIEClient(\"http://localhost:8080\")\n\n# Generate embeddings\nresult = client.encode(\"sentence-transformers/all-MiniLM-L6-v2\", Item(text=\"Hello world\"))\nprint(result[\"dense\"].shape)  # (384,)\n\n# Rerank search results\nscores = client.score(\n    \"cross-encoder/ms-marco-MiniLM-L-6-v2\",\n    Item(text=\"What is machine learning?\"),\n    [Item(text=\"ML learns from data.\"), Item(text=\"The weather is sunny.\")],\n)\nprint(scores[\"scores\"][0])  # {'item_id': 'item-0', 'score': -7.1, 'rank': 0}\n\n# Extract entities\nresult = client.extract(\n    \"urchade/gliner_multi-v2.1\",\n    Item(text=\"Tim Cook is the CEO of Apple.\"),\n    labels=[\"person\", \"organization\"],\n)\nprint(result[\"entities\"][0])\n# {'text': 'Tim Cook', 'label': 'person', 'score': 0.992, 'start': 0, 'end': 8, ...}\n```\n\nText generation runs on the GPU generation image; stop the first server, then start this one on the same port:\n\n```\n# Linux, NVIDIA GPU (for generation on Apple Silicon via MLX, see the docs below)\ndocker run --gpus all -p 8080:8080 \\\n  -v sie-hf-cache:/app/.cache/huggingface \\\n  ghcr.io/superlinked/sie-server:latest-cuda12-sglang\nresult = client.generate(\n    \"Qwen/Qwen3-0.6B\",\n    \"Reply with a single word: the capital of France.\",\n    max_new_tokens=16,\n    temperature=0.0,\n)\nprint(result[\"text\"])  # Paris\n```\n\nFor generation on Apple Silicon (MLX), the TypeScript walkthrough, and every configuration in between, see the [quickstart guide](https://superlinked.com/docs/quickstart/), [TypeScript SDK docs](https://superlinked.com/docs/reference/typescript-sdk/), and [SDK reference](https://superlinked.com/docs/reference/sdk/).\n\nThe same code works against a production cluster. SIE ships a load-balancing gateway, KEDA autoscaling (scale to zero), Grafana dashboards, and Terraform modules for [GKE](https://github.com/superlinked/terraform-google-sie), [EKS](https://github.com/superlinked/terraform-aws-sie), and [AKS](https://github.com/superlinked/terraform-azure-sie). Not just the server, the whole stack. All Apache 2.0.\n\n```\n# pick one values overlay: values-gke.yaml / values-aws.yaml / values-aks.yaml\n# (pin a chart version for reproducible installs, e.g. --version 0.6.18)\nhelm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \\\n  --namespace sie --create-namespace \\\n  --set hfToken.create=true \\\n  --set hfToken.value=YOUR_HF_TOKEN \\\n  -f https://raw.githubusercontent.com/superlinked/sie/main/deploy/helm/sie-cluster/values-gke.yaml\n```\n\nSee the [deployment guide](https://superlinked.com/docs/deployment/).\n\nTelemetry: SIE collects anonymous usage data (version, OS, architecture, GPU type) to understand adoption. No IP addresses, hostnames, or request data are collected. Disable with`SIE_TELEMETRY_DISABLED=1`\n\nor`DO_NOT_TRACK=1`\n\n.\n\n[ Model catalog](https://superlinked.com/models): every model is a config in\n\n[; pass its Hugging Face ID to the SDK.](https://github.com/superlinked/sie/tree/main/packages/sie_server/models)\n\n`packages/sie_server/models/`\n\n[ Integrations](https://superlinked.com/docs/integrations/): setup guides for all nine framework and vector-store integrations, in Python and TypeScript.\n\n[ Examples](/superlinked/sie/blob/main/examples): A quickstart notebook and an end-to-end project gallery.\n\n[ MCP edge](/superlinked/sie/blob/main/packages/sie_mcp): offload document work from Claude and other MCP clients to your cluster and save agent tokens.\n\n[ Why we built SIE](https://www.youtube.com/watch?v=qdh_x-uRs9g): The motivation, told at AI Engineer Europe 2026.\n\n[ superlinked.com/docs](https://superlinked.com/docs) | Apache 2.0", "url": "https://wpnews.pro/news/show-hn-self-hosted-inference-for-agents", "canonical_source": "https://github.com/superlinked/sie", "published_at": "2026-08-10 13:55:11+00:00", "updated_at": "2026-08-10 14:11:30.251266+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "developer-tools", "generative-ai"], "entities": ["Superlinked", "SIE", "OpenAI", "LangChain", "LlamaIndex", "Haystack", "DSPy", "CrewAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-self-hosted-inference-for-agents", "markdown": "https://wpnews.pro/news/show-hn-self-hosted-inference-for-agents.md", "text": "https://wpnews.pro/news/show-hn-self-hosted-inference-for-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-self-hosted-inference-for-agents.jsonld"}}