# Show HN: Self-Hosted Inference for Agents

> Source: <https://github.com/superlinked/sie>
> Published: 2026-08-10 13:55:11+00:00

**Self-hosted inference for agents. Every open model your agents call, served from one cluster in your cloud.**

[Docs](https://superlinked.com/docs/) |
[Quickstart](https://superlinked.com/docs/quickstart/) |
[API Reference](https://superlinked.com/docs/reference/api/) |
[Models](https://superlinked.com/models)

⭐ *Help us reach more developers and grow the SIE community. Star this repo!*

SIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, and the agent loop itself. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.

- OpenAI-compatible API for drop-in migration:
`/v1/embeddings`

,`/v1/chat/completions`

,`/v1/completions`

,`/v1/responses`

- Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB
- Serves multiple models simultaneously with on-demand loading and LRU eviction
- Ships the full production stack: load-balancing gateway, KEDA autoscaling, Grafana dashboards, Terraform for GKE, EKS, and AKS
- Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB

The repository root is a virtual Python workspace. From the repository root, install and verify every workspace member with the committed lock (the audio-prep member requires its documented native build prerequisites):

```
uv python install 3.12
uv lock --check
uv sync --frozen --all-packages
uv run --frozen --project . --no-sync pytest -c pyproject.toml
```

Package membership is explicit in the root `pyproject.toml`

; a package joins
the workspace only in the same change that adds its complete source.

One SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; browse [ packages/sie_server/models/](https://github.com/superlinked/sie/tree/main/packages/sie_server/models) for the full set.

| Task | What it does | Models |
|---|---|---|
Search |
Embed, match, and rerank to retrieve the right context. | `bge-m3` , `splade-v3` , `colbertv2` , `qwen3-reranker` |
Document to markdown |
PDFs, Office files, and scans become clean markdown. | `lightonocr` , `glm-ocr` , `mineru` , `paddleocr-vl` , `docling` |
Structured output |
Schema-valid JSON, extracted or generated. | `gliner2` , `nuner-zero` , `qwen3.6-27b` |
Guard content |
A safety verdict with a probability you threshold. | `granite-guardian-2b` |
Run the agent loop |
Plan steps and call tools with an open LLM, streaming included. | `qwen3.6-27b` |

Prefer a notebook? [ examples/quickstart.ipynb](/superlinked/sie/blob/main/examples/quickstart.ipynb) runs this same flow, on your machine or a free Colab GPU.

**1. Start the server**

```
# macOS (Apple Silicon) or Linux, native (requires Python 3.12)
pip install "sie-server[local]" && sie-server serve

# Linux, NVIDIA GPU
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-default

# Linux, NVIDIA GPU — Transformers 5 OCR models (LightOnOCR and GLM-OCR)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-transformers5

# Linux, CPU
docker run -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cpu-default
```

Docker images are bundle-specific so dependency-incompatible model families stay isolated. Use the
`transformers5`

image for LightOnOCR or GLM-OCR; the `default`

image intentionally does not advertise them.

```
# in a second terminal
curl http://localhost:8080/readyz   # expect: ok
```

The server speaks the OpenAI API out of the box, embeddings and generation alike (the cluster gateway serves `/v1/chat/completions`

, `/v1/completions`

, and `/v1/responses`

). Your first call needs nothing but curl:

```
curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Hello world"}'
# {"object": "list", "data": [{"object": "embedding", "embedding": [-0.0344, 0.0310, ...
```

Each model's first call downloads its weights (progress appears in the server terminal). Later calls skip the download; inference latency depends on the model, task, hardware, and batch size.

**2. Install the SDK**

```
pip install sie-sdk                # Python
npm install @superlinked/sie-sdk   # TypeScript (pnpm and yarn work too)
```

**3. Generate embeddings, rerank, and extract entities**

``` python
from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Generate embeddings
result = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Hello world"))
print(result["dense"].shape)  # (384,)

# Rerank search results
scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text="What is machine learning?"),
    [Item(text="ML learns from data."), Item(text="The weather is sunny.")],
)
print(scores["scores"][0])  # {'item_id': 'item-0', 'score': -7.1, 'rank': 0}

# Extract entities
result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text="Tim Cook is the CEO of Apple."),
    labels=["person", "organization"],
)
print(result["entities"][0])
# {'text': 'Tim Cook', 'label': 'person', 'score': 0.992, 'start': 0, 'end': 8, ...}
```

Text generation runs on the GPU generation image; stop the first server, then start this one on the same port:

```
# Linux, NVIDIA GPU (for generation on Apple Silicon via MLX, see the docs below)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-sglang
result = client.generate(
    "Qwen/Qwen3-0.6B",
    "Reply with a single word: the capital of France.",
    max_new_tokens=16,
    temperature=0.0,
)
print(result["text"])  # Paris
```

For generation on Apple Silicon (MLX), the TypeScript walkthrough, and every configuration in between, see the [quickstart guide](https://superlinked.com/docs/quickstart/), [TypeScript SDK docs](https://superlinked.com/docs/reference/typescript-sdk/), and [SDK reference](https://superlinked.com/docs/reference/sdk/).

The same code works against a production cluster. SIE ships a load-balancing gateway, KEDA autoscaling (scale to zero), Grafana dashboards, and Terraform modules for [GKE](https://github.com/superlinked/terraform-google-sie), [EKS](https://github.com/superlinked/terraform-aws-sie), and [AKS](https://github.com/superlinked/terraform-azure-sie). Not just the server, the whole stack. All Apache 2.0.

```
# pick one values overlay: values-gke.yaml / values-aws.yaml / values-aks.yaml
# (pin a chart version for reproducible installs, e.g. --version 0.6.18)
helm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \
  --namespace sie --create-namespace \
  --set hfToken.create=true \
  --set hfToken.value=YOUR_HF_TOKEN \
  -f https://raw.githubusercontent.com/superlinked/sie/main/deploy/helm/sie-cluster/values-gke.yaml
```

See the [deployment guide](https://superlinked.com/docs/deployment/).

Telemetry: SIE collects anonymous usage data (version, OS, architecture, GPU type) to understand adoption. No IP addresses, hostnames, or request data are collected. Disable with`SIE_TELEMETRY_DISABLED=1`

or`DO_NOT_TRACK=1`

.

[ Model catalog](https://superlinked.com/models): every model is a config in

[; pass its Hugging Face ID to the SDK.](https://github.com/superlinked/sie/tree/main/packages/sie_server/models)

`packages/sie_server/models/`

[ Integrations](https://superlinked.com/docs/integrations/): setup guides for all nine framework and vector-store integrations, in Python and TypeScript.

[ Examples](/superlinked/sie/blob/main/examples): A quickstart notebook and an end-to-end project gallery.

[ MCP edge](/superlinked/sie/blob/main/packages/sie_mcp): offload document work from Claude and other MCP clients to your cluster and save agent tokens.

[ Why we built SIE](https://www.youtube.com/watch?v=qdh_x-uRs9g): The motivation, told at AI Engineer Europe 2026.

[ superlinked.com/docs](https://superlinked.com/docs) | Apache 2.0