{"slug": "building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference", "title": "Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference", "summary": "A developer has released Sluice, an open-source QoS-aware capacity governance proxy for self-hosted LLM inference. Sluice sits in front of vLLM deployments to enforce tenant-based service tiers, preserving guaranteed traffic while shedding best-effort requests under GPU memory pressure. The project is available on GitHub.", "body_md": "📦 Project: [https://github.com/VampiricCyborg/sluice](https://github.com/VampiricCyborg/sluice)\n\nA self-hosted vLLM deployment runs on a GPU pool of fixed size. That's the fact that changes everything about how you have to think about load.\n\nOnce that pool's KV-cache capacity comes under pressure, requests don't naturally understand business tiers. They don't know that one tenant is an enterprise customer with a contract, another is an internal team, and a third is a batch job that can wait. Without something external saying otherwise, the system's behavior under contention is driven by arrival order and backend scheduling — and every tenant, regardless of what they were promised, experiences the same degraded latency, queueing, or failure.\n\nThere's no built-in mechanism that says:\n\nPreserve Guaranteed traffic, reduce the work done by Standard traffic, and shed Best-Effort traffic first.\n\nSluice exists to make that decision — before the request ever reaches vLLM.\n\nConcretely, it evaluates three live signals per decision cycle:\n\nTenants are mapped to tiers in `config/tenants.yaml`\n\n:\n\n```\ntenants:\n  support-bot: Guaranteed\n  coding-assistant: Standard\n  batch-analytics: Best-Effort\n```\n\nand each tier has an explicit SLA target, hard-coded in `sluice_proxy/app.py`\n\n:\n\n```\nsla_targets = {\n    \"Guaranteed\": 2000,\n    \"Standard\": 1000,\n    \"Best-Effort\": 500,\n}\n```\n\nThe important thing to understand about Sluice's scope from the start: it is not trying to make vLLM's own execution more efficient. It governs *which* requests reach the inference engine, *how much work* they're allowed to request, and *which backend pool* they land on. Everything downstream of that decision is still vLLM's job.\n\nIt's worth being precise about why this problem doesn't already have a home in the stack, because Sluice's design only makes sense in contrast to the layers around it.\n\n**Kubernetes** can place workloads, restart failed replicas, and add capacity if an autoscaler is configured. None of that helps when GPU capacity is fixed or slow to provision — which, for self-hosted inference, is closer to the default case than the exception. Kubernetes has no concept of a \"tenant\" or a \"tier\" at the request level; it schedules pods, not chat completions.\n\n**vLLM's own scheduler** is the right layer for token-level execution, batching, and KV-cache management inside a worker — and it's *better* at that than anything sitting in front of it could be. Sluice deliberately stays outside that loop. It has no interest in improving token scheduling; it adds tenant identity, tier semantics, and an auditable reason for each request-level action, then gets out of the way.\n\n**A generic API gateway** (Envoy, Kong) gives you authentication, routing, retries, and rate limiting — genuinely useful, and Sluice doesn't try to replace any of it. But rate limits alone can't express \"degrade this tier's `max_tokens`\n\nbefore rejecting that tier's requests, based on live GPU pressure.\" That's a specialized policy signal and action set that a generic gateway has no vocabulary for.\n\nThe actual boundary in code is narrow. The original Phase 1 implementation of `sluice_proxy/app.py`\n\ndid five things: identify the tenant from request headers, look up their tier, read cache pressure, apply a policy decision, and optionally rewrite `max_tokens`\n\nbefore forwarding. It never touched token scheduling, batching, or KV-cache allocation — those stayed vLLM's problem, from the very first commit.\n\nThe `is_shapeable()`\n\nfunction makes this boundary explicit today. Only `POST`\n\n, `PUT`\n\n, `PATCH`\n\nrequests to `/v1/chat/completions`\n\nor `/v1/completions`\n\nare eligible for shaping at all. Everything else passes through untouched. And even for shapeable requests, Sluice only ever changes two fields: `max_tokens`\n\n(via `apply_max_tokens()`\n\n) and `model`\n\n(via `apply_model()`\n\n, for fallback routing). That's the entire surface area of what Sluice is willing to touch in a request body.\n\nThe same discipline shows up in cluster routing. `ClusterRoutingStage`\n\nin `sluice_proxy/policy.py`\n\ncarries this as an explicit design intent in its own docstring:\n\nQoS route between pools, never utilization/load balance.\n\nRouting is a service-policy decision — Guaranteed always gets on-demand, Standard gets on-demand, Best-Effort prefers spot unless spot pressure or a health failure forces eviction — not a round-robin balancing act. And the Kubernetes deployment in `k8s/sluice-ops.yaml`\n\nis intentionally just a Deployment, a Service, a ConfigMap, and probes. No operator, no autoscaler, no scheduler extension.\n\nBefore getting into design, it's worth stating plainly what Sluice owns, because everything downstream follows from this boundary. The clearest way to see it is through its actual interfaces.\n\nThe request path is a single catch-all:\n\n```\n/v1/{path:path}\n```\n\nFor every request that hits it, Sluice does exactly seven things: resolve tenant and tier, read current telemetry, evaluate policy, apply shaping/queueing/fallback/rejection/routing, forward to the selected backend, record the result in PostgreSQL, and emit a structured log.\n\nOperational visibility is a separate, small surface: `/healthz`\n\n, `/livez`\n\n, `/readyz`\n\n, `/metrics`\n\n, `/status`\n\n, `/decisions/recent`\n\n. And for local development, a narrow set of simulation endpoints — `PUT`\n\n/`DELETE`\n\non `/admin/pressure`\n\n, `/admin/queue-depth`\n\n, `/admin/sla-violation-rate`\n\n— let you force a signal value without waiting on real telemetry. These are intentionally minimal development controls, not a control-plane API, and they stay unauthenticated even when API-key auth is otherwise enabled on `/v1/*`\n\n— a decision that's convenient for local testing and a real risk if ever exposed (more on that in Section 11).\n\nThe external dependencies are equally explicit: Prometheus for live inference telemetry, PostgreSQL for durable decisions and SLA history, Redis for distributed queue/admission state, and any number of vLLM-compatible HTTP backends. Sluice doesn't implement any of these things — it coordinates between them.\n\n```\n                 REQUEST\n                    │\n                    ▼\n            ┌──────────────┐\n            │    Sluice    │\n            └──────┬───────┘\n                   │\n      ┌────────────▼────────────┐\n      │   Pressure Evaluation   │\n      │ KV Cache │ Queue │ SLA  │\n      └────────────┬────────────┘\n                   │\n                   ▼\n           ┌───────────────┐\n           │ Tenant Policy │\n           └───────┬───────┘\n                   │\n                   ▼\n           ┌───────────────┐\n           │ Action Select │\n           │ Pass/Degrade  │\n           │ Queue/Fallback│\n           │ Reject        │\n           └───────┬───────┘\n                   │\n                   ▼\n           ┌───────────────┐\n           │ Cluster Route │\n           └───────┬───────┘\n                   │\n          ┌────────┴────────┐\n          ▼                 ▼\n    On-Demand GPU       Spot GPU\n```\n\nSluice's decision logic didn't start as a pipeline. It started as one function.\n\nIn the first commit of the policy engine (`c101afb`\n\n), `CapacityPolicy`\n\nhad a single `decide()`\n\nmethod with nested tier conditionals:\n\n```\nif tier == \"Best-Effort\":\n    if pressure >= 60:\n        reject\nelif tier == \"Standard\":\n    if pressure >= 90:\n        reject\n    if pressure >= 70:\n        degrade\nelif tier == \"Guaranteed\":\n    if pressure >= 100:\n        reject\n    if pressure >= 95:\n        degrade\n```\n\nThat was fine when there was exactly one signal. It stopped being fine the moment a second one showed up. Commit `91c00fa`\n\nextended `decide()`\n\nto accept `pressure`\n\n, `queue_depth`\n\n, and `sla_violation_rate`\n\ntogether, and the shape it took was still recognizably one function — checking whether *any* signal crossed a reject threshold, then whether any crossed a degrade threshold. It worked, but the conditional surface was growing in a direction that wasn't going to scale to fallback routing, queueing, or multi-cluster decisions without becoming unreadable.\n\nCommit `ca80787`\n\nis where the real refactor happened: four explicit stages, each owning one concern.\n\n``` php\nPressureEvaluationStage.evaluate(tier, signals) -> PressureEvaluation\nTenantPolicyStage.apply(evaluation, *, shapeable, fallback_active, fallback_model) -> PressureEvaluation\nActionSelectionStage.select(evaluation, *, shapeable, fallback_active, fallback_model) -> PolicyDecision\nClusterRoutingStage.route(tier, decision, clusters, health) -> PolicyDecision\n```\n\norchestrated by a single `PolicyPipeline.run()`\n\n. Pressure Evaluation turns raw signals into a named state (Normal, Elevated, Critical). Tenant Policy is the extension point for tier-specific semantics. Action Selection picks the actual intervention. Cluster Routing decides which pool handles it.\n\nI want to be honest about one detail here rather than present this as a clean four-stage design from the start: `TenantPolicyStage.apply()`\n\nis currently a no-op — it returns the evaluation unchanged. The refactor established the stage *boundary*, but not every piece of tier-specific behavior actually lives there yet. Some of it is still inside `CapacityPolicy._select_action()`\n\nand `_thresholds_for()`\n\n. That's a real, current gap between the architecture's intent and its full implementation — not a mistake, just unfinished separation. The pipeline shape was worth building before every piece of logic was moved into it, because it made everything that came after (fallback, queueing, multi-cluster routing) addable without another rewrite.\n\nThree named states drive everything downstream:\n\n```\nPressureState.NORMAL\nPressureState.ELEVATED\nPressureState.CRITICAL\n```\n\nThe combination rule across three signals is deliberately conservative — a max-severity OR, not an average: any available signal crossing a reject threshold produces `CRITICAL`\n\n; failing that, any signal crossing a degrade threshold produces `ELEVATED`\n\n; otherwise `NORMAL`\n\n. If every signal is unavailable, the policy fails open rather than guessing.\n\nThe stabilization defaults, set in `CapacityPolicy.__init__()`\n\n:\n\n```\nSLUICE_STATE_CONSECUTIVE_SAMPLES = 2\nSLUICE_STATE_COOLDOWN_SECONDS = 2\nSLUICE_STATE_HYSTERESIS = 5\n```\n\nI'll say up front that these are operational defaults, not values fitted to production traces — there's no evidence in the repository of them being derived from real traffic, and that's worth being upfront about rather than implying more rigor than exists.\n\nWhat they buy you is protection against a specific, well-understood failure mode: a signal hovering right at a threshold causing the system to flip actions on every poll. `PressureStabilityModel.observe()`\n\nenforces three conditions before a transition is allowed to happen: the new state must persist for two consecutive samples, it must respect the cooldown window, and moving *down* a severity level requires crossing a lower recovery threshold than the one that triggered the escalation — not just dipping back under the original line.\n\n`tests/test_fallback_queue.py`\n\nshows this concretely. Starting from Normal:\n\n``` php\n75 at t=1 -> no transition (first sample, waiting for confirmation)\n75 at t=2 -> Elevated (confirmed)\n\n69 at t=3 -> remains Elevated (below the up-threshold of 70, but still above recovery threshold of 65)\n64 at t=4 -> no transition (waiting for confirmation)\n64 at t=5 -> Normal (confirmed)\n```\n\nThe interesting line is `69 at t=3`\n\n. Without an asymmetric recovery threshold, a value that dips just under the escalation threshold would immediately flip the state back — and if pressure is oscillating around 70, that's a state flip on every poll. With the wider recovery band, 69 simply isn't low enough to count as recovery yet.\n\nI should be precise about what evidence backs this: there's no log excerpt in the repository from a real pre-hysteresis deployment flapping in production — the evidence is test-driven, not observed. The synthetic benchmark's Phase 2 run does record `action_flapping_count: 0`\n\n, but that demonstrates the *stabilized* simulator didn't flap under a controlled pressure timeline; it doesn't prove a live, unstabilized system would have.\n\nFallback gets its own tracker with its own defaults, since \"sustained\" pressure for a fallback decision is a meaningfully different bar than a degrade decision:\n\n```\nSLUICE_FALLBACK_PRESSURE_AT = 85\nSLUICE_FALLBACK_RECOVER_BELOW = 75\nSLUICE_FALLBACK_SUSTAINED_INTERVALS = 3\nSLUICE_FALLBACK_COOLDOWN_SECONDS = 2\n```\n\nSluice has five possible outcomes for a request, escalating roughly with tier and pressure.\n\n**Pass** is the default — signals under threshold, a non-shapeable endpoint, or telemetry unavailable and failing open. The reason is always recorded explicitly (`below_tier_pressure_threshold`\n\n, `cache_pressure_unavailable_fail_open`\n\n, `endpoint_not_shapeable`\n\n, `signals_unavailable_fail_open`\n\n), because a \"pass\" with no explanation is just as much a policy decision as a rejection and deserves the same audit trail.\n\n**Degrade** only applies to shapeable completion endpoints, and only lowers `max_tokens`\n\n— never touches anything else in the payload:\n\n```\nSTANDARD_MAX_TOKENS = 256\nGUARANTEED_MAX_TOKENS = 512\n```\n\n`apply_max_tokens()`\n\nis conservative by design: if `max_tokens`\n\nis missing, set it to the cap; if it's not an integer, set it to the cap; if it exceeds the cap, replace it; if it's already under the cap, leave it alone. Best-Effort has no degrade step at all — it moves straight toward rejection, since there's no meaningful \"smaller\" version of a request that's already lowest priority.\n\n**Queue** is Standard-tier only, and only fires in the Elevated state — Guaranteed never queues, and Best-Effort doesn't get access to the Standard queue. The bounds are tight on purpose:\n\n```\nSLUICE_QUEUE_MAX_SIZE = 100\nSLUICE_QUEUE_MAX_WAIT_SECONDS = 2\nSLUICE_QUEUE_POLL_SECONDS = 0.05\n```\n\nThe local, single-replica implementation is a `BoundedRequestQueue`\n\n: acquire a lock, check size against the bound, increment, poll for release, stop at the deadline, decrement in a `finally`\n\nblock so accounting can't leak. A full queue produces `queue_dropped`\n\nand a `429`\n\n; a timed-out wait produces `queue_timed_out`\n\nand a `429`\n\n. The distributed version (Section 7) replaces this with a Redis sorted set keyed by deadline, but the lifecycle events — `queue_queued`\n\n, `queue_released`\n\n, `queue_timed_out`\n\n, `queue_dropped`\n\n— stay identical either way, so the ledger doesn't care which implementation handled a given request.\n\n**Fallback** is driven by a separate `SustainedPressureTracker`\n\n. With the defaults above, pressure has to sit at or above 85% for three consecutive polling intervals before fallback activates — a single spike doesn't trigger it, because routing an entire tier to a smaller model is a heavier intervention than a temporary queue wait, and it should require more evidence. When it does trigger and a fallback model is configured, the decision comes back as:\n\n```\nPolicyDecision(\n    Decision.FALLBACK,\n    \"sustained_pressure_routed_to_quantized_model\",\n    route_model=fallback_model,\n)\n```\n\nand the proxy rewrites the request body's `model`\n\nfield before forwarding — the one other field, besides `max_tokens`\n\n, that Sluice will ever touch. Recovery requires pressure back below 75% for three intervals, mirroring the asymmetric up/down logic from Section 5. The fallback target itself is plain config:\n\n```\nfallbacks:\n  - name: quantized\n    url: http://localhost:8000\n    model: quantized-model\n```\n\n**Reject** returns a `429`\n\nand never contacts vLLM at all:\n\n```\n{\n  \"error\": {\n    \"message\": \"Request rejected by Sluice due to capacity signals\",\n    \"type\": \"sluice_capacity\",\n    \"code\": \"capacity_rejected\",\n    \"reason\": \"standard_rejected_at_signal_threshold\",\n    \"tier\": \"Standard\",\n    \"pressure\": 95,\n    \"queue_depth\": 0,\n    \"sla_violation_rate\": 0\n  }\n}\n```\n\nWorth distinguishing: a *policy* rejection (capacity-driven, `sluice_capacity`\n\n) is a different thing from a *backend* failure. If no healthy backend is available at all, the response is a `503`\n\nwith `type: sluice_backend, code: backend_unavailable`\n\n— the caller shouldn't have to guess whether they were shaped by policy or failed by infrastructure.\n\nThe original queue was process-local, full stop. `_size`\n\nlived inside one process's memory, membership and ordering were represented by waiting coroutines in that same process, and FIFO ordering — along with any future in-flight counter — simply didn't cross replica boundaries. That's documented plainly in `DISTRIBUTED_STATE_DESIGN.md`\n\nas the actual state inventory that needed to move.\n\nThe race this creates with two replicas is the textbook check-then-act problem: replica A reads the current in-flight count and sees room below capacity; replica B reads the same count and sees the same room; both decide independently that they can admit; both increment; the shared capacity is exceeded by however many replicas raced past the check simultaneously.\n\nThere's no production incident log proving this happened — but it's exactly the race that any naive multi-replica admission scheme has, and it's the one the distributed implementation exists to prevent. The proof lives in `tests/test_distributed_queue.py`\n\n, using an asyncio lock to model Redis's script atomicity: `test_atomic_inflight_admission_allows_only_one_replica()`\n\nspins up two coordinators with `max_inflight=1`\n\nand calls `acquire_inflight(\"Standard\")`\n\non both concurrently. The assertion is `sorted(results) == [False, True]`\n\n. A naive read-then-write implementation would be vulnerable to both returning `True`\n\n.\n\nThe coordination primitive is Redis, but specifically **atomic Lua scripts**, not a distributed lock:\n\n`ZCARD`\n\nagainst the queue bound, adds the request to a sorted set scored by deadline.The reasoning for Lua over a lock: the critical section here is short and entirely key-local, and Redis executes a script atomically without introducing lock ownership, expiry semantics, or the possibility of deadlock. A lock is the right tool when you need to hold a critical section across multiple round trips; this doesn't need that.\n\nThe distributed test suite covers more than the single admission race — shared queue ordering across replicas, queue bounds enforced across replicas, timeout cleanup, and fail-open behavior when Redis itself is unreachable. That last one is a deliberate, documented tradeoff: **Redis outages fail open, which preserves request availability at the cost of strictly enforced distributed bounds.** It's not a hidden gap — it's a choice, and it's the right one for a system whose job is protecting SLA compliance, not the one whose job is refusing to ever slightly over-admit.\n\nBefore the numbers, one correction to make against the project's own README: the narrative describes four separate workload profiles. What `benchmarks/load_test.py`\n\nactually runs is three modes — `direct`\n\n, `phase1`\n\n, `phase2`\n\n— against a single generated 600-request pressure timeline: 20% pressure at the start, a climb to 75%, a spike to 98%, then back down to 20%. It's a synthetic, deterministic simulation, and calling it four workload profiles overstates what the harness currently does. Worth fixing in the docs; worth being honest about here.\n\n**Direct mode** (no Sluice at all) is the baseline:\n\n| Tier | P50 | P95 | P99 | SLA violations |\n|---|---|---|---|---|\n| Guaranteed | 1542.9 ms | 2134.3 ms | 2134.3 ms | 30.0% |\n| Standard | 1200.0 ms | 1660.0 ms | 1660.0 ms | 60.0% |\n| Best-Effort | 771.4 ms | 1067.1 ms | 1067.1 ms | 60.0% |\n\nNo proxy overhead, and no protection — every tier bears the same overload equally.\n\n**Phase 1** (pressure-only policy) already shows the mechanism working:\n\n| Tier | P50 | SLA violations | Reject | Degrade | Overhead |\n|---|---|---|---|---|---|\n| Guaranteed | 1237.9 ms | 0.0% | 0% | 30.0% | 0.0022 ms |\n| Standard | 700.0 ms | 0.0% | 30.0% | 30.0% | 0.0022 ms |\n| Best-Effort | 450.0 ms | 0.0% | 60.0% | 0% | 0.0022 ms |\n\n**Phase 2** (full multi-signal pipeline) is the headline:\n\n| Tier | P50 | P99 | SLA violations | Reject | Fallback | Queue wait | Overhead |\n|---|---|---|---|---|---|---|---|\n| Guaranteed | 1024.5 ms | 2134.3 ms | 2.0% |\n0% | 30.0% | 0 ms | 0.0058 ms |\n| Standard | 796.8 ms | 1401.2 ms | 32.14% | 2.0% | 30.0% | 40 ms | 0.0059 ms |\n| Best-Effort | 450.0 ms | 771.4 ms | 41.61% | 31.5% | 30.0% | 0 ms | 0.0056 ms |\n\nThe headline number: **Guaranteed-tier SLA violations fell from 30.0% direct to 2.0% under Phase 2**, at a median proxy overhead of roughly 0.006 ms — negligible next to inference latency measured in seconds.\n\nThe less flattering, equally important number: Standard and Best-Effort are *not* fully protected in this profile — 32% and 42% SLA violations respectively, even under active policy. That's not a hidden failure; it's what the tradeoff actually looks like. Sluice reduces overload impact for lower tiers by design, it doesn't eliminate it, because eliminating it would mean taking protection away from the tier that's supposed to have it. The Phase 2 run also recorded `recovery_time_ms: 18000`\n\n, `state_transition_count: 9`\n\n, and `action_flapping_count: 0`\n\n— the last one being the direct evidence that the hysteresis work from Section 5 held up under this load pattern, at least in simulation.\n\nIt matters to be precise about what this evaluation actually is: `evaluations/phase4_2`\n\nis a deterministic, deployment-shaped *simulator* — it models shared tier admission, replica failure, load-balancer detection delay, spot backend failure and health detection, tier routing, and tenant scale. It is not a live Kubernetes kill test against a running cluster. The results below describe how the current implementation *should* behave under these conditions, modeled faithfully against the real policy code — not a production incident record.\n\n**Replica failure.** With simulated capacities of 8/12/6 (Guaranteed/Standard/Best-Effort), a failure at 1000ms, detection at 1200ms, and measurement through 2600ms, the maximum observed in-flight counts were 7, 11, and 5 respectively — the configured capacities were never exceeded, even through the failure window.\n\n| Window | Error rate | P95 latency | SLA violations |\n|---|---|---|---|\n| Before | 0.00% | 620 ms | 0.00% |\n| During | 6.25% | 620 ms | 0.00% |\n| After | 0.00% | 620 ms | 0.00% |\n\nThe client-visible errors during the failure window came specifically from requests sent to the failed replica in the 200ms gap before detection — after detection, the surviving replica handled traffic without ever exceeding the shared capacity counters. Modeled recovery time was 1600ms from the start of the failure.\n\n**Spot-pool failure**, with the same routing rules from Section 2 (Guaranteed and Standard to on-demand, Best-Effort preferring spot):\n\n| Window | Error rate | P95 latency |\n|---|---|---|\n| Before | 0.00% | 620 ms |\n| During | 5.71% | 620 ms |\n| After | 0.00% | 620 ms |\n\nGuaranteed traffic was unaffected throughout, by construction of the routing policy. Best-Effort traffic split 36 requests to spot against 20 to on-demand across the run, with eviction becoming active 200ms after the failure and recovery converging 1200ms after detection.\n\n**Tenant scale**, at 10, 50, and 100 simulated tenants cycled across the three tiers:\n\n| Tenants | Requests | Mapping errors | Median overhead | P95 overhead |\n|---|---|---|---|---|\n| 10 | 200 | 0 | 0.0045 ms | 0.0063 ms |\n| 50 | 1000 | 0 | 0.0043 ms | 0.0050 ms |\n| 100 | 2000 | 0 | 0.0053 ms | 0.0083 ms |\n\nZero tenant-to-tier mapping errors and sub-0.01ms P95 policy overhead at 100 tenants. But it's worth being clear about what this does and doesn't test: Sluice's policy is *tier*-scoped, not *tenant*-quota-scoped — the scale test verifies that mapping and per-request overhead hold up as tenant count grows, not that individual tenants within a tier get fair treatment against each other. That's a real distinction, not a technicality.\n\nThe README states these as non-goals: AI-generated policies, Kubernetes operator behavior, infrastructure autoscaling, billing or tenant management, implementing an inference engine, multi-region orchestration, replacing backend schedulers, becoming an agent framework.\n\nWhat's worth adding here is that the git history backs this up as genuine restraint, not retrospective framing. There's no commit history showing an autoscaler, an operator, or a billing subsystem that was attempted and later ripped out — the boundaries were never crossed in the first place. The Kubernetes deployment stayed plain manifests from the start; there's no abandoned Helm chart or half-built operator sitting in history.\n\nThe one place that comes closest to a partial implementation is the `TenantPolicyStage`\n\nno-op from Section 4. It's worth naming honestly: that's not a disguised tenancy or billing system waiting to be finished — it's an incomplete internal separation within the policy pipeline, a structural boundary that was drawn before every piece of logic was moved across it. Different category of gap than the non-goals above, and it's called out again in the next section.\n\nSome of these are already documented; several are rough edges surfaced by going through the code directly rather than the top-level docs.\n\n**The HA proof is simulated, not live.** `evaluations/phase4_2/run.py`\n\ndefines its own `SharedAdmission`\n\nclass — it doesn't connect to the real Redis coordinator or exercise an actual Kubernetes deployment. The report is honest about this in its own text, but it bears repeating here: this should never be presented as a production chaos-engineering result.\n\n**The Kubernetes example doesn't configure real in-flight limits.** The manifest enables Redis and queueing but leaves `SLUICE_TIER_INFLIGHT_CAPACITY`\n\nat its default of `0`\n\n, which the implementation treats as *unlimited* admission. The Section 9 simulator uses explicit capacities of 8/12/6 — those aren't the defaults of the actual deployed manifest. That's a meaningful gap between what's demonstrated and what's deployed by default.\n\n**The Redis queue claim accounting has a real edge case.** `CLAIM_SCRIPT`\n\nincrements the Redis in-flight key whenever a queued request is claimed — but `release_inflight()`\n\nreturns immediately when `max_inflight <= 0`\n\n, meaning a claim can increment a counter that then never gets decremented. Since enforcement only happens when `max_inflight > 0`\n\n, this doesn't cause incorrect rejections today, but it's stale state accumulating quietly, and it needs cleanup before this could be called hardened.\n\n**Graceful shutdown is approximate, not exact.** The shutdown path sets `app.state.draining = True`\n\nand waits on active-request and queue-size counts. But the middleware's active-request count ends when the endpoint returns a `StreamingResponse`\n\n— not when the stream body actually finishes sending, which happens later in `stream_body()`\n\n. Shutdown drain accounting and true streaming completion aren't perfectly aligned. Queued requests, separately, aren't explicitly cancelled during shutdown; they ride out their normal timeout or release conditions, bounded by the shutdown grace period.\n\n**Readiness is intentionally basic.** `/readyz`\n\nchecks the primary backend, PostgreSQL, and Redis — it doesn't require every fallback backend and every cluster pool to be healthy. Reasonable for failover behavior, but \"ready\" doesn't mean \"every configured dependency is healthy.\"\n\n**Metrics are minimal by design, but that has a real cost.** The dependency-free Prometheus exporter in `sluice_proxy/telemetry.py`\n\ncovers decision counters, pressure and pressure-state gauges, queue depth, and the latest proxy-overhead latency — but that latency metric is a current-value gauge, not a proper histogram with buckets or quantiles. Metrics also live in process memory, scraped per-replica, with no aggregation layer.\n\n**Admin endpoints remain unauthenticated**, even with API-key enforcement on `/v1/*`\n\n. This is called out in the README, but it's worth restating plainly: it's a real deployment risk if those paths are ever exposed outside a local environment. Related: the Kubernetes API keys themselves currently live in a ConfigMap, not a Secret — a demo-appropriate shortcut, not a production one.\n\n**The ledger schema evolves via startup DDL**, not a migration system — `DecisionLedger.connect()`\n\nruns `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`\n\nstatements at connection time. Fine for a demo; not a substitute for a real migration lifecycle.\n\n**Unknown startup health is treated as available.** `BackendHealthRegistry`\n\ninitializes every target to `None`\n\n, and routing treats health as available unless it's been explicitly marked `False`\n\n. That preserves fail-open behavior at startup, but it means a backend that hasn't been checked yet is briefly treated as usable regardless of whether it actually is.\n\n**And the benchmark narrative overstates its own workload matrix** — four profiles claimed, one deterministic timeline across three modes actually implemented, as covered in Section 8. Small, but worth fixing, and worth naming here rather than letting the discrepancy stand unaddressed.\n\nThe hardest problem here was never the first policy threshold — that part was genuinely simple. The difficulty was keeping behavior coherent as the system accumulated more signals, more actions, more backend pools, and eventually multiple replicas, all interacting: telemetry freshness, per-tier thresholds, hysteresis, fallback state, queue state, backend health, cluster routing, Redis admission, ledger durability, and streaming request cleanup all had to stay consistent with each other at once.\n\nThe single most consequential decision was moving admission state out of process memory and into Redis-backed coordination — preserving FIFO queue behavior and admission bounds across replicas without turning Redis into a hard availability dependency. The tradeoff landed explicitly on the side of availability: Redis outages fail open, which means request availability is preserved at the cost of strict distributed bounds during an outage. That's documented, and it's tested, not just asserted.\n\nThe second was preserving backward compatibility while the policy model grew — Phase 2's implementation still falls back to the original pressure-only decision path when no other signals are present, which is precisely what made a fair Phase 1 vs. Phase 2 benchmark comparison possible in the first place, rather than comparing two systems that had quietly become incomparable.\n\nIf this were being started over, the changes worth making are concrete, not aspirational: split `sluice_proxy/app.py`\n\n— at 675 lines it currently owns application assembly, lifecycle, middleware, routing, policy integration, queue integration, admin endpoints, health, and metrics all at once. Make `TenantPolicyStage`\n\na real stage instead of a no-op. Configure tier in-flight capacities from the start rather than shipping the reference deployment at unlimited. Build the benchmark harness around one canonical event model that can run both deterministic simulation and live HTTP/Kubernetes tests, instead of the two diverging as they have. Add integration tests against real Redis and real PostgreSQL, not just their local doubles. Replace the latest-value latency gauge with real histograms. Move API keys into Secrets and protect admin routes independently from inference auth. And close the Redis queue/in-flight accounting edge case before calling the distributed admission path hardened.\n\nThere's no evidence in the git history of major dead ends or discarded architectures — the evolution reads as almost entirely additive: a single pressure-threshold function, extended to multi-signal policy, refactored into explicit pipeline stages, extended again with fallback, queueing, and cluster routing, backed by a reliability evaluation harness, made distributed with Redis, wrapped in operational deployment tooling, and finally stress-tested against simulated failure and scale.\n\nThat progression is what makes the most honest description of what this project actually is:\n\nSluice is a deliberately narrow, explainable capacity-governance control plane whose strongest contribution is the integration of tier policy, live inference telemetry, request shaping, fallback routing, distributed admission, and durable decision evidence — not a new inference scheduler, not an autoscaler, and not a hardened production platform. What the benchmarks and chaos evaluation demonstrate is that the mechanism works as designed and holds up under simulated contention and failure. What they don't demonstrate — and what the limitations section says plainly — is that it's been proven against real production traffic yet. That's the next honest claim to go earn, not the one to imply already having.", "url": "https://wpnews.pro/news/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference", "canonical_source": "https://dev.to/vampiriccyborg/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-13ja", "published_at": "2026-08-14 20:02:48+00:00", "updated_at": "2026-08-14 20:36:30.563369+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-products", "developer-tools"], "entities": ["Sluice", "vLLM", "GitHub", "VampiricCyborg"], "alternates": {"html": "https://wpnews.pro/news/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference", "markdown": "https://wpnews.pro/news/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference.md", "text": "https://wpnews.pro/news/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference.txt", "jsonld": "https://wpnews.pro/news/building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference.jsonld"}}