📦 Project: https://github.com/VampiricCyborg/sluice
A self-hosted vLLM deployment runs on a GPU pool of fixed size. That's the fact that changes everything about how you have to think about load.
Once that pool's KV-cache capacity comes under pressure, requests don't naturally understand business tiers. They don't know that one tenant is an enterprise customer with a contract, another is an internal team, and a third is a batch job that can wait. Without something external saying otherwise, the system's behavior under contention is driven by arrival order and backend scheduling — and every tenant, regardless of what they were promised, experiences the same degraded latency, queueing, or failure.
There's no built-in mechanism that says:
Preserve Guaranteed traffic, reduce the work done by Standard traffic, and shed Best-Effort traffic first.
Sluice exists to make that decision — before the request ever reaches vLLM.
Concretely, it evaluates three live signals per decision cycle:
Tenants are mapped to tiers in config/tenants.yaml
:
tenants:
support-bot: Guaranteed
coding-assistant: Standard
batch-analytics: Best-Effort
and each tier has an explicit SLA target, hard-coded in sluice_proxy/app.py
:
sla_targets = {
"Guaranteed": 2000,
"Standard": 1000,
"Best-Effort": 500,
}
The important thing to understand about Sluice's scope from the start: it is not trying to make vLLM's own execution more efficient. It governs which requests reach the inference engine, how much work they're allowed to request, and which backend pool they land on. Everything downstream of that decision is still vLLM's job.
It's worth being precise about why this problem doesn't already have a home in the stack, because Sluice's design only makes sense in contrast to the layers around it.
Kubernetes can place workloads, restart failed replicas, and add capacity if an autoscaler is configured. None of that helps when GPU capacity is fixed or slow to provision — which, for self-hosted inference, is closer to the default case than the exception. Kubernetes has no concept of a "tenant" or a "tier" at the request level; it schedules pods, not chat completions.
vLLM's own scheduler is the right layer for token-level execution, batching, and KV-cache management inside a worker — and it's better at that than anything sitting in front of it could be. Sluice deliberately stays outside that loop. It has no interest in improving token scheduling; it adds tenant identity, tier semantics, and an auditable reason for each request-level action, then gets out of the way.
A generic API gateway (Envoy, Kong) gives you authentication, routing, retries, and rate limiting — genuinely useful, and Sluice doesn't try to replace any of it. But rate limits alone can't express "degrade this tier's max_tokens
before rejecting that tier's requests, based on live GPU pressure." That's a specialized policy signal and action set that a generic gateway has no vocabulary for.
The actual boundary in code is narrow. The original Phase 1 implementation of sluice_proxy/app.py
did five things: identify the tenant from request headers, look up their tier, read cache pressure, apply a policy decision, and optionally rewrite max_tokens
before forwarding. It never touched token scheduling, batching, or KV-cache allocation — those stayed vLLM's problem, from the very first commit.
The is_shapeable()
function makes this boundary explicit today. Only POST
, PUT
, PATCH
requests to /v1/chat/completions
or /v1/completions
are eligible for shaping at all. Everything else passes through untouched. And even for shapeable requests, Sluice only ever changes two fields: max_tokens
(via apply_max_tokens()
) and model
(via apply_model()
, for fallback routing). That's the entire surface area of what Sluice is willing to touch in a request body.
The same discipline shows up in cluster routing. ClusterRoutingStage
in sluice_proxy/policy.py
carries this as an explicit design intent in its own docstring:
QoS route between pools, never utilization/load balance.
Routing is a service-policy decision — Guaranteed always gets on-demand, Standard gets on-demand, Best-Effort prefers spot unless spot pressure or a health failure forces eviction — not a round-robin balancing act. And the Kubernetes deployment in k8s/sluice-ops.yaml
is intentionally just a Deployment, a Service, a ConfigMap, and probes. No operator, no autoscaler, no scheduler extension.
Before getting into design, it's worth stating plainly what Sluice owns, because everything downstream follows from this boundary. The clearest way to see it is through its actual interfaces.
The request path is a single catch-all:
/v1/{path:path}
For every request that hits it, Sluice does exactly seven things: resolve tenant and tier, read current telemetry, evaluate policy, apply shaping/queueing/fallback/rejection/routing, forward to the selected backend, record the result in PostgreSQL, and emit a structured log.
Operational visibility is a separate, small surface: /healthz
, /livez
, /readyz
, /metrics
, /status
, /decisions/recent
. And for local development, a narrow set of simulation endpoints — PUT
/DELETE
on /admin/pressure
, /admin/queue-depth
, /admin/sla-violation-rate
— let you force a signal value without waiting on real telemetry. These are intentionally minimal development controls, not a control-plane API, and they stay unauthenticated even when API-key auth is otherwise enabled on /v1/*
— a decision that's convenient for local testing and a real risk if ever exposed (more on that in Section 11).
The external dependencies are equally explicit: Prometheus for live inference telemetry, PostgreSQL for durable decisions and SLA history, Redis for distributed queue/admission state, and any number of vLLM-compatible HTTP backends. Sluice doesn't implement any of these things — it coordinates between them.
REQUEST
│
▼
┌──────────────┐
│ Sluice │
└──────┬───────┘
│
┌────────────▼────────────┐
│ Pressure Evaluation │
│ KV Cache │ Queue │ SLA │
└────────────┬────────────┘
│
▼
┌───────────────┐
│ Tenant Policy │
└───────┬───────┘
│
▼
┌───────────────┐
│ Action Select │
│ Pass/Degrade │
│ Queue/Fallback│
│ Reject │
└───────┬───────┘
│
▼
┌───────────────┐
│ Cluster Route │
└───────┬───────┘
│
┌────────┴────────┐
▼ ▼
On-Demand GPU Spot GPU
Sluice's decision logic didn't start as a pipeline. It started as one function.
In the first commit of the policy engine (c101afb
), CapacityPolicy
had a single decide()
method with nested tier conditionals:
if tier == "Best-Effort":
if pressure >= 60:
reject
elif tier == "Standard":
if pressure >= 90:
reject
if pressure >= 70:
degrade
elif tier == "Guaranteed":
if pressure >= 100:
reject
if pressure >= 95:
degrade
That was fine when there was exactly one signal. It stopped being fine the moment a second one showed up. Commit 91c00fa
extended decide()
to accept pressure
, queue_depth
, and sla_violation_rate
together, and the shape it took was still recognizably one function — checking whether any signal crossed a reject threshold, then whether any crossed a degrade threshold. It worked, but the conditional surface was growing in a direction that wasn't going to scale to fallback routing, queueing, or multi-cluster decisions without becoming unreadable.
Commit ca80787
is where the real refactor happened: four explicit stages, each owning one concern.
PressureEvaluationStage.evaluate(tier, signals) -> PressureEvaluation
TenantPolicyStage.apply(evaluation, *, shapeable, fallback_active, fallback_model) -> PressureEvaluation
ActionSelectionStage.select(evaluation, *, shapeable, fallback_active, fallback_model) -> PolicyDecision
ClusterRoutingStage.route(tier, decision, clusters, health) -> PolicyDecision
orchestrated by a single PolicyPipeline.run()
. Pressure Evaluation turns raw signals into a named state (Normal, Elevated, Critical). Tenant Policy is the extension point for tier-specific semantics. Action Selection picks the actual intervention. Cluster Routing decides which pool handles it.
I want to be honest about one detail here rather than present this as a clean four-stage design from the start: TenantPolicyStage.apply()
is currently a no-op — it returns the evaluation unchanged. The refactor established the stage boundary, but not every piece of tier-specific behavior actually lives there yet. Some of it is still inside CapacityPolicy._select_action()
and _thresholds_for()
. That's a real, current gap between the architecture's intent and its full implementation — not a mistake, just unfinished separation. The pipeline shape was worth building before every piece of logic was moved into it, because it made everything that came after (fallback, queueing, multi-cluster routing) addable without another rewrite.
Three named states drive everything downstream:
PressureState.NORMAL
PressureState.ELEVATED
PressureState.CRITICAL
The combination rule across three signals is deliberately conservative — a max-severity OR, not an average: any available signal crossing a reject threshold produces CRITICAL
; failing that, any signal crossing a degrade threshold produces ELEVATED
; otherwise NORMAL
. If every signal is unavailable, the policy fails open rather than guessing.
The stabilization defaults, set in CapacityPolicy.__init__()
:
SLUICE_STATE_CONSECUTIVE_SAMPLES = 2
SLUICE_STATE_COOLDOWN_SECONDS = 2
SLUICE_STATE_HYSTERESIS = 5
I'll say up front that these are operational defaults, not values fitted to production traces — there's no evidence in the repository of them being derived from real traffic, and that's worth being upfront about rather than implying more rigor than exists.
What they buy you is protection against a specific, well-understood failure mode: a signal hovering right at a threshold causing the system to flip actions on every poll. PressureStabilityModel.observe()
enforces three conditions before a transition is allowed to happen: the new state must persist for two consecutive samples, it must respect the cooldown window, and moving down a severity level requires crossing a lower recovery threshold than the one that triggered the escalation — not just dipping back under the original line.
tests/test_fallback_queue.py
shows this concretely. Starting from Normal:
75 at t=1 -> no transition (first sample, waiting for confirmation)
75 at t=2 -> Elevated (confirmed)
69 at t=3 -> remains Elevated (below the up-threshold of 70, but still above recovery threshold of 65)
64 at t=4 -> no transition (waiting for confirmation)
64 at t=5 -> Normal (confirmed)
The interesting line is 69 at t=3
. Without an asymmetric recovery threshold, a value that dips just under the escalation threshold would immediately flip the state back — and if pressure is oscillating around 70, that's a state flip on every poll. With the wider recovery band, 69 simply isn't low enough to count as recovery yet.
I should be precise about what evidence backs this: there's no log excerpt in the repository from a real pre-hysteresis deployment flapping in production — the evidence is test-driven, not observed. The synthetic benchmark's Phase 2 run does record action_flapping_count: 0
, but that demonstrates the stabilized simulator didn't flap under a controlled pressure timeline; it doesn't prove a live, unstabilized system would have.
Fallback gets its own tracker with its own defaults, since "sustained" pressure for a fallback decision is a meaningfully different bar than a degrade decision:
SLUICE_FALLBACK_PRESSURE_AT = 85
SLUICE_FALLBACK_RECOVER_BELOW = 75
SLUICE_FALLBACK_SUSTAINED_INTERVALS = 3
SLUICE_FALLBACK_COOLDOWN_SECONDS = 2
Sluice has five possible outcomes for a request, escalating roughly with tier and pressure.
Pass is the default — signals under threshold, a non-shapeable endpoint, or telemetry unavailable and failing open. The reason is always recorded explicitly (below_tier_pressure_threshold
, cache_pressure_unavailable_fail_open
, endpoint_not_shapeable
, signals_unavailable_fail_open
), because a "pass" with no explanation is just as much a policy decision as a rejection and deserves the same audit trail.
Degrade only applies to shapeable completion endpoints, and only lowers max_tokens
— never touches anything else in the payload:
STANDARD_MAX_TOKENS = 256
GUARANTEED_MAX_TOKENS = 512
apply_max_tokens()
is conservative by design: if max_tokens
is missing, set it to the cap; if it's not an integer, set it to the cap; if it exceeds the cap, replace it; if it's already under the cap, leave it alone. Best-Effort has no degrade step at all — it moves straight toward rejection, since there's no meaningful "smaller" version of a request that's already lowest priority.
Queue is Standard-tier only, and only fires in the Elevated state — Guaranteed never queues, and Best-Effort doesn't get access to the Standard queue. The bounds are tight on purpose:
SLUICE_QUEUE_MAX_SIZE = 100
SLUICE_QUEUE_MAX_WAIT_SECONDS = 2
SLUICE_QUEUE_POLL_SECONDS = 0.05
The local, single-replica implementation is a BoundedRequestQueue
: acquire a lock, check size against the bound, increment, poll for release, stop at the deadline, decrement in a finally
block so accounting can't leak. A full queue produces queue_dropped
and a 429
; a timed-out wait produces queue_timed_out
and a 429
. The distributed version (Section 7) replaces this with a Redis sorted set keyed by deadline, but the lifecycle events — queue_queued
, queue_released
, queue_timed_out
, queue_dropped
— stay identical either way, so the ledger doesn't care which implementation handled a given request.
Fallback is driven by a separate SustainedPressureTracker
. With the defaults above, pressure has to sit at or above 85% for three consecutive polling intervals before fallback activates — a single spike doesn't trigger it, because routing an entire tier to a smaller model is a heavier intervention than a temporary queue wait, and it should require more evidence. When it does trigger and a fallback model is configured, the decision comes back as:
PolicyDecision(
Decision.FALLBACK,
"sustained_pressure_routed_to_quantized_model",
route_model=fallback_model,
)
and the proxy rewrites the request body's model
field before forwarding — the one other field, besides max_tokens
, that Sluice will ever touch. Recovery requires pressure back below 75% for three intervals, mirroring the asymmetric up/down logic from Section 5. The fallback target itself is plain config:
fallbacks:
- name: quantized
url: http://localhost:8000
model: quantized-model
Reject returns a 429
and never contacts vLLM at all:
{
"error": {
"message": "Request rejected by Sluice due to capacity signals",
"type": "sluice_capacity",
"code": "capacity_rejected",
"reason": "standard_rejected_at_signal_threshold",
"tier": "Standard",
"pressure": 95,
"queue_depth": 0,
"sla_violation_rate": 0
}
}
Worth distinguishing: a policy rejection (capacity-driven, sluice_capacity
) is a different thing from a backend failure. If no healthy backend is available at all, the response is a 503
with type: sluice_backend, code: backend_unavailable
— the caller shouldn't have to guess whether they were shaped by policy or failed by infrastructure.
The original queue was process-local, full stop. _size
lived inside one process's memory, membership and ordering were represented by waiting coroutines in that same process, and FIFO ordering — along with any future in-flight counter — simply didn't cross replica boundaries. That's documented plainly in DISTRIBUTED_STATE_DESIGN.md
as the actual state inventory that needed to move.
The race this creates with two replicas is the textbook check-then-act problem: replica A reads the current in-flight count and sees room below capacity; replica B reads the same count and sees the same room; both decide independently that they can admit; both increment; the shared capacity is exceeded by however many replicas raced past the check simultaneously.
There's no production incident log proving this happened — but it's exactly the race that any naive multi-replica admission scheme has, and it's the one the distributed implementation exists to prevent. The proof lives in tests/test_distributed_queue.py
, using an asyncio lock to model Redis's script atomicity: test_atomic_inflight_admission_allows_only_one_replica()
spins up two coordinators with max_inflight=1
and calls acquire_inflight("Standard")
on both concurrently. The assertion is sorted(results) == [False, True]
. A naive read-then-write implementation would be vulnerable to both returning True
.
The coordination primitive is Redis, but specifically atomic Lua scripts, not a distributed lock:
ZCARD
against the queue bound, adds the request to a sorted set scored by deadline.The reasoning for Lua over a lock: the critical section here is short and entirely key-local, and Redis executes a script atomically without introducing lock ownership, expiry semantics, or the possibility of deadlock. A lock is the right tool when you need to hold a critical section across multiple round trips; this doesn't need that.
The distributed test suite covers more than the single admission race — shared queue ordering across replicas, queue bounds enforced across replicas, timeout cleanup, and fail-open behavior when Redis itself is unreachable. That last one is a deliberate, documented tradeoff: Redis outages fail open, which preserves request availability at the cost of strictly enforced distributed bounds. It's not a hidden gap — it's a choice, and it's the right one for a system whose job is protecting SLA compliance, not the one whose job is refusing to ever slightly over-admit.
Before the numbers, one correction to make against the project's own README: the narrative describes four separate workload profiles. What benchmarks/load_test.py
actually runs is three modes — direct
, phase1
, phase2
— against a single generated 600-request pressure timeline: 20% pressure at the start, a climb to 75%, a spike to 98%, then back down to 20%. It's a synthetic, deterministic simulation, and calling it four workload profiles overstates what the harness currently does. Worth fixing in the docs; worth being honest about here.
Direct mode (no Sluice at all) is the baseline:
| Tier | P50 | P95 | P99 | SLA violations |
|---|---|---|---|---|
| Guaranteed | 1542.9 ms | 2134.3 ms | 2134.3 ms | 30.0% |
| Standard | 1200.0 ms | 1660.0 ms | 1660.0 ms | 60.0% |
| Best-Effort | 771.4 ms | 1067.1 ms | 1067.1 ms | 60.0% |
No proxy overhead, and no protection — every tier bears the same overload equally.
Phase 1 (pressure-only policy) already shows the mechanism working:
| Tier | P50 | SLA violations | Reject | Degrade | Overhead |
|---|---|---|---|---|---|
| Guaranteed | 1237.9 ms | 0.0% | 0% | 30.0% | 0.0022 ms |
| Standard | 700.0 ms | 0.0% | 30.0% | 30.0% | 0.0022 ms |
| Best-Effort | 450.0 ms | 0.0% | 60.0% | 0% | 0.0022 ms |
Phase 2 (full multi-signal pipeline) is the headline:
| Tier | P50 | P99 | SLA violations | Reject | Fallback | Queue wait | Overhead |
|---|---|---|---|---|---|---|---|
| Guaranteed | 1024.5 ms | 2134.3 ms | 2.0% | ||||
| 0% | 30.0% | 0 ms | 0.0058 ms | ||||
| Standard | 796.8 ms | 1401.2 ms | 32.14% | 2.0% | 30.0% | 40 ms | 0.0059 ms |
| Best-Effort | 450.0 ms | 771.4 ms | 41.61% | 31.5% | 30.0% | 0 ms | 0.0056 ms |
The headline number: Guaranteed-tier SLA violations fell from 30.0% direct to 2.0% under Phase 2, at a median proxy overhead of roughly 0.006 ms — negligible next to inference latency measured in seconds.
The less flattering, equally important number: Standard and Best-Effort are not fully protected in this profile — 32% and 42% SLA violations respectively, even under active policy. That's not a hidden failure; it's what the tradeoff actually looks like. Sluice reduces overload impact for lower tiers by design, it doesn't eliminate it, because eliminating it would mean taking protection away from the tier that's supposed to have it. The Phase 2 run also recorded recovery_time_ms: 18000
, state_transition_count: 9
, and action_flapping_count: 0
— the last one being the direct evidence that the hysteresis work from Section 5 held up under this load pattern, at least in simulation.
It matters to be precise about what this evaluation actually is: evaluations/phase4_2
is a deterministic, deployment-shaped simulator — it models shared tier admission, replica failure, load-balancer detection delay, spot backend failure and health detection, tier routing, and tenant scale. It is not a live Kubernetes kill test against a running cluster. The results below describe how the current implementation should behave under these conditions, modeled faithfully against the real policy code — not a production incident record.
Replica failure. With simulated capacities of 8/12/6 (Guaranteed/Standard/Best-Effort), a failure at 1000ms, detection at 1200ms, and measurement through 2600ms, the maximum observed in-flight counts were 7, 11, and 5 respectively — the configured capacities were never exceeded, even through the failure window.
| Window | Error rate | P95 latency | SLA violations |
|---|---|---|---|
| Before | 0.00% | 620 ms | 0.00% |
| During | 6.25% | 620 ms | 0.00% |
| After | 0.00% | 620 ms | 0.00% |
The client-visible errors during the failure window came specifically from requests sent to the failed replica in the 200ms gap before detection — after detection, the surviving replica handled traffic without ever exceeding the shared capacity counters. Modeled recovery time was 1600ms from the start of the failure.
Spot-pool failure, with the same routing rules from Section 2 (Guaranteed and Standard to on-demand, Best-Effort preferring spot):
| Window | Error rate | P95 latency |
|---|---|---|
| Before | 0.00% | 620 ms |
| During | 5.71% | 620 ms |
| After | 0.00% | 620 ms |
Guaranteed traffic was unaffected throughout, by construction of the routing policy. Best-Effort traffic split 36 requests to spot against 20 to on-demand across the run, with eviction becoming active 200ms after the failure and recovery converging 1200ms after detection.
Tenant scale, at 10, 50, and 100 simulated tenants cycled across the three tiers:
| Tenants | Requests | Mapping errors | Median overhead | P95 overhead |
|---|---|---|---|---|
| 10 | 200 | 0 | 0.0045 ms | 0.0063 ms |
| 50 | 1000 | 0 | 0.0043 ms | 0.0050 ms |
| 100 | 2000 | 0 | 0.0053 ms | 0.0083 ms |
Zero tenant-to-tier mapping errors and sub-0.01ms P95 policy overhead at 100 tenants. But it's worth being clear about what this does and doesn't test: Sluice's policy is tier-scoped, not tenant-quota-scoped — the scale test verifies that mapping and per-request overhead hold up as tenant count grows, not that individual tenants within a tier get fair treatment against each other. That's a real distinction, not a technicality.
The README states these as non-goals: AI-generated policies, Kubernetes operator behavior, infrastructure autoscaling, billing or tenant management, implementing an inference engine, multi-region orchestration, replacing backend schedulers, becoming an agent framework.
What's worth adding here is that the git history backs this up as genuine restraint, not retrospective framing. There's no commit history showing an autoscaler, an operator, or a billing subsystem that was attempted and later ripped out — the boundaries were never crossed in the first place. The Kubernetes deployment stayed plain manifests from the start; there's no abandoned Helm chart or half-built operator sitting in history.
The one place that comes closest to a partial implementation is the TenantPolicyStage
no-op from Section 4. It's worth naming honestly: that's not a disguised tenancy or billing system waiting to be finished — it's an incomplete internal separation within the policy pipeline, a structural boundary that was drawn before every piece of logic was moved across it. Different category of gap than the non-goals above, and it's called out again in the next section.
Some of these are already documented; several are rough edges surfaced by going through the code directly rather than the top-level docs.
The HA proof is simulated, not live. evaluations/phase4_2/run.py
defines its own SharedAdmission
class — it doesn't connect to the real Redis coordinator or exercise an actual Kubernetes deployment. The report is honest about this in its own text, but it bears repeating here: this should never be presented as a production chaos-engineering result.
The Kubernetes example doesn't configure real in-flight limits. The manifest enables Redis and queueing but leaves SLUICE_TIER_INFLIGHT_CAPACITY
at its default of 0
, which the implementation treats as unlimited admission. The Section 9 simulator uses explicit capacities of 8/12/6 — those aren't the defaults of the actual deployed manifest. That's a meaningful gap between what's demonstrated and what's deployed by default.
The Redis queue claim accounting has a real edge case. CLAIM_SCRIPT
increments the Redis in-flight key whenever a queued request is claimed — but release_inflight()
returns immediately when max_inflight <= 0
, meaning a claim can increment a counter that then never gets decremented. Since enforcement only happens when max_inflight > 0
, this doesn't cause incorrect rejections today, but it's stale state accumulating quietly, and it needs cleanup before this could be called hardened.
Graceful shutdown is approximate, not exact. The shutdown path sets app.state.draining = True
and waits on active-request and queue-size counts. But the middleware's active-request count ends when the endpoint returns a StreamingResponse
— not when the stream body actually finishes sending, which happens later in stream_body()
. Shutdown drain accounting and true streaming completion aren't perfectly aligned. Queued requests, separately, aren't explicitly cancelled during shutdown; they ride out their normal timeout or release conditions, bounded by the shutdown grace period.
Readiness is intentionally basic. /readyz
checks the primary backend, PostgreSQL, and Redis — it doesn't require every fallback backend and every cluster pool to be healthy. Reasonable for failover behavior, but "ready" doesn't mean "every configured dependency is healthy."
Metrics are minimal by design, but that has a real cost. The dependency-free Prometheus exporter in sluice_proxy/telemetry.py
covers decision counters, pressure and pressure-state gauges, queue depth, and the latest proxy-overhead latency — but that latency metric is a current-value gauge, not a proper histogram with buckets or quantiles. Metrics also live in process memory, scraped per-replica, with no aggregation layer.
Admin endpoints remain unauthenticated, even with API-key enforcement on /v1/*
. This is called out in the README, but it's worth restating plainly: it's a real deployment risk if those paths are ever exposed outside a local environment. Related: the Kubernetes API keys themselves currently live in a ConfigMap, not a Secret — a demo-appropriate shortcut, not a production one.
The ledger schema evolves via startup DDL, not a migration system — DecisionLedger.connect()
runs ALTER TABLE ... ADD COLUMN IF NOT EXISTS
statements at connection time. Fine for a demo; not a substitute for a real migration lifecycle.
Unknown startup health is treated as available. BackendHealthRegistry
initializes every target to None
, and routing treats health as available unless it's been explicitly marked False
. That preserves fail-open behavior at startup, but it means a backend that hasn't been checked yet is briefly treated as usable regardless of whether it actually is.
And the benchmark narrative overstates its own workload matrix — four profiles claimed, one deterministic timeline across three modes actually implemented, as covered in Section 8. Small, but worth fixing, and worth naming here rather than letting the discrepancy stand unaddressed.
The hardest problem here was never the first policy threshold — that part was genuinely simple. The difficulty was keeping behavior coherent as the system accumulated more signals, more actions, more backend pools, and eventually multiple replicas, all interacting: telemetry freshness, per-tier thresholds, hysteresis, fallback state, queue state, backend health, cluster routing, Redis admission, ledger durability, and streaming request cleanup all had to stay consistent with each other at once.
The single most consequential decision was moving admission state out of process memory and into Redis-backed coordination — preserving FIFO queue behavior and admission bounds across replicas without turning Redis into a hard availability dependency. The tradeoff landed explicitly on the side of availability: Redis outages fail open, which means request availability is preserved at the cost of strict distributed bounds during an outage. That's documented, and it's tested, not just asserted.
The second was preserving backward compatibility while the policy model grew — Phase 2's implementation still falls back to the original pressure-only decision path when no other signals are present, which is precisely what made a fair Phase 1 vs. Phase 2 benchmark comparison possible in the first place, rather than comparing two systems that had quietly become incomparable.
If this were being started over, the changes worth making are concrete, not aspirational: split sluice_proxy/app.py
— at 675 lines it currently owns application assembly, lifecycle, middleware, routing, policy integration, queue integration, admin endpoints, health, and metrics all at once. Make TenantPolicyStage
a real stage instead of a no-op. Configure tier in-flight capacities from the start rather than shipping the reference deployment at unlimited. Build the benchmark harness around one canonical event model that can run both deterministic simulation and live HTTP/Kubernetes tests, instead of the two diverging as they have. Add integration tests against real Redis and real PostgreSQL, not just their local doubles. Replace the latest-value latency gauge with real histograms. Move API keys into Secrets and protect admin routes independently from inference auth. And close the Redis queue/in-flight accounting edge case before calling the distributed admission path hardened.
There's no evidence in the git history of major dead ends or discarded architectures — the evolution reads as almost entirely additive: a single pressure-threshold function, extended to multi-signal policy, refactored into explicit pipeline stages, extended again with fallback, queueing, and cluster routing, backed by a reliability evaluation harness, made distributed with Redis, wrapped in operational deployment tooling, and finally stress-tested against simulated failure and scale.
That progression is what makes the most honest description of what this project actually is:
Sluice is a deliberately narrow, explainable capacity-governance control plane whose strongest contribution is the integration of tier policy, live inference telemetry, request shaping, fallback routing, distributed admission, and durable decision evidence — not a new inference scheduler, not an autoscaler, and not a hardened production platform. What the benchmarks and chaos evaluation demonstrate is that the mechanism works as designed and holds up under simulated contention and failure. What they don't demonstrate — and what the limitations section says plainly — is that it's been proven against real production traffic yet. That's the next honest claim to go earn, not the one to imply already having.