A developer argues that the industry's focus on LLM gateway latency benchmarks is misguided for agent systems. Production agents require session persistence, cost attribution, model routing, fallback …
Nexus Labs enabled speculative decoding in vLLM for a fine-tuned 8B model, achieving a 1.9x throughput gain, but discovered that greedy decoding with a draft model is not bit-identical to greedy decod…
A developer argues that coding agents like Claude Code and Codex require two distinct infrastructure layers: a control plane for agent lifecycle, permissions, and scheduling, and a fast data plane for…
Bifrost, an open-source AI gateway written in Go, can handle thousands of concurrent LLM requests on Kubernetes with only 11 microseconds of overhead per request at 5,000 requests per second. The gate…
A new research report analyzing 38 open-source and managed LLM proxy projects finds the market rapidly maturing from simple protocol translators into comprehensive governance platforms. LiteLLM leads …
Nexus Labs replaced 60% of its 11,247-line Python middleware for per-tenant LLM cost attribution, rate limiting, and provider failover with Bifrost's virtual keys and governance features. The migratio…
Photoroom reduced its vision-language model inference costs by approximately 62% within three weeks by deploying Bifrost as a semantic caching layer in front of the VLM step of its product-photo diffu…
Photoroom's SDXL LoRA fine-tuning for a product photography model silently corrupted its adapter weights over six days due to a bf16 gradient accumulation issue. The custom training loop, forked from …
A developer has compiled a list of 10 open source tools that can give developers a competitive advantage, including the Godot Engine for game development, Turborepo for monorepo builds, and Nest.js fo…
A 30-day production test at Nexus Labs compared three AI gateways—Bifrost, LiteLLM, and Portkey—measuring failover latency and overhead. Bifrost added 11ms p99 overhead with automatic provider fallbac…
Amazon SageMaker AI now supports OpenAI-compatible API for real-time inference endpoints, allowing users to invoke models by changing only the endpoint URL without custom clients or code rewrites. The…
Bifrost is a routing layer for production LLM applications that implements automatic fallback mechanisms to eliminate single points of failure. Instead of embedding manual fallback logic into applicat…