{"slug": "infercrane-deploy-and-safely-evolve-self-hosted-ai-inference", "title": "InferCrane – Deploy and safely evolve self-hosted AI inference", "summary": "InferCrane, an open-source inference operations platform, released v1.0.0-rc.1, a public beta CLI and SDK that deploys, operates, and safely evolves self-hosted AI inference behind a stable OpenAI-compatible endpoint. The platform adds evidence-gated release lifecycle management—deterministic promotion, rejection, and rollback with persisted records—to raw vLLM, SGLang, custom OCI, and existing endpoints across AWS, GCP, Kubernetes, and RunPod, while preserving the active revision when evidence is missing.", "body_md": "**Give InferCrane a model. Get a production endpoint.**\n\nOpen-source infrastructure for deploying, operating, optimizing, and safely evolving\n\nopen-weight and custom-model inference behind one stable OpenAI-compatible endpoint.\n\n[Run the five-minute local proof](#install)\n· [Read the quickstart](https://docs.infercrane.com/quickstart)\n\n[\n](/infercrane/infercrane/blob/main/docs/images/product/github-product-demo.gif)\n\nPlan before mutation · survive disconnects · explain latency · guard every release\n\nInstall the `v1.0.0-rc.1`\n\npublic beta CLI with Homebrew or use the matching SDK prerelease:\n\n```\nbrew install infercrane/tap/infercrane\npython -m pip install 'infercrane==1.0.0rc1'\nnpm install '@infercrane/sdk@1.0.0-rc.1'\n```\n\nRelease archives and Terraform provider binaries are available from the\n[ v1.0.0-rc.1 prerelease](https://github.com/infercrane/infercrane/releases/tag/v1.0.0-rc.1).\nTo run the complete GPU-free product proof without creating cloud resources:\n\n```\ngit clone https://github.com/infercrane/infercrane.git\ncd infercrane\nmake demo\n```\n\nThe proof connects an OpenAI-compatible worker, sends and inspects a request, creates an isolated candidate, records a deterministic Release Guard rejection, verifies that production traffic did not move, and removes its disposable stack.\n\nInferCrane is the open-source inference operations platform for self-hosted models. Your application keeps one model identity while InferCrane changes the model artifact, runtime, accelerator, provider, replica count, and active revision underneath it.\n\n**Deploy or adopt:** operate vLLM, SGLang, custom OCI, and compatible existing endpoints across AWS, GCP, Kubernetes, and RunPod.**Keep one endpoint:** route revisions and providers behind a stable OpenAI-compatible contract.**Prove every change:** persist request, benchmark, quality, cost, and rollout evidence before promotion—and preserve the active revision when evidence is missing.\n\n**Raw vLLM and Kubernetes:** the serving engine and manifests do not by themselves provide an evidence-gated release lifecycle. InferCrane adds deterministic promotion, rejection, rollback, and a persisted record of what changed.**LiteLLM:** it is an excellent routing layer. InferCrane additionally owns deployment lifecycle, evidence-gated promotion, and rollback; it can also connect to an existing LiteLLM endpoint without taking infrastructure ownership.**A managed inference platform:** it is often the right choice when a team wants the provider to operate its infrastructure. InferCrane is for teams that want the control plane, capacity, and billing boundary to remain in infrastructure they own.**Scripts and CI:** scripts can deploy a revision. InferCrane standardizes the durable reject/promote/rollback decision and the evidence attached to it.\n\nSee the [detailed comparison](/infercrane/infercrane/blob/main/docs/compare.mdx), including the boundaries InferCrane does not own.\n\nThe local proof needs no GPU or cloud account. Real-provider support remains exact-tuple qualified; InferCrane reports missing model/runtime/hardware evidence as unknown instead of turning it into a compatibility claim.\n\n**See the system boundary**\n\n```\nApplications and agents\n          │\n          ▼\nStable OpenAI-compatible endpoint\n          │\n          ▼\nInferCrane: route · observe · optimize · release · recover\n          │\n          ├── deploy new inference\n          ├── adopt an existing workload\n          └── govern a model API or gateway\n          │\n          ▼\nvLLM · SGLang · custom OCI · experimental Dynamo\nAWS · GCP · Kubernetes · RunPod · existing infrastructure\n```\n\n| Goal | InferCrane workflow |\n|---|---|\n| Put a model into production | Initialize a workload, review the serving plan, deploy, then call its stable endpoint. |\n| Adopt existing inference | Connect vLLM, SGLang, LiteLLM, or another compatible endpoint without transferring lifecycle ownership. |\n| Ship a safer revision | Benchmark and replay an isolated candidate, attach quality evidence, then let Release Guard promote or reject it. |\n| Understand production failures | Trace queue wait, attempts, runtime, revision, latency, saturation, and durable operations without storing prompt content. |\n| Optimize performance and cost | Propose serving configurations, measure comparable candidates on real hardware, and persist only qualified evidence. |\n| Survive infrastructure delays | Submit idempotent durable operations that continue after the CLI or control plane process disconnects. |\n\nLocal fixtures prove application and lifecycle behavior. They do not prove GPU performance, cloud capacity, runtime compatibility, or model quality. InferCrane keeps those evidence boundaries explicit.\n\nStart with a curated recipe:\n\n```\ninfercrane workload init ./support --recipe qwen3-8b\ncd support\ninfercrane workload plan\ninfercrane workload deploy --wait\n```\n\nOr bring another compatible immutable model identity:\n\n```\ninfercrane workload init ./agent-model --model mistralai/Mistral-7B-Instruct-v0.3\ncd agent-model\ninfercrane workload plan\ninfercrane workload deploy --wait\n```\n\nRecipes are reproducible configuration starting points, not benchmark claims or an allowlist. Evidence remains bound to the exact model commit, runtime, accelerator, provider, cache state, and workload.\n\nAlready operating a workload? Connect it first:\n\n```\ninfercrane connect https://vllm.internal/v1 --as support-production\ninfercrane doctor support-production\ninfercrane observe support-production\n```\n\nInferCrane can observe an existing endpoint before it manages traffic or infrastructure. Provider credentials and request content do not enter the browser console.\n\nInferCrane separates a modeled proposal from measured and qualified evidence:\n\n```\ninfercrane optimize propose llama-3.1-8b-instruct \\\n  --provider aws \\\n  --region eu-central-1 \\\n  --gpu L40S \\\n  --objective interactive \\\n  --write-dir .infercrane/candidates\nmodel + hardware + workload + SLO + cost target\n                       │\n                       ▼\n              candidate serving plans\n                       │\n                       ▼\n          AIPerf + replay + quality evidence\n                       │\n                       ▼\n            performance · errors · cost\n                       │\n                 ┌─────┴─────┐\n                 ▼           ▼\n              promote      reject\n```\n\nInferCrane composes replaceable execution technology instead of rebuilding it. vLLM, SGLang, and custom OCI are current execution paths. Dynamo is experimental. TensorRT-LLM, LMCache, NIXL, and external optimizers remain capability boundaries until their exact adapters and hardware tuples are qualified. InferCrane owns serving-plan identity, durable operations, comparable evidence, routing policy, promotion, and rollback.\n\n**Stable endpoint identity:** applications do not change when the serving plan changes.**Bounded overload:** admission limits, explicit`429`\n\nand`Retry-After`\n\n, one end-to-end deadline, and bounded retries prevent unlimited queue growth.**Request-path isolation:** gateways route from immutable in-memory snapshots and never query PostgreSQL on the inference request path.**Durable operations:** deployment, scaling, deletion, and release work is idempotent, restart-safe, cancellable, and inspectable.**Release evidence:** benchmark, replay, quality, reliability, and sourced cost evidence can block a candidate before traffic moves.**Content-free operations:** request evidence records operational metadata without persisting prompts or model outputs.**Explicit ownership:** existing runtimes, gateways, training systems, sandboxes, and clouds stay replaceable behind versioned contracts.\n\nRead the [architecture](https://docs.infercrane.com/architecture/system),\n[system invariants](/infercrane/infercrane/blob/main/docs/architecture/invariants.md), and\n[data flows](/infercrane/infercrane/blob/main/docs/architecture/data-flows.md) for the complete design.\n\n| Interface | Status and purpose |\n|---|---|\n| CLI and control API | Primary deployment, operation, evidence, and administration interfaces. |\n| OpenAI-compatible gateway | Capability-gated Chat, Completions, Embeddings, Responses, and online batch paths. The pinned vLLM profile currently qualifies Chat plus model-compatible Completions and Embeddings; unsupported capabilities fail before upstream transmission. |\n| Python and TypeScript SDKs | Public beta packages: `infercrane==1.0.0rc1` and `@infercrane/sdk@1.0.0-rc.1` . Generated from the checked OpenAPI contract. |\n| Terraform provider | Logical deployment lifecycle with guarded updates and import. Release binaries and source are public; Registry publication is pending. |\n| Terminal workspace | Fleet attention, evidence inspection, and state-valid guarded actions. |\n| Browser console | Separate deny-by-default private-preview application using the same control API. |\n| Read-only MCP server | Closed-world operational inspection without deployment, scaling, promotion, deletion, budget, or secret tools. |\n\nInferCrane `v1.0.0-rc.1`\n\nis the first public beta. The stable `v1.0.0`\n\nrelease will promote the exact\nqualified product contract after the prerelease cycle; no earlier development tag should be treated\nas a supported public release.\n\n- Local race, PostgreSQL, fault-injection, Docker, Kind, KWOK, package, migration, security, and documentation gates are automated.\n- AWS has exact-tuple real GPU evidence for vLLM, SGLang, custom OCI, model identity, requests, bounded benchmarks, durable deletion, and final zero managed-resource inventory.\n- GCP GPU, real GPU Kubernetes/KServe, additional model/runtime/GPU tuples, and several distributed optimization paths still require separate real-infrastructure evidence.\n- No benchmark is generalized beyond the exact tuple and workload that produced it.\n\nSee the authoritative [compatibility and qualification policy](/infercrane/infercrane/blob/main/docs/compatibility.md),\n[AWS evidence](/infercrane/infercrane/blob/main/docs/testing/aws-real-evidence.md), and\n[feature qualification matrix](/infercrane/infercrane/blob/main/docs/testing/feature-qualification-matrix.md) before relying on an\nexact provider, runtime, model, or accelerator combination.\n\n[Five-minute quickstart](https://docs.infercrane.com/quickstart)[Product concepts](https://docs.infercrane.com/concepts)[Build new inference](https://docs.infercrane.com/showcase/build-inference)[Connect existing inference](https://docs.infercrane.com/showcase/connect-existing)[Safe releases](https://docs.infercrane.com/showcase/safe-rollouts)[Provider setup](https://docs.infercrane.com/provider-setup)[Production operations](https://docs.infercrane.com/production)[Python SDK](https://docs.infercrane.com/integrations/python)[TypeScript SDK](https://docs.infercrane.com/integrations/typescript)[Terraform provider](https://docs.infercrane.com/integrations/terraform)[Security](/infercrane/infercrane/blob/main/SECURITY.md)[Support](/infercrane/infercrane/blob/main/SUPPORT.md)\n\nMintlify generates [ llms.txt](https://docs.infercrane.com/llms.txt) and\n\n[from the public documentation. Every public documentation page is also available as Markdown by appending](https://docs.infercrane.com/llms-full.txt)\n\n`llms-full.txt`\n\n`.md`\n\nto its URL.Contributions are welcome. Start with [CONTRIBUTING.md](/infercrane/infercrane/blob/main/CONTRIBUTING.md), follow the\n[Code of Conduct](/infercrane/infercrane/blob/main/CODE_OF_CONDUCT.md), and sign commits with `git commit -s`\n\n. Changes must include\ntests and relevant documentation. Durable architecture, security, storage, and dependency changes\nmust update their authoritative public documentation.\n\nNever disclose credentials, prompts, model responses, private endpoints, or suspected\nvulnerabilities in a public issue. Use the private reporting process in [SECURITY.md](/infercrane/infercrane/blob/main/SECURITY.md).\nQuestions and reproducible defects follow [SUPPORT.md](/infercrane/infercrane/blob/main/SUPPORT.md).\n\nInferCrane Community is available under the [Apache License 2.0](/infercrane/infercrane/blob/main/LICENSE). Hosted and enterprise products\nare separate distributions and are not licensed by this repository. Release archives also include\n[third-party notices](/infercrane/infercrane/blob/main/THIRD_PARTY_NOTICES.md) and a release-specific SPDX SBOM. The InferCrane name\nand crane logo remain subject to the [trademark policy](/infercrane/infercrane/blob/main/TRADEMARKS.md).", "url": "https://wpnews.pro/news/infercrane-deploy-and-safely-evolve-self-hosted-ai-inference", "canonical_source": "https://github.com/infercrane/infercrane", "published_at": "2026-08-27 16:49:27+00:00", "updated_at": "2026-08-27 17:19:32.988725+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "ai-products", "developer-tools"], "entities": ["InferCrane", "vLLM", "SGLang", "LiteLLM", "AWS", "GCP", "Kubernetes", "RunPod"], "alternates": {"html": "https://wpnews.pro/news/infercrane-deploy-and-safely-evolve-self-hosted-ai-inference", "markdown": "https://wpnews.pro/news/infercrane-deploy-and-safely-evolve-self-hosted-ai-inference.md", "text": "https://wpnews.pro/news/infercrane-deploy-and-safely-evolve-self-hosted-ai-inference.txt", "jsonld": "https://wpnews.pro/news/infercrane-deploy-and-safely-evolve-self-hosted-ai-inference.jsonld"}}