If you are building AI applications, you quickly hit a fork in the road:
In this guide, I'll walk through how we solved this by building a lightweight FastAPI gateway deployed to Azure Container Apps (ACA) configured with automated scale-to-zero.
The gateway sits between client applications and the underlying Hugging Face Transformers model:
/v1/chat/completions and /v1/models. Existing LangChain, LlamaIndex, or frontend chat UIs only need their base_url pointed to the Azure endpoint.TextIteratorStreamer for full typewriter streaming in client UIs.
Here is the simplified implementation of the OpenAI-compatible streaming endpoint:
import json
import asyncio
from fastapi import FastAPI, Depends, HTTPException, Security
from fastapi.responses import StreamingResponse
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel
from typing import List, Optional
app = FastAPI(title="Private Azure AI Gateway")
security = HTTPBearer()
class ChatMessage(BaseModel):
role: str
content: str
class ChatCompletionRequest(BaseModel):
model: str
messages: List[ChatMessage]
stream: Optional[bool] = False
temperature: Optional[float] = 0.7
max_tokens: Optional[int] = 512
@app.post("/v1/chat/completions")
async def chat_completions(
req: ChatCompletionRequest,
auth: HTTPAuthorizationCredentials = Security(security)
):
if req.stream:
async def event_generator():
tokens = ["Hello", " from", " self-hosted", " DeepSeek!"]
for token in tokens:
chunk = {
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"choices": [{"delta": {"content": token}, "finish_reason": None}]
}
yield f"data: {json.dumps(chunk)}\n\n"
await asyncio.sleep(0.05)
yield "data: [DONE]\n\n"
return StreamingResponse(event_generator(), media_type="text/event-stream")
return {
"id": "chatcmpl-123",
"object": "chat.completion",
"choices": [{
"message": {"role": "assistant", "content": "Sample response"},
"finish_reason": "stop"
}]
}
In the deployment configuration, the secret sauce is setting min_replicas = 0:
{
"min_replicas": 0,
"max_replicas": 3,
"scale_to_zero": true,
"idle_timeout_seconds": 300
}
When deployed via Azure CLI:
az containerapp create \
--name deepseek-gateway \
--resource-group my-ai-rg \
--environment my-aca-env \
--image myacr.azurecr.io/deepseek-gateway:latest \
--target-port 8080 \
--ingress external \
--min-replicas 0 \
--max-replicas 3 \
--scale-rule-name http-rule \
--scale-rule-type http \
--scale-rule-http-concurrency 10
In your existing Python application, you do not need to install custom Hugging Face libraries. Just point the official OpenAI SDK to your Azure endpoint:
from openai import OpenAI
client = OpenAI(
base_url="https://deepseek-gateway.azurecontainerapps.io/v1",
api_key="your-secure-bearer-key"
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
messages=[{"role": "user", "content": "Write a Python script to sort a list of dicts."}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
The complete open-source repository is available on GitHub:
👉 GitHub: faizalbagwan786/azure-model-deployer
If you want the production-ready boilerplate with pre-configured NVIDIA CUDA 12.1 GPU Dockerfiles, vision support (SmolVLM), test suite, and 1-command Azure CLI automated deployment scripts, check out the Gumroad Release ($10).
If you are working on deploying custom private models on Azure or optimizing cloud AI costs, feel free to connect or ask questions in the comments below!