cd /news/ai-infrastructure/intermittent-500s-on-inference-endpo… · home topics ai-infrastructure article
[ARTICLE · art-59141] src=discuss.huggingface.co ↗ pub= topic=ai-infrastructure verified=true sentiment=↓ negative

Intermittent 500s on Inference Endpoint - requests not reaching the container

A user reports intermittent HTTP 500 errors on a custom-handler Inference Endpoint for text embeddings, with ~70% of requests failing since early morning on 2026-07-14. The failures appear to occur at the gateway/router layer before reaching the container, as container logs only show successful requests. The user has ruled out GPU-specific, endpoint-specific, and code-related issues, and seeks information about a possible known problem with the us-east-1 inference gateway.

read1 min views1 publishedJul 14, 2026

I’m using a personal account. I’m running a custom-handler Inference Endpoint for text embeddings (Llama-3.1-8B). Since early this morning on 2026-07-14, I’m getting intermittent HTTP 500s (~70% of requests fail, ~30% return 200 normally).

What I’ve ruled out:

Not GPU-specific - happens on both L40S and A100.

Not endpoint-specific - brand-new endpoints show the same. Not my code/deps - the same build ran hundreds of thousands of requests fine yesterday.

Container is healthy - model loads successfully, and successful requests return correct results in 30–415 ms.

Key finding: the container logs only show the successful requests. The failed (500) requests don’t appear in the container logs at all — they seem to be failing at the gateway/router layer before reaching the container.

The status page shows no incident. Is there a known issue with the us-east-1 inference gateway, or something on my account causing this? Happy to share endpoint IDs privately.

Thanks!

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @llama-3.1-8b 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/intermittent-500s-on…] indexed:0 read:1min 2026-07-14 ·