Hi Hugging Face team,
We are seeing reproducible HTTP 400 failures through Hugging Face Inference Providers on the exact route:
openai/gpt-oss-20b:groq
The workload uses strict structured outputs for a deterministic entity-matching classification. The inputs are public and non-personal. We ran bounded diagnostics without retries, prompt tuning, or fixture changes.
Every failed call returned:
invalid_request_error
json_validate_failed
structured_generation_failed
Five repetitions per conservative review
fixture:
`entity-review-01`
: 0/5 successful`entity-review-02`
: 1/5 successfulThe one successful response matched the expected decision and confidence exactly. The same 188-byte strict output schema succeeded on a comparator. Request sizes were similar, so schema shape and request size do not explain the fixture-specific failures.
Failed request IDs for entity-review-01
:
req_01m0jacw8zemabfp437wn7hdt8
req_01m0jacwxjeeva9k8rd84gc3pj
req_01m0jacxdne98r2akb1janc4w8
req_01m0jacy5qep39padbzhy7tzsp
req_01m0jacyw4eqvrhwd1p4b0pfak
Failed request IDs for entity-review-02
:
req_01m0jad0rvedvva48h0mg0h4qk
req_01m0jad1dse1zsj83nvcjb40yd
req_01m0jad2fcexc89hmwe2p34arb
req_01m0jad329en9s172f1vrcqrvg
We then kept the prompt and inputs unchanged but replaced the response schema in memory with singleton enums requiring the already-checked output: match=review
, confidence=low
.
Results across three repetitions per fixture:
`entity-review-01`
: 3/3 successful`entity-review-02`
: 2/3 successfulThe remaining failure had the same fingerprint even though the model had no enum choice:
req_01m0jzsav8e71rdt2j9qc4k27m
This suggests enum selection contributes materially, but does not fully explain the route-level structured-generation failures.
Could you please confirm:
json_validate_failed
originated (HF router validation, provider validation, or generation)?openai/gpt-oss-20b:groq
has a known reliability limitation with strict JSON-schema outputs?The evidence is stored in a private repository, so the complete reproducible record is included here instead of linking to inaccessible CI pages.
Normal-schema targeted diagnostic:
95c970d8f66979413bed18b680b86cfeeeec8780
32490428886
hf-benchmark-report-32490428886
(artifact 9449532403
)sha256:c04babef1bb956d0b6f3633887521fbe6583d1e805bd85407dd998250ba1d7ef
1.3.0
Forced-singleton-schema diagnostic:
49ea63697f5afb82cccad93294ed858770c93232
32523118374
hf-benchmark-report-32523118374
(artifact 9461257213
)sha256:da529cd0ef0fd601b9cae7dc948b72acdc6d3ce1ccf5be3f3ce1ffc0a486a769
1.4.0
Both diagnostics used the same policy (`phase-12-2-hugging-face-public-evaluation-v1`
), registry (`hugging-face-structured-evaluation@1.1.0`
), and fixtures (`public-business-controlled-ai-v1@1.0.0`
). Raw prompts, inputs, outputs, provider messages, failed generations, and credentials were not retained. Authority leakage was zero.