Dynamo 1.6.2 Turns an Oversized-Prompt HTTP 500 Into a 400: Why Error Codes Decide Your Retry Load, Plus the Opt-In GC Freeze for SGLang Workers NVIDIA released Dynamo v1.6.2, a patch on v1.6.1 that changes how the TensorRT-LLM backend handles oversized prompts: requests exceeding the engine's max_num_tokens now return HTTP 400 invalid_request_error instead of HTTP 500 api_error, and the worker keeps running. The release also adds an opt-in --freeze-gc-after-init flag (DYN_SGL_FREEZE_GC_AFTER_INIT) for SGLang decode and aggregated workers to reduce streaming stalls, and moves the SGLang image to a uv-installed CPython 3.12.16, which breaks Python scripting in cuda-gdb inside that image. NVIDIA Dynamo v1.6.2 tagged 2026-11-05 is a patch release on v1.6.1, and the two changes worth your attention are small in code and large in operations. First, a TensorRT-LLM prompt longer than the engine's max num tokens now returns HTTP 400 invalid request error instead of HTTP 500 api error , and the worker keeps running. Second, SGLang decode and aggregated workers get an opt-in --freeze-gc-after-init flag DYN SGL FREEZE GC AFTER INIT meant to reduce streaming stalls in long-running workers. Backend versions are unchanged from v1.6.1: SGLang v0.5.21, TensorRT-LLM v1.3.0rc29, vLLM v0.31.0. Below, the release notes' statements are kept separate from my own arithmetic, and the charts are marked illustrative. Status codes are not cosmetic. They are the contract that every layer between the user and the GPU uses to decide what to do next. Before 1.6.2, an oversized non-streaming /v1/messages request on the TensorRT-LLM backend returned a 500. The v1.6.0 release notes list this as a known issue with no workaround. A request that can never succeed was labeled as a transient server failure. Any retry policy that treats 5xx as retryable then re-sends a request that fails identically every time. Here is the simple arithmetic. Suppose a client library allows 3 retries on 5xx. One oversized prompt becomes 4 attempts that reach the frontend. If 50 requests per second are oversized a misconfigured agent loop pasting a whole repository into context is a realistic source , that is 200 attempts per second of pure waste, versus 50 with a 400. The numbers in the chart below are illustrative, not measured. The point is the multiplier, which is set by your retry policy and not by Dynamo. There is a second cost. Error-rate dashboards and SLO burn alerts usually count 5xx as your failures and 4xx as the caller's. An api error spike from oversized prompts pages the on-call engineer for something that is not an outage. Moving the response to 400 puts the signal in the right bucket. From the v1.6.2 notes: | Area | Change | |---|---| | Bug fix | TensorRT-LLM oversized prompt: HTTP 400 invalid request error , worker keeps running 16193 | | Bug fix | SGLang opt-in --freeze-gc-after-init / DYN SGL FREEZE GC AFTER INIT , off by default 15203, 16761 | | Dependency | OS packages refreshed libgnutls30t64 , libbrotli1 | | Dependency | SGLang image moves from Ubuntu Python 3.12.3 to uv-installed CPython 3.12.16 OpenSSL 3.5.9 | | Dependency | Planner image on Python 3.13.18 slim, distroless runtime base | | Dependency | Mooncake floor 0.3.13.post1 to 0.3.14 SGLang and vLLM images | | Dependency | PyNvVideoCodec 2.2.3 to 2.2.4, pyasn1 floored at =0.6.5,<0.7, diffusers pinned at 0.40.0 | The release notes flag one item as able to affect existing workflows: the SGLang image's Python change. Because the image no longer uses Ubuntu's libpython3.12 , cuda-gdb inside that image no longer supports Python scripting. If your debugging runbook relies on Python pretty-printers in cuda-gdb , read that line twice. You can test the contract directly. The sketch below sends a deliberately oversized prompt and reports the status. The endpoint, model name, and sizes are placeholders you set for your own deployment. python import httpx, time URL = "http://localhost:8000/v1/messages" your frontend MAX NUM TOKENS = 8192 your engine's max num tokens def probe n words: int : payload = { "model": "your-model", "max tokens": 16, "messages": {"role": "user", "content": "word " n words} , } t0 = time.perf counter r = httpx.post URL, json=payload, timeout=60 return r.status code, r.json .get "error", {} .get "type" , time.perf counter - t0 status, err, dt = probe MAX NUM TOKENS 2 print status, err, f"{dt:.2f}s" expected on 1.6.2 + TensorRT-LLM: 400 invalid request error on 1.6.0/1.6.1: 500 api error documented known issue Then check the layers in front of it. Three places commonly retry on 5xx without anyone remembering they do: Log the attempt count per request id for a day. If oversized prompts show up as repeated attempts, the fix is not only the upgrade. Add a cheap pre-check at the gateway that rejects prompts over the token budget before they reach a GPU worker, and make the budget a shared configuration value rather than a number copied into three repositories. One more detail from the v1.6.0 known-issues list: when a worker rejects a request, the frontend returns a generic error without the worker's message. The 1.6.2 notes carry the v1.6.0 known issues forward except the oversized-prompt one, so on 1.6.2 you should expect the correct class of status but not necessarily a helpful body. Do not build client logic that parses error text. The second change targets Python's garbage collector inside SGLang decode and aggregated workers. The release notes describe the goal as reducing streaming stalls in long-running workers, state that it is off by default, and advise checking memory use against a control worker before enabling it widely. The mechanism it builds on is standard CPython. The cyclic collector periodically walks tracked container objects looking for reference cycles. In a worker that loaded a large amount of Python state at startup config objects, tokenizer structures, scheduler tables , each full collection has a lot to walk, and a long walk is a pause on a thread that may also be feeding tokens to streams. Python exposes gc.freeze , which moves everything currently tracked into a permanent generation that later collections skip. The release notes name the flag; I have not read the implementation, so treat the following as the general technique and not a description of Dynamo's code. python import gc, time def build startup state : stand-in for config, tokenizer tables, scheduler structures return {"id": i, "next": None} for i in range 2 000 000 state = build startup state def full gc ms : t = time.perf counter gc.collect return time.perf counter - t 1000 before = full gc ms gc.freeze move startup objects out of future scans after = full gc ms print f"full collection before freeze: {before:.1f} ms, after: {after:.1f} ms" Run it on your own hardware. Exact numbers depend on the machine and object count, but the shape is consistent: the pause drops once long-lived startup objects stop being scanned. The trade-off is built in as well. Frozen objects are never collected, so any cycle among them that later becomes garbage stays in memory. That is why the advice to compare against a control worker is the right one. DYN SGL FREEZE GC AFTER INIT on the canary only. If you have no inter-token gap metric today, add one first. Without it you cannot tell whether the flag helped. The compatibility table is unchanged from v1.6.1, which is itself useful: a patch that does not move the engine versions means model-level behavior should not shift. What can shift is the runtime underneath. | Component | v1.6.2 | |---|---| | SGLang | v0.5.21 | | TensorRT-LLM | v1.3.0rc29 | | vLLM | v0.31.0 | | UCX | v1.21.0 | | CUDA variants | SGLang 13.0, TensorRT-LLM 13.1, vLLM 13.0 | Note how far these pins sit from the newest upstream tags. As of 2026-11-21, vLLM's latest tag is v0.34.0 and SGLang's is v0.5.23, so a Dynamo image is a tested bundle that trails the engines' newest releases by design. If you need a specific engine feature, check the pin before you assume the image has it. For the upgrade diff, look at three things in this order: the image's Python version and any native wheels you layer on top the SGLang change replaces the interpreter , the Mooncake floor if you use KV transfer, and any debugging tooling that touches cuda-gdb Python scripting. Does 1.6.2 fix every oversized-prompt path? The notes say it fixes the v1.6.0 known issue for non-streaming /v1/messages on TensorRT-LLM. They claim nothing about other backends, so test each backend you run. Will clients now stop retrying? Only if their policy treats 400 as non-retryable, which is the common default. Verify yours with a probe like the one above. Is --freeze-gc-after-init safe to turn on everywhere? The notes ship it off by default and recommend comparing memory against a control worker first. Treat it as a canary-first change. Does the patch change model outputs? Engine pins are unchanged from v1.6.1, so the release does not move engine versions. Rebuilt images still deserve your normal regression run. Where is the primary source? The v1.6.2 release notes on the project's GitHub releases page, and the v1.6.0 notes for the known-issues list this patch partly resolves. Sources: NVIDIA Dynamo v1.6.2 and v1.6.0 release notes github.com/ai-dynamo/dynamo/releases ; vLLM and SGLang release tags on GitHub, checked 2026-11-21.