An AI agent read a Shift_JIS page as 208,953 replacement characters A pull request (#188766) in the openclaw repository fixes a link-understanding bug in which non-UTF-8 web pages reached the model as garbled text, with the Shift_JIS page at abehiroshi.la.coocan.jp producing 208,953 replacement characters before the repair. The fix makes link understanding reuse the project's existing charset-aware web response decoder, and a Node 24 secretless Docker Gateway transcript showed Shift_JIS titles becoming readable while UTF-8 controls remained unchanged. Automated review scored the change 5/6 on overall readiness, proof confidence and patch quality, found no actionable issues, and marked it ready for maintainer review with CI still running. Repository navigation fix link-understanding : non-UTF-8 pages reach the model as garbled text - 188766 Conversation | 🦞👀 ClawSweeper picked this up. Pull request received. I will update this pull request when review starts. ClawSweeper review complete ClawSweeper finished reviewing this revision. The review result is being finalized. View the workflow run https://github.com/openclaw/clawsweeper/actions/runs/41138427848 . | clawsweeper https://github.com/apps/clawsweeper Bot added clawsweeper:intake:bug https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug clawsweeper:intake:needs-environment https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment P2 https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3AP2 rating: 🦐 gold shrimp https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22rating%3A%20%F0%9F%A6%90%20gold%20shrimp%22 status: 📣 needs proof https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22status%3A%20%F0%9F%93%A3%20needs%20proof%22 clawsweeper:intake:bug https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug clawsweeper:intake:needs-environment https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment Oct 30, 2026 event-34737915416 | Codex review: needs maintainer review before merge. What this changes This PR makes link understanding reuse the existing charset-aware web response decoder. Example: An agent reads the Shift JIS page at http://abehiroshi.la.coocan.jp/ http://abehiroshi.la.coocan.jp/ . Review scores Product Kind: Bug fix · Worth it: Yes · Fix scope: Complete User problem: Agents receive unreadable text from linked pages declaring supported legacy encodings. Reason: The repair restores readable page content through an established decoder without adding configuration, dependencies, or a competing policy. Merge readiness ✅ Ready for maintainer review This PR fixes a current decoding bug with a minimal shared-owner change; the earlier finding is resolved, no new defects were found, and CI is still running. Priority: P2 Reviewed head: 73ec930363d6b9a218a091b5f0c31f57101b6ea8 Before merge None. Findings None. Agent review details | | Measure | Result | What it means | |---|---|---| | Overall readiness | 🦞 diamond lobster 5/6 | A focused shared-owner repair with relevant shipped Gateway evidence, reported red/green regression results, and no open findings. | | Proof confidence | 🦞 diamond lobster 5/6 | Sufficient terminal : The supplied Node 24 secretless Docker Gateway transcript exercises sessions.create and chat.send through guarded public HTTP, the real CLI processor, agent runtime, model request, final reply, and history; Shift JIS titles become readable while UTF-8 controls remain unchanged, and the proven candidate has the reviewed head's identical tree. The deterministic model endpoint proves text propagation rather than semantic model quality. | | Patch quality | 🦞 diamond lobster 5/6 | No actionable review findings were identified. | How this fits together Link understanding fetches message URLs through the guarded HTTP transport, decodes bounded response bytes, and supplies text to configured CLI processors and agent input. php flowchart LR A Message URL -- B Guarded HTTP fetch B -- C Bounded response bytes C -- D Shared charset decoder D -- E CLI link processor E -- F Agent model input F -- G Assistant reply Technical review Best possible solution: Merge the shared-decoder repair once the existing CI gates pass. Do we have a high-confidence way to reproduce the issue? Current-main source establishes unconditional UTF-8 decoding, and the supplied Gateway comparison demonstrates the resulting garbled title before the repair. Is this the best way to solve the issue? Reusing the existing decoder is the smallest complete fix: it removes the divergent decoding decision while preserving transport limits, cancellation, and permissive fallback. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against e279b64ffe86 https://github.com/openclaw/openclaw/commit/e279b64ffe86578edff6c72cd39f98893a0e5ff2 . Provenance checked - Link response decoding changes intended behavior with a stated reason 5fe1414 https://github.com/openclaw/openclaw/commit/5fe1414dfd69510a1669ebbc0f9516515d57f84f : This refactor carried existing link processing into its application owner while preserving behavior; it does not establish when unconditional UTF-8 decoding originated. - Shared decoder export keeps the original intent d185f9a https://github.com/openclaw/openclaw/commit/d185f9a0b484b144494d24bad18cba59e3740ec1 : The existing decoder preserves charset handling and safe truncated-prefix decoding; the PR exports it internally without changing its implementation. - Transport regression consolidation keeps the original intent 4ec56e7 https://github.com/openclaw/openclaw/commit/4ec56e7ff47c78414e883376dbc5fb71846a6c7e : Prior test consolidation retained distinct transport and lifecycle scenarios; the revised fixture adds charset coverage while retaining URL selection and ordering assertions. Testing Proof path: shipped entry point. Security None. Evidence What I checked: - Verified introduced change: The introduced diff exports the existing decoder, replaces the link fetcher's unconditional UTF-8 decoding, and incorporates a Shift JIS regression into existing transport coverage. src/link-understanding/apply.ts:48 https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/link-understanding/apply.ts L48 , 73ec930363d6 https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8 - Current main still contains the defect: Current main decodes fetched link bytes with new TextDecoder .decode buffer , ignoring declared legacy encodings before passing content to processors. src/link-understanding/apply.ts:47 https://github.com/openclaw/openclaw/blob/e279b64ffe86578edff6c72cd39f98893a0e5ff2/src/link-understanding/apply.ts L47 , e279b64ffe86 https://github.com/openclaw/openclaw/commit/e279b64ffe86578edff6c72cd39f98893a0e5ff2 - Existing decoding contract: The shared decoder preserves BOM precedence, header and document charset detection, and nonfatal UTF-8 fallback for unsupported labels; existing tests cover these contracts. src/agents/tools/web-shared.ts:170 https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/agents/tools/web-shared.ts L170 , 73ec930363d6 https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8 - Shipped Gateway evidence: obviyus supplied before/after terminal output in fix link-understanding : non-UTF-8 pages reach the model as garbled text 188766 comment https://github.com/openclaw/openclaw/pull/188766 issuecomment-6535787163 : isolated built Gateways fetched public pages through the normal guard, ran a real CLI processor, and carried corrected titles through actual model requests, replies, and history. The deterministic model fixture echoes the supplied title without decoding or repairing it; UTF-8 controls remain unchanged. 73ec930363d6 https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8 - Proof applies to the current tree: The tested candidate and reviewed head both resolve to tree 6b9d2585e88508e4f86e5d5c5f78834cf54f390e, verifying the recorded proof's unchanged source inputs. 73ec930363d6 https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8 - Earlier finding resolved: The separate regression table was removed and its meaningful charset assertion consolidated into the existing transport case, eliminating the test growth that prompted the earlier allocation finding. src/link-understanding/runner.transport.test.ts:123 https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/link-understanding/runner.transport.test.ts L123 , 73ec930363d6 https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8 PR surface Source +1, Tests 0. Total +1 across 3 files. Added test files: 0. View PR surface stats | Area | Files | Added | Removed | Net | |---|---|---|---|---| | Source | 2 | 3 | 2 | +1 | | Tests | 1 | 5 | 5 | 0 | | Docs | 0 | 0 | 0 | 0 | | Config | 0 | 0 | 0 | 0 | | Generated | 0 | 0 | 0 | 0 | | Other | 0 | 0 | 0 | 0 | | Total | 3 | 8 | 7 | +1 | Labels Label changes: - add proof: sufficient : Contributor real behavior proof is sufficient. - add rating: 🦞 diamond lobster : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster. - add status: 👀 ready for maintainer look : ClawSweeper has no concrete contributor-facing blocker left for this PR. - remove rating: 🦐 gold shrimp : Current PR rating is rating: 🦞 diamond lobster , so this older rating label is no longer current. - remove status: 📣 needs proof : Current PR status label is status: 👀 ready for maintainer look . Label justifications: - P2 : Declared legacy-charset pages produce incorrect agent input, with impact limited to configured link-understanding flows. - rating: 🦞 diamond lobster : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster. - status: 👀 ready for maintainer look : ClawSweeper has no concrete contributor-facing blocker left for this PR. - proof: sufficient : Contributor real behavior proof is sufficient. Rating scale 6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof a screenshot, video, or linked artifact that directly shows the changed behavior. Workflow ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired. History Review history 1 earlier review cycle - reviewed 2026-10-30T22:33:32.625Z sha 38de2ab https://github.com/openclaw/openclaw/commit/38de2ab4d00779a1d8867fdad9c8cf993fdcfc3d :: needs real behavior proof before merge. :: P2 Allocate the added regression-test lines Reviewed October 30, 2026, 7:41 PM ET / 23:41 UTC Revision 2 . | Correctness gate and shipped Gateway proof This is not performance-only : the same public page reaches both the actual model request and the final assistant reply with the wrong characters on main, and readable Japanese after the repair. The build and test proof ran at 86a4b241c744b6e2de9f70a93fbe23c6a70a692f . Before publication, the guarded push detected the contributor's new allocation commit 3cf4ae615b2f ; I merged it intact and removed that now-unused 13-line allocation because the consolidated tests have zero growth. The resulting final source tree is identical to the proven tree: both Git tree IDs are 6b9d2585e88508e4f86e5d5c5f78834cf54f390e . No behavior, source, test, dependency or build input changed; only commit ancestry changed, so the recorded proof is reused rather than rebuilt. Shipped route: operator WebSocket sessions.create → chat.send → normal OpenClaw agent preprocessing → STRICT guarded public HTTP fetch → real configured CLI link processor → real OpenClaw agent runtime → OpenAI-compatible model fixture → agent.wait terminal reply and chat.history . The CLI processor reads the real fetched document from stdin, records its decoded title and replacement-character count, and outputs only LINK TITLE=