{"slug": "an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters", "title": "An AI agent read a Shift_JIS page as 208,953 replacement characters", "summary": "A pull request (#188766) in the openclaw repository fixes a link-understanding bug in which non-UTF-8 web pages reached the model as garbled text, with the Shift_JIS page at abehiroshi.la.coocan.jp producing 208,953 replacement characters before the repair. The fix makes link understanding reuse the project's existing charset-aware web response decoder, and a Node 24 secretless Docker Gateway transcript showed Shift_JIS titles becoming readable while UTF-8 controls remained unchanged. Automated review scored the change 5/6 on overall readiness, proof confidence and patch quality, found no actionable issues, and marked it ready for maintainer review with CI still running.", "body_md": "## Repository navigation\n\n# fix(link-understanding): non-UTF-8 pages reach the model as garbled text - #188766\n\n## Conversation\n\n| 🦞👀 ClawSweeper picked this up. Pull request received. I will update this pull request when review starts. ClawSweeper review complete ClawSweeper finished reviewing this revision. The review result is being finalized. [View the workflow run](https://github.com/openclaw/clawsweeper/actions/runs/41138427848) . | \n\n[clawsweeper](https://github.com/apps/clawsweeper)Bot added\n\n[clawsweeper:intake:bug](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug)\n\n[clawsweeper:intake:needs-environment](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment)\n\n[P2](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3AP2)\n\n[rating: 🦐 gold shrimp](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22rating%3A%20%F0%9F%A6%90%20gold%20shrimp%22)\n\n[status: 📣 needs proof](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22status%3A%20%F0%9F%93%A3%20needs%20proof%22)\n\n[clawsweeper:intake:bug](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug)\n\n[clawsweeper:intake:needs-environment](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment)\n\n[Oct 30, 2026](#event-34737915416)\n\n| Codex review: needs maintainer review before merge. What this changes This PR makes link understanding reuse the existing charset-aware web response decoder. **Example:** An agent reads the Shift_JIS page at[http://abehiroshi.la.coocan.jp/](http://abehiroshi.la.coocan.jp/) .  Review scores Product **Kind:** Bug fix ·**Worth it:** Yes ·**Fix scope:** Complete **User problem:** Agents receive unreadable text from linked pages declaring supported legacy encodings. **Reason:** The repair restores readable page content through an established decoder without adding configuration, dependencies, or a competing policy. Merge readiness ✅ **Ready for maintainer review** This PR fixes a current decoding bug with a minimal shared-owner change; the earlier finding is resolved, no new defects were found, and CI is still running. **Priority:** P2 **Reviewed head:**`73ec930363d6b9a218a091b5f0c31f57101b6ea8` Before merge None. Findings None. **Agent review details** | \n\n| Measure | Result | What it means | \n|---|---|---|\n| **Overall readiness** | 🦞 diamond lobster **(5/6)** | A focused shared-owner repair with relevant shipped Gateway evidence, reported red/green regression results, and no open findings. | \n| **Proof confidence** | 🦞 diamond lobster **(5/6)** | Sufficient (terminal): The supplied Node 24 secretless Docker Gateway transcript exercises sessions.create and chat.send through guarded public HTTP, the real CLI processor, agent runtime, model request, final reply, and history; Shift_JIS titles become readable while UTF-8 controls remain unchanged, and the proven candidate has the reviewed head's identical tree. The deterministic model endpoint proves text propagation rather than semantic model quality. | \n| **Patch quality** | 🦞 diamond lobster **(5/6)** | No actionable review findings were identified. | \n\n### How this fits together\n\nLink understanding fetches message URLs through the guarded HTTP transport, decodes bounded response bytes, and supplies text to configured CLI processors and agent input.\n\n``` php\nflowchart LR\n  A[Message URL] --> B[Guarded HTTP fetch]\n  B --> C[Bounded response bytes]\n  C --> D[Shared charset decoder]\n  D --> E[CLI link processor]\n  E --> F[Agent model input]\n  F --> G[Assistant reply]\n```\n\n### Technical review\n\nBest possible solution:\n\nMerge the shared-decoder repair once the existing CI gates pass.\n\nDo we have a high-confidence way to reproduce the issue?\n\nCurrent-main source establishes unconditional UTF-8 decoding, and the supplied Gateway comparison demonstrates the resulting garbled title before the repair.\n\nIs this the best way to solve the issue?\n\nReusing the existing decoder is the smallest complete fix: it removes the divergent decoding decision while preserving transport limits, cancellation, and permissive fallback.\n\nAGENTS.md: found and applied where relevant.\n\nCodex review notes: model internal, reasoning medium; reviewed against [e279b64ffe86](https://github.com/openclaw/openclaw/commit/e279b64ffe86578edff6c72cd39f98893a0e5ff2).\n\n### Provenance checked\n\n- `Link response decoding` changes intended behavior with a stated reason ([`5fe1414`](https://github.com/openclaw/openclaw/commit/5fe1414dfd69510a1669ebbc0f9516515d57f84f) : This refactor carried existing link processing into its application owner while preserving behavior; it does not establish when unconditional UTF-8 decoding originated.)\n- `Shared decoder export` keeps the original intent ([`d185f9a`](https://github.com/openclaw/openclaw/commit/d185f9a0b484b144494d24bad18cba59e3740ec1) : The existing decoder preserves charset handling and safe truncated-prefix decoding; the PR exports it internally without changing its implementation.)\n- `Transport regression consolidation` keeps the original intent ([`4ec56e7`](https://github.com/openclaw/openclaw/commit/4ec56e7ff47c78414e883376dbc5fb71846a6c7e) : Prior test consolidation retained distinct transport and lifecycle scenarios; the revised fixture adds charset coverage while retaining URL selection and ordering assertions.)\n\n### Testing\n\nProof path: shipped entry point.\n\n### Security\n\nNone.\n\n### Evidence\n\nWhat I checked:\n\n- **Verified introduced change:** The introduced diff exports the existing decoder, replaces the link fetcher's unconditional UTF-8 decoding, and incorporates a Shift_JIS regression into existing transport coverage. ([`src/link-understanding/apply.ts:48`](https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/link-understanding/apply.ts#L48) ,[73ec930363d6](https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8) )\n- **Current main still contains the defect:** Current main decodes fetched link bytes with new TextDecoder().decode(buffer), ignoring declared legacy encodings before passing content to processors. ([`src/link-understanding/apply.ts:47`](https://github.com/openclaw/openclaw/blob/e279b64ffe86578edff6c72cd39f98893a0e5ff2/src/link-understanding/apply.ts#L47) ,[e279b64ffe86](https://github.com/openclaw/openclaw/commit/e279b64ffe86578edff6c72cd39f98893a0e5ff2) )\n- **Existing decoding contract:** The shared decoder preserves BOM precedence, header and document charset detection, and nonfatal UTF-8 fallback for unsupported labels; existing tests cover these contracts. ([`src/agents/tools/web-shared.ts:170`](https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/agents/tools/web-shared.ts#L170) ,[73ec930363d6](https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8) )\n- **Shipped Gateway evidence:** obviyus supplied before/after terminal output in[fix(link-understanding): non-UTF-8 pages reach the model as garbled text #188766 (comment)](https://github.com/openclaw/openclaw/pull/188766#issuecomment-6535787163) : isolated built Gateways fetched public pages through the normal guard, ran a real CLI processor, and carried corrected titles through actual model requests, replies, and history. The deterministic model fixture echoes the supplied title without decoding or repairing it; UTF-8 controls remain unchanged. ([73ec930363d6](https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8) )\n- **Proof applies to the current tree:** The tested candidate and reviewed head both resolve to tree 6b9d2585e88508e4f86e5d5c5f78834cf54f390e, verifying the recorded proof's unchanged source inputs. ([73ec930363d6](https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8) )\n- **Earlier finding resolved:** The separate regression table was removed and its meaningful charset assertion consolidated into the existing transport case, eliminating the test growth that prompted the earlier allocation finding. ([`src/link-understanding/runner.transport.test.ts:123`](https://github.com/openclaw/openclaw/blob/73ec930363d6b9a218a091b5f0c31f57101b6ea8/src/link-understanding/runner.transport.test.ts#L123) ,[73ec930363d6](https://github.com/openclaw/openclaw/commit/73ec930363d6b9a218a091b5f0c31f57101b6ea8) )\n\n### PR surface\n\nSource +1, Tests 0. Total +1 across 3 files. Added test files: 0.\n\n## View PR surface stats\n\n| Area | Files | Added | Removed | Net | \n|---|---|---|---|---|\n| Source | 2 | 3 | 2 | +1 | \n| Tests | 1 | 5 | 5 | 0 | \n| Docs | 0 | 0 | 0 | 0 | \n| Config | 0 | 0 | 0 | 0 | \n| Generated | 0 | 0 | 0 | 0 | \n| Other | 0 | 0 | 0 | 0 | \n| **Total** | **3** | **8** | **7** | **+1** | \n\n### Labels\n\nLabel changes:\n\n- add `proof: sufficient` : Contributor real behavior proof is sufficient.\n- add `rating: 🦞 diamond lobster` : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.\n- add `status: 👀 ready for maintainer look` : ClawSweeper has no concrete contributor-facing blocker left for this PR.\n- remove `rating: 🦐 gold shrimp` : Current PR rating is`rating: 🦞 diamond lobster` , so this older rating label is no longer current.\n- remove `status: 📣 needs proof` : Current PR status label is`status: 👀 ready for maintainer look` .\n\nLabel justifications:\n\n- `P2` : Declared legacy-charset pages produce incorrect agent input, with impact limited to configured link-understanding flows.\n- `rating: 🦞 diamond lobster` : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.\n- `status: 👀 ready for maintainer look` : ClawSweeper has no concrete contributor-facing blocker left for this PR.\n- `proof: sufficient` : Contributor real behavior proof is sufficient.\n\n### Rating scale\n\n6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.\n\n### Workflow\n\nClawSweeper edits this one comment on every review. Comment `@clawsweeper re-review` for a fresh review only. Repair and merge execution lanes are retired.\n\n### History\n\n## Review history (1 earlier review cycle)\n\n- reviewed 2026-10-30T22:33:32.625Z sha [`38de2ab`](https://github.com/openclaw/openclaw/commit/38de2ab4d00779a1d8867fdad9c8cf993fdcfc3d) :: needs real behavior proof before merge. :: [P2] Allocate the added regression-test lines\n\n*Reviewed October 30, 2026, 7:41 PM ET / 23:41 UTC (Revision 2).*\n\n| Correctness gate and shipped Gateway proof This is **not performance-only** : the same public page reaches both the actual model request and the final assistant reply with the wrong characters on main, and readable Japanese after the repair.  The build and test proof ran at `86a4b241c744b6e2de9f70a93fbe23c6a70a692f` . Before publication, the guarded push detected the contributor's new allocation commit`3cf4ae615b2f` ; I merged it intact and removed that now-unused 13-line allocation because the consolidated tests have zero growth. The resulting final source tree is**identical** to the proven tree: both Git tree IDs are`6b9d2585e88508e4f86e5d5c5f78834cf54f390e` . No behavior, source, test, dependency or build input changed; only commit ancestry changed, so the recorded proof is reused rather than rebuilt. **Shipped route:** operator WebSocket`sessions.create` →`chat.send` → normal OpenClaw agent preprocessing → STRICT guarded public HTTP fetch → real configured CLI link processor → real OpenClaw agent runtime → OpenAI-compatible model fixture →`agent.wait` terminal reply and`chat.history` . The CLI processor reads the real fetched document from stdin, records its decoded title and replacement-character count, and outputs only `LINK_TITLE=<JSON title>` . The deterministic model fixture records the actual model request and echoes that supplied title without repairing or decoding it. No product modules, fetch guard, decoder, dispatch, runtime or history reader are mocked. The public HTTP fetches use real DNS/TLS/redirect handling; no SSRF exception or network preload is installed. This proves the production agent/model**path** , not a paid live-model service or semantic summarization quality. The same titles were observed in each actual model request and in the final assistant message. Example terminal replies: \n\n```\nmain:      Observed link title: �������̃z�[���y�[�W\ncandidate: Observed link title: 阿部寛のホームページ\ncontrol:   Observed link title: 気象庁 Japan Meteorological Agency\n```\n\n All **8** agent turns completed with exactly**1 model request per turn** , requested/effective provider and model unchanged,`rerouted: false` , and native receipts reporting`sourceReplyDelivered: true` /`terminalDisposition: visible` . Both the terminal reply and complete native history were checked; a missing link result fails the harness rather than masquerading as a passing request. Replay commands inside the isolated container: \n\n```\npnpm install --frozen-lockfile\npnpm build\nnode /proof/agent-proof.mjs base\nnode /proof/agent-proof.mjs candidate\nCI=1 OPENCLAW_VITEST_RUNTIME=node OPENCLAW_VITEST_MAX_WORKERS=1 \\\n  pnpm test src/link-understanding/runner.transport.test.ts \\\n  src/agents/tools/web-shared.test.ts \\\n  src/agents/agent-tools.prefix-stability.test.ts --maxWorkers=1\n```\n\n Focused candidate results: **33 passed** — transport**5** , shared response decoder**14** , prompt-prefix stability**14** . The same final transport test file on main gives**4 passed / 1 failed** at the decoding assertion: expected`日本語` , received`���{��` ; URL selection/order assertions still pass. Scoped type-aware lint, formatting and whitespace checks pass. The extra 13-line table was removed and its charset regression folded into the existing URL-selection/transport case. Missing-label and unsupported-label fallback, BOM precedence, document metadata and response isolation retain the existing shared decoder coverage. Final production delta **+1** , tests**0** ; no test allocation record or ceiling change. Retained raw evidence/harness: `proofs/188766/{agent-proof.mjs,base-agent-proof.json,candidate-agent-proof.json,base-processor.jsonl,candidate-processor.jsonl,base-model-requests.jsonl,candidate-model-requests.jsonl,candidate-focused-tests.log,base-regression.log,verification-results.json}` . Only task-owned synthetic provider credentials were used; public page titles are the only quoted page content. | \n\n| Input | Main round 1 / 2 | Candidate round 1 / 2 | \n|---|---|---|\n| [Public Shift_JIS page](http://abehiroshi.la.coocan.jp/) — HTTP redirects to HTTPS;`Content-Type: text/html` , charset declared in the document | `�������̃z�[���y�[�W` ;**13 / 13 U+FFFD** in processor input | `阿部寛のホームページ` ;**0 / 0 U+FFFD** | \n| [UTF-8 control](https://www.jma.go.jp/jma/index.html) —`text/html; charset=UTF-8` | `気象庁 Japan Meteorological Agency` ;**0 / 0 U+FFFD** | Identical title; **0 / 0 U+FFFD** | \n\n| [@clawsweeper](https://github.com/clawsweeper) re-review | \n\n| 🦞👀 Exact review queued. Re-review progress:  | \n\n[clawsweeper](https://github.com/apps/clawsweeper)Bot added\n\n[clawsweeper:intake:bug](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug)\n\n[clawsweeper:intake:needs-environment](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment)\n\n[clawsweeper:intake:needs-repro](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-repro)\n\n[proof: sufficient](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22proof%3A%20sufficient%22)\n\n[rating: 🦞 diamond lobster](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22rating%3A%20%F0%9F%A6%9E%20diamond%20lobster%22)\n\n[status: 👀 ready for maintainer look](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22status%3A%20%F0%9F%91%80%20ready%20for%20maintainer%20look%22)\n\n[rating: 🦐 gold shrimp](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22rating%3A%20%F0%9F%A6%90%20gold%20shrimp%22)\n\n[status: 📣 needs proof](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3A%22status%3A%20%F0%9F%93%A3%20needs%20proof%22)\n\n[clawsweeper:intake:bug](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Abug)\n\n[clawsweeper:intake:needs-environment](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-environment)\n\n[clawsweeper:intake:needs-repro](https://github.com/openclaw/openclaw/issues?q=state%3Aopen%20label%3Aclawsweeper%3Aintake%3Aneeds-repro)\n\n[Oct 30, 2026](#event-34741429886)\n\n[github-actions](https://github.com/apps/github-actions)Bot pushed a commit to rishisathrughnarishisathrughna9432/openclaw that referenced this pull request\n\n[Oct 30, 2026](#ref-commit-cf63a88)\n\n[github-actions](https://github.com/apps/github-actions)Bot pushed a commit to Desicool/openclaw that referenced this pull request\n\n[Oct 31, 2026](#ref-commit-3208852)\n\n[Learn more about bidirectional Unicode characters](https://github.co/hiddenchars)", "url": "https://wpnews.pro/news/an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters", "canonical_source": "https://github.com/openclaw/openclaw/pull/188766", "published_at": "2026-10-31 09:05:23+00:00", "updated_at": "2026-10-31 09:18:32.295298+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["openclaw", "ClawSweeper", "Codex", "abehiroshi.la.coocan.jp", "Shift_JIS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters", "markdown": "https://wpnews.pro/news/an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters.md", "text": "https://wpnews.pro/news/an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters.txt", "jsonld": "https://wpnews.pro/news/an-ai-agent-read-a-shift-jis-page-as-208953-replacement-characters.jsonld"}}