An AI agent read a Shift_JIS page as 208,953 replacement characters

wpnews.pro

Conversation #

| 🦞👀 ClawSweeper picked this up. Pull request received. I will update this pull request when review starts. ClawSweeper review complete ClawSweeper finished reviewing this revision. The review result is being finalized. View the workflow run . |

clawsweeperBot added

clawsweeper:intake:bug

clawsweeper:intake:needs-environment

P2

rating: 🦐 gold shrimp

status: 📣 needs proof

clawsweeper:intake:bug

clawsweeper:intake:needs-environment

Oct 30, 2026

| Codex review: needs maintainer review before merge. What this changes This PR makes link understanding reuse the existing charset-aware web response decoder. Example: An agent reads the Shift_JIS page athttp://abehiroshi.la.coocan.jp/ . Review scores Product Kind: Bug fix ·Worth it: Yes ·Fix scope: Complete User problem: Agents receive unreadable text from linked pages declaring supported legacy encodings. Reason: The repair restores readable page content through an established decoder without adding configuration, dependencies, or a competing policy. Merge readiness ✅ Ready for maintainer review This PR fixes a current decoding bug with a minimal shared-owner change; the earlier finding is resolved, no new defects were found, and CI is still running. Priority: P2 Reviewed head:73ec930363d6b9a218a091b5f0c31f57101b6ea8 Before merge None. Findings None. Agent review details |

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	A focused shared-owner repair with relevant shipped Gateway evidence, reported red/green regression results, and no open findings.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The supplied Node 24 secretless Docker Gateway transcript exercises sessions.create and chat.send through guarded public HTTP, the real CLI processor, agent runtime, model request, final reply, and history; Shift_JIS titles become readable while UTF-8 controls remain unchanged, and the proven candidate has the reviewed head's identical tree. The deterministic model endpoint proves text propagation rather than semantic model quality.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

How this fits together

Link understanding fetches message URLs through the guarded HTTP transport, decodes bounded response bytes, and supplies text to configured CLI processors and agent input.

flowchart LR
  A[Message URL] --> B[Guarded HTTP fetch]
  B --> C[Bounded response bytes]
  C --> D[Shared charset decoder]
  D --> E[CLI link processor]
  E --> F[Agent model input]
  F --> G[Assistant reply]

Technical review

Best possible solution:

Merge the shared-decoder repair once the existing CI gates pass.

Do we have a high-confidence way to reproduce the issue?

Current-main source establishes unconditional UTF-8 decoding, and the supplied Gateway comparison demonstrates the resulting garbled title before the repair.

Is this the best way to solve the issue?

Reusing the existing decoder is the smallest complete fix: it removes the divergent decoding decision while preserving transport limits, cancellation, and permissive fallback.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against e279b64ffe86.

Provenance checked

Link response decoding changes intended behavior with a stated reason (5fe1414 : This refactor carried existing link processing into its application owner while preserving behavior; it does not establish when unconditional UTF-8 decoding originated.)
Shared decoder export keeps the original intent (d185f9a : The existing decoder preserves charset handling and safe truncated-prefix decoding; the PR exports it internally without changing its implementation.)
Transport regression consolidation keeps the original intent (4ec56e7 : Prior test consolidation retained distinct transport and lifecycle scenarios; the revised fixture adds charset coverage while retaining URL selection and ordering assertions.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

Verified introduced change: The introduced diff exports the existing decoder, replaces the link fetcher's unconditional UTF-8 decoding, and incorporates a Shift_JIS regression into existing transport coverage. (src/link-understanding/apply.ts:48 ,73ec930363d6 )
Current main still contains the defect: Current main decodes fetched link bytes with new TextDecoder().decode(buffer), ignoring declared legacy encodings before passing content to processors. (src/link-understanding/apply.ts:47 ,e279b64ffe86 )
Existing decoding contract: The shared decoder preserves BOM precedence, header and document charset detection, and nonfatal UTF-8 fallback for unsupported labels; existing tests cover these contracts. (src/agents/tools/web-shared.ts:170 ,73ec930363d6 )
Shipped Gateway evidence: obviyus supplied before/after terminal output infix(link-understanding): non-UTF-8 pages reach the model as garbled text #188766 (comment) : isolated built Gateways fetched public pages through the normal guard, ran a real CLI processor, and carried corrected titles through actual model requests, replies, and history. The deterministic model fixture echoes the supplied title without decoding or repairing it; UTF-8 controls remain unchanged. (73ec930363d6 )
Proof applies to the current tree: The tested candidate and reviewed head both resolve to tree 6b9d2585e88508e4f86e5d5c5f78834cf54f390e, verifying the recorded proof's unchanged source inputs. (73ec930363d6 )
Earlier finding resolved: The separate regression table was removed and its meaningful charset assertion consolidated into the existing transport case, eliminating the test growth that prompted the earlier allocation finding. (src/link-understanding/runner.transport.test.ts:123 ,73ec930363d6 )

PR surface

Source +1, Tests 0. Total +1 across 3 files. Added test files: 0.

View PR surface stats #

Area	Files	Added	Removed	Net
Source	2	3	2	+1
Tests	1	5	5	0
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	8	7	+1

Labels

Label changes:

add proof: sufficient : Contributor real behavior proof is sufficient.
add rating: 🦞 diamond lobster : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
add status: 👀 ready for maintainer look : ClawSweeper has no concrete contributor-facing blocker left for this PR.
remove rating: 🦐 gold shrimp : Current PR rating israting: 🦞 diamond lobster , so this older rating label is no longer current.
remove status: 📣 needs proof : Current PR status label isstatus: 👀 ready for maintainer look .

Label justifications:

P2 : Declared legacy-charset pages produce incorrect agent input, with impact limited to configured link-understanding flows.
rating: 🦞 diamond lobster : Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look : ClawSweeper has no concrete contributor-facing blocker left for this PR.
proof: sufficient : Contributor real behavior proof is sufficient.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle) #

reviewed 2026-10-30T22:33:32.625Z sha 38de2ab :: needs real behavior proof before merge. :: [P2] Allocate the added regression-test lines

Reviewed October 30, 2026, 7:41 PM ET / 23:41 UTC (Revision 2).

| Correctness gate and shipped Gateway proof This is not performance-only : the same public page reaches both the actual model request and the final assistant reply with the wrong characters on main, and readable Japanese after the repair. The build and test proof ran at 86a4b241c744b6e2de9f70a93fbe23c6a70a692f . Before publication, the guarded push detected the contributor's new allocation commit3cf4ae615b2f ; I merged it intact and removed that now-unused 13-line allocation because the consolidated tests have zero growth. The resulting final source tree isidentical to the proven tree: both Git tree IDs are6b9d2585e88508e4f86e5d5c5f78834cf54f390e . No behavior, source, test, dependency or build input changed; only commit ancestry changed, so the recorded proof is reused rather than rebuilt. Shipped route: operator WebSocketsessions.create →chat.send → normal OpenClaw agent preprocessing → STRICT guarded public HTTP fetch → real configured CLI link processor → real OpenClaw agent runtime → OpenAI-compatible model fixture →agent.wait terminal reply andchat.history . The CLI processor reads the real fetched document from stdin, records its decoded title and replacement-character count, and outputs only LINK_TITLE=<JSON title> . The deterministic model fixture records the actual model request and echoes that supplied title without repairing or decoding it. No product modules, fetch guard, decoder, dispatch, runtime or history reader are mocked. The public HTTP fetches use real DNS/TLS/redirect handling; no SSRF exception or network preload is installed. This proves the production agent/modelpath , not a paid live-model service or semantic summarization quality. The same titles were observed in each actual model request and in the final assistant message. Example terminal replies:

main:      Observed link title: �������̃z�[���y�[�W
candidate: Observed link title: 阿部寛のホームページ
control:   Observed link title: 気象庁 Japan Meteorological Agency

All 8 agent turns completed with exactly1 model request per turn , requested/effective provider and model unchanged,rerouted: false , and native receipts reportingsourceReplyDelivered: true /terminalDisposition: visible . Both the terminal reply and complete native history were checked; a missing link result fails the harness rather than masquerading as a passing request. Replay commands inside the isolated container:

pnpm install --frozen-lockfile
pnpm build
node /proof/agent-proof.mjs base
node /proof/agent-proof.mjs candidate
CI=1 OPENCLAW_VITEST_RUNTIME=node OPENCLAW_VITEST_MAX_WORKERS=1 \
  pnpm test src/link-understanding/runner.transport.test.ts \
  src/agents/tools/web-shared.test.ts \
  src/agents/agent-tools.prefix-stability.test.ts --maxWorkers=1

Focused candidate results: 33 passed — transport5 , shared response decoder14 , prompt-prefix stability14 . The same final transport test file on main gives4 passed / 1 failed at the decoding assertion: expected日本語 , received��{�� ; URL selection/order assertions still pass. Scoped type-aware lint, formatting and whitespace checks pass. The extra 13-line table was removed and its charset regression folded into the existing URL-selection/transport case. Missing-label and unsupported-label fallback, BOM precedence, document metadata and response isolation retain the existing shared decoder coverage. Final production delta +1 , tests0 ; no test allocation record or ceiling change. Retained raw evidence/harness: proofs/188766/{agent-proof.mjs,base-agent-proof.json,candidate-agent-proof.json,base-processor.jsonl,candidate-processor.jsonl,base-model-requests.jsonl,candidate-model-requests.jsonl,candidate-focused-tests.log,base-regression.log,verification-results.json} . Only task-owned synthetic provider credentials were used; public page titles are the only quoted page content. |

Input	Main round 1 / 2	Candidate round 1 / 2
Public Shift_JIS page — HTTP redirects to HTTPS;`Content-Type: text/html` , charset declared in the document	`��̃z�[��y�[�W` ;13 / 13 U+FFFD in processor input	`阿部寛のホームページ` ;0 / 0 U+FFFD
UTF-8 control —`text/html; charset=UTF-8`	`気象庁 Japan Meteorological Agency` ;0 / 0 U+FFFD	Identical title; 0 / 0 U+FFFD