{"slug": "synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris", "title": "SynapseHealer: Building an Autonomous, Self-Healing Interoperability DLQ with InterSystems IRIS AI Hub", "summary": "A developer built SynapseHealer, an autonomous dead-letter-queue repair pipeline on InterSystems IRIS AI Hub that uses a governed %AI.Agent with three fixed tool calls — vector runbook search, payload validation and risk scoring — to auto-heal, escalate or quarantine failed healthcare and supply-chain integration messages. Across 10 regression suites the system reported 479 prompt and 176 completion tokens, about $0.00018 per heal, roughly 145 ms end to end and a 90% auto-heal rate on seven canonical chaos scenarios, with every inference logged to an audit telemetry table and surfaced in a Next.js dashboard.", "body_md": "Built for the InterSystems Programming Contest: AI Agents in Interoperability Workflows (2026)\n\nIn healthcare and supply-chain integrations, the dead-letter queue is where transactions go to die. A vendor renames `client_uuid` to `customer_id` with no warning. An EHR bridge emits `1985-02-31`, an impossible calendar day. An HL7 `ADT^A01` arrives with `PV1` before `PID`. A lab note contains an unescaped pipe. The integration engine rejects the message, and it sits in the queue until an engineer opens a hex dump, decodes the schema violation, writes a one-off script, and resubmits — days later, if at all.\n\nSynapseHealer replaces that manual bottleneck with an autonomous pipeline on InterSystems IRIS AI Hub: a custom inbound adapter, native vector search over repair runbooks, a governed `%AI.Agent`, and a tri-branch router that auto-heals, escalates to the InterSystems Workflow Engine, or quarantines.\n\n`Synapse.Process.DeadLetterHealerBP` is the arbitrator. `OnRequest` takes a `Synapse.Msg.DeadLetterRequest` from `ChaosIngestService`, and `ProcessDeadLetter` picks the execution mode: `live` when `SYNAPSE_MODE=live` and an API key exists, otherwise `replay` — a deterministic mode that needs zero API keys.\n\nIn live mode it instantiates `Synapse.Agent.DeadLetterHealerAgent`, an `%AI.Agent` bound to `Synapse.Agent.HealerToolSet`, sends it the payload ID, source system, format, validation error, and raw body, strips any markdown fences, and parses the JSON contract: `decisionBranch`, `confidenceScore`, `riskLevel`, `matchedPatternId`, `healedPayload`, `reasoning`.\n\nThe agent cannot improvise. Its toolset — `Synapse.Agent.HealerToolSet` (`%AI.ToolSet`) wrapping `Synapse.Agent.HealerTools` (`%AI.Tool`) — exposes exactly three governed calls: `SearchRunbookRAG`, `ValidateRepairedPayload`, and `CalculateRiskScore`. It looks up the nearest verified runbook in the 384-dimensional vector index (`VECTOR_COSINE`, sub-45ms), applies the deterministic transformation, and must pass validation before anything is resubmitted.\n\nCredentials never appear in plaintext logs. `Synapse.Config.AIHubSetup.StoreWalletSecret` writes the API key into the IRIS Wallet (`%Wallet.KeyValue`, collection `SynapseSecrets`, key `SynapseSecrets.LLMKey`) at startup, and `BuildProviderSettings` references it through `@{env:LLM_API_KEY}` indirection instead of embedding the literal key. The same setup registers `%ConfigStore` and `Ens.Config.Credentials` when those classes exist, with graceful fallback on builds that lack them.\n\nWe default to `gemini-1.5-flash` (`gpt-4o-mini` is the OpenAI alternative). Heal decisions are structural, not creative, so a fast, cheap model is the right choice.\n\nEvery inference lands in `Synapse.Data.AuditTelemetry`, a persistent table storing prompt tokens, completion tokens, estimated USD cost, latency, the full tool-call trace, decision branch, risk level, and workflow status. Cost is computed from the counts — `prompt × 0.00000015 + completion × 0.0000006` — and written to the table in real time; the Next.js dashboard reads `/api/synapse/status` and shows the cumulative USD counter and auto-heal rate live.\n\nAcross our 10 regression suites: 479 prompt tokens, 176 completion tokens, $0.00018 per heal, ~145 ms end to end, 90% auto-heal rate across the seven canonical chaos scenarios. Replay mode consumes no tokens, and the telemetry table is pre-seeded with golden traces for every scenario, so the dashboard is populated from the first second.\n\nEach heal produces a governed three-step trace, stored in `ToolTraceJSON`:\n\n`SearchRunbookRAG` — error and format in, matched runbook and similarity out (~42 ms)`ValidateRepairedPayload` — strict HL7 v2 / JSON syntax and schema check (~56 ms)`CalculateRiskScore` — clinical and financial risk classification (~35 ms)\nThe trace is surfaced live in the Next.js console: a step gauge per call, side-by-side diff of the corrupted and healed payloads, and the agent's reasoning. `HUMAN_WORKFLOW` messages land in the workflow queue where an operator Approves (→ `ResubmitDeliveryOp`) or Rejects (→ `QuarantineVaultOp`) with reviewer notes. Everything is queryable via `/api/synapse/workflow/tasks` and the persistent audit ledger.\n\nWe separated development-time agentic tooling from runtime governance. During development, we paired our IDE with the `iris-agentic-dev` MCP toolkit (`iris_symbols_local` for local AST verification across 150+ symbols and `objectscript-guardrails` to enforce method-scoped `Quit` rules and `$IsObject` checks before container compilation). At runtime inside the container, governance is 100% native to IRIS AI Hub—powered by an `%AI.ToolSet` of `%AI.Tool` methods rather than an external sidecar.\n\nThe ObjectScript guardrails are the part I trust most:\n\n`ProcedureBlock`, and the business process checks `%Dictionary.CompiledClass.%ExistsId` before instantiating the agent, so a missing compile fails closed instead of guessing.`MSH`, put `PV1` before `PID`, use non-` YYYYMMDD` dates in `PID-7`, leave unescaped pipes in `OBX`, or retain legacy keys like `client_uuid` after a v2 migration. `<script`, `eval(`, `union select`, `drop table`, `--`, or `javascript:`, and escalates impossible dates like February 31 to human review.`<DIRECTORY>` bug the hard way: `Config.Databases.Create` registers the database in `iris.cpf` but does not format the physical `IRIS.DAT` file, crashing auto-start on restart. The fix: a declarative `merge.cpf` (`[Actions]`: `CreateDatabase`, `CreateNamespace`, `Interop=1`) plus an `iris.script` fallback that initializes `SYS.Database` when absent.\nFor %AI syntax, the agent resolves its provider via `AIHubSetup.GetProviderConfig` and `BuildProviderSettings`, creates `%AI.Provider`, calls `agent.CreateSession()` then `agent.Chat(session, prompt)`, and parses the result with `%DynamicObject.%FromJSON`. It compiled cleanly on the IRIS 2027.1 AI Hub EAP build.\n\nPackaging is automated: `module.xml` declares `<SystemRequirements Version=\">=2026.3\" Interoperability=\"enabled\"/>`, and `SetupAll` — run by IPM on install — executes `Configure()` (Wallet, config store, credentials), seeds the runbooks and replay telemetry, then starts `Synapse.Production`. `zpm install synapse-healer` or `zpm load /opt/irisapp -v` gives you a running production with a seeded dashboard. The module shipped with +11 contest bonus points across the custom adapter, embedded Python, vector search, workflow engine, and IPM integration.\n\n```\n# Prerequisite: Load the InterSystems AI Hub EAP tarball (Build 109 or 154)\n# docker image load -i iris-community-2027.1.0AI.109.0-docker.tar.gz\n\ngit clone https://github.com/josephjbassey/synapse-healer.git\ncd synapse-healer\ndocker compose up -d\ncd ui && npm install && npm run dev\n```\n\nOpen `http://localhost:3000`, press 1–7 to inject chaos payloads, and watch the pipeline heal them. No API key required; set `SYNAPSE_MODE=live` with an `LLM_API_KEY` for live AI Hub inference.\n\nIt's not magic. It's governed tool calls, deterministic validation, and a telemetry table that actually gets read.", "url": "https://wpnews.pro/news/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris", "canonical_source": "https://dev.to/josephjbassey/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-intersystems-iris-ai-2hjj", "published_at": "2026-11-03 22:36:09+00:00", "updated_at": "2026-11-03 22:52:30.697208+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "large-language-models", "mlops"], "entities": ["InterSystems IRIS AI Hub", "SynapseHealer", "InterSystems", "Gemini 1.5 Flash", "GPT-4o-mini", "Next.js"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris", "markdown": "https://wpnews.pro/news/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris.md", "text": "https://wpnews.pro/news/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris.txt", "jsonld": "https://wpnews.pro/news/synapsehealer-building-an-autonomous-self-healing-interoperability-dlq-with-iris.jsonld"}}