Ergo Framework – Erlang/OTP-style distributed systems in Go

Ergo Framework 3.3.0 is about seeing what a running cluster is doing, and proving what it will do before it runs. A trace now follows a message across nodes, the Observer became a full cluster inspector that an AI agent can drive, and a testing framework ships with the framework itself. Saturn is MIT from this release on, so every module of the ecosystem is free for production use.

Highlights #

Distributed tracing. A trace context propagates across Send, Call, spawn and every hop a message takes between nodes, with configurable sampling and cross-node clock-skew correction. The new Pulse application exports the collected spans over OTLP to Tempo, Jaeger or anything else that speaks it, and ships a Grafana dashboard.

Testing framework. ergo.services/ergo/testing gives four layers on one fluent assertion grammar: unit runs a single actor against a mock node, stage runs real nodes over the real protocol, mock provides standalone gen.* mocks for code that consumes the interfaces, and check is the assertion core they share. You assert over messages, calls, spawns, links, monitors and events, positively and negatively.

Observer, and MCP for AI agents. The Observer application is now a full cluster inspector: node info with live memory graphs, the process list with mailbox depth, latency, running time and wakeups, per-process detail with supervision tree and live HandleInspect state, meta processes, applications, the network stack and its wire-type registry, events, a live log stream, and on-demand goroutine and heap profiles with no special build. The same application serves an MCP endpoint: 38 tools and 13 resource lenses over the live cluster, 26 of them read-only, with capability ceilings that can only narrow. One Observer inspects every node in the cluster, because every node already runs the built-in system application.

act.Router. A behavior that picks the destination from the message instead of taking the next worker in order. It can own its routes or resolve them through the node registry, the routing table is editable while it runs, and high-priority traffic bypasses routing so an admin path stays open while the routed path is busy.

EDF schema evolution. With NetworkFlags.EnableSchemaEvolution on both nodes, fields appended to the end of a registered struct stay wire-compatible: a peer that does not know them skips them, a peer missing them zero-fills. Off by default, so EDF stays strict unless you ask for it.

Grid. A new application giving a distributed registry, key lifecycle monitors and per-key process groups. Every node keeps a full local copy, so a lookup is a local read; writes are serialized per key by a shard actor and replicate in the background. AP and eventually consistent.

argus. A vet tool for the actor-model invariants the compiler cannot express: shared memory in a message, an unbounded wait in a callback, a goroutine reaching back into actor state, a HandleCall that can never reply, and thirty-one more. Runs as go vet -vettool=$(which argus) ./..., with a baseline mode for codebases that have never seen it.

Breaking changes #

Applications must embed app.Application. ApplicationBehavior gained lifecycle callbacks and Load lost its node gen.Node parameter:

// before
type MyApp struct{}
func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { ... }
func (a *MyApp) Start(mode gen.ApplicationMode) {}

// after
type MyApp struct{ app.Application }
func (a *MyApp) Load(args ...any) (gen.ApplicationSpec, error) { ... }

The embed supplies the framework plumbing and no-op defaults for the new callbacks. If you used node inside Load, switch to a.Node(). If your Start was an empty stub, delete it; if it had logic, its signature is now Start(ref gen.Ref, mode gen.ApplicationMode). Step-by-step in the Application documentation.

Also #

• gen.ProcessBehavior requiresProcessKind() gen.ProcessKind . The base behaviors (act.Actor ,act.Supervisor ,act.Pool ,act.Router ,act.WebWorker ) report their own, so only hand-written behaviors are affected.
• The trace build tag is nowverbose : go run --tags verbose ./cmd.

Also in this release #

• Network . Automaticfragmentation of large messages, application-levelkeepalive that catches the silent failures TCP keepalive cannot, a handshake deadline, and a fix for the dead loop two nodes could enter when dialing each other at the same time.
• Observability without s.Node.Info() and the heap inspector no longer stop the world; memory and GC figures come from runtime/metrics. NewNodeShortInfo /RemoteNodeShortInfo /ProcessShortInfo are the cheap counterparts for polling a whole cluster, alongside per-event metrics, process lifecycle counters, per-process running time, wakeups, init time, and mailbox latency under-tags=latency .
• Application teardown. An application'sTerminate now runs after every one of its processes has terminated, its ownTerminate included, so closing a shared resource there is safe. Detached processes are stopped with the application instead of outliving it.
• Errors that keep their causes.gen.Errorf collects %w arguments so errors.Is anderrors.As still find them, and the chainsurvives the wire when both nodes enable NetworkFlags.EnableWrappedErrors.
• Radar ,Health ,Sentry . One application serving Kubernetes probes and a Prometheus endpoint on one port; the Health actor behind it; and a logger that forwards framework panics and errors to Sentry with stack traces from the panic site.
• Metrics actor expanded well beyond node/network telemetry, with a complete Grafana cluster dashboard.
• Target Manager rewritten lock-free. One lock used to serialize every link, monitor and event subscription on the node; a node now carries hundreds of thousands of them without contention
• Saturn moved from Business Source License 1.1 to MIT. No node cap, no licence to buy.

Documentation #

New articles on docs.ergo.services:

• FAQ - the questions that come up before the first node starts
• AI Agents - the skills and agents shipped for working on an Ergo codebase
• Router -act.Router , its routing modes and runtime management
• Distributed Tracing - samplers, business spans and exporters
• Inspecting Actor State -HandleInspect and reading a live process
• Inspecting With Observer - the cluster inspector, panel by panel
• Inspecting With an AI Agent - the MCP surface, its lenses and capability ceilings
• Actor Model Vet Tool -argus , its rules and how to adopt it on existing code

Testing:

• Overview - the four layers and which one answers which question
• Check - the assertion grammar shared by every layer
• Mock - standalonegen.* mocks for code that consumes the interfaces

Extra library:

• Grid - distributed registry, key monitors and process groups
• Pulse - OTLP exporter for the collected spans
• Radar - health probes and Prometheus metrics on one port
• Health actor - Kubernetes liveness, readiness and startup probes
• Sentry logger - forwards panics and errors to a Sentry project

Full list of changes: CHANGELOG.md