Government open data looks deceptively easy from a distance.
Find an endpoint. Make a request. Normalize the JSON. Ship an API.
That is enough for a demo. It is not enough for a production system that has to answer a consequential question next month, after the publisher changes a field, caps a response, moves an endpoint, or returns an empty page that looks exactly like “there are no records.”
We have been building CivicDataForge around one question:
What has to exist between an official government publisher and a software system—or an AI agent—before the result deserves to be called evidence?
This post is the practical answer we have reached so far. It is not a claim that government data can be made perfectly clean. It is an account of the failure modes we hit, the architecture we adopted, and the boundaries we refuse to erase.
The first serious failure mode is pagination.
Government-data platforms often impose a maximum record count. A response can be valid JSON, have a 200
status, and still represent only the first page.
ArcGIS feature services expose controls such as resultOffset
and resultRecordCount
, and may signal that a transfer limit was exceeded. Socrata datasets similarly support paging and query controls. If a collector ignores those contracts, “the source returned 1,000 rows” can quietly become “the jurisdiction has 1,000 records.”
Those are not the same statement.
Our collection rule became:
The important product decision is step four. A partial page is useful for exploration, but it cannot prove that a record disappeared between two runs.
Official references:
Suppose an address lookup returns no permit record.
It is tempting to return NOT_PERMITTED
or ILLEGAL
. That is usually stronger than the evidence.
The selected publisher may not cover the full legal regime. The address may be formatted differently. A local permit may exist in another system. The source may be stale. The query may be incomplete. Or there may simply be no published match.
We use an explicit, fail-closed decision vocabulary:
EVIDENCE_FOUND
NO_PUBLISHED_MATCH
REVIEW_REQUIRED
SOURCE_UNAVAILABLE
SCOPE_INCOMPLETE
This vocabulary is deliberately less dramatic than a binary yes/no result. That is the point.
An API should not manufacture certainty because a consumer wants a Boolean. If a downstream workflow requires a Boolean, the customer should own the policy that maps evidence states to an action—and should preserve the original state for review.
Normalization is necessary. It is also where evidence can lose its identity.
We normalize names, addresses, dates, identifiers, and source-specific statuses, but retain enough lineage to answer:
A minimal evidence envelope looks roughly like this:
{
"source": {
"publisher": "issuing authority",
"url": "https://official.example/dataset",
"retrieved_at": "2026-08-24T12:00:00Z"
},
"scope": {
"query": "bounded source query",
"complete": true
},
"record": {
"source_id": "publisher-owned-key",
"normalized_identifier": "stable-identifier",
"record_hash": "sha256:..."
},
"decision": {
"state": "EVIDENCE_FOUND",
"receipt_hash": "sha256:..."
}
}
The exact fields vary by source. The invariant is that the normalized result does not sever the path back to the official record.
A source can be reachable and stale.
It can have a stable schema and return incomplete data. It can publish a recent file whose internal dates are old. It can change content without changing a URL. It can change a schema without failing a request.
So one green check is not enough.
We monitor independent dimensions:
When freshness cannot be measured, we say freshness_not_measured
. We do not let an availability check imply that the data is current.
This distinction matters because a production monitor should be able to quarantine a bad update instead of replacing a known baseline with an empty or damaged response.
“Send a webhook when something changes” is only the visible edge of a reliable change system.
The harder questions are:
Apify webhooks retry failed deliveries. That is helpful, but it means consumers must be idempotent. Our recommended key is the Actor run ID plus event type, with the downstream handler acknowledging quickly and queueing the expensive work.
idempotency_key = actor_run_id + ":" + event_type
We treat a webhook as a notification about an evidence object, not as the evidence object itself. The durable dataset, record hashes, scope, and decision receipt remain the inspectable source of truth.
Official reference: Apify webhook actions and retry behavior.
Publishing an MCP server does not automatically make a tool safe or useful.
An agent still needs to know:
We expose a stable Streamable HTTP MCP endpoint and machine-readable catalogs, but the more important artifact is the task-to-tool selection contract.
For example:
{
"task": "verify India company evidence",
"preferred_identifier": "Corporate Identification Number",
"minimum_input": "exact CIN preferred",
"boundary": "research evidence, not automatic KYC or eligibility clearance"
}
The transport gets the tool into the agent. The selection contract helps the agent call the right tool and preserve the result's meaning.
The MCP specification recommends Streamable HTTP for remote servers and defines the transport behavior. Authentication, origin validation, and secret handling still belong to the implementation and client configuration—not to public examples with embedded credentials.
Official reference: MCP transports specification.
The architecture now has six explicit services:
That framing changes what we build.
A new source does not enter production because we found an endpoint. It needs a source and rights passport, a completeness test, a normalization contract, health probes, a bounded decision contract, and an acceptance-tested delivery path.
Likewise, a connector is not “available” because we can imagine it. Public APIs, datasets, schedules, webhooks, file exports, and the remote MCP endpoint are live interfaces. A customer-specific adapter, SFTP route, or platform-native app remains scoped work until its real downstream path passes an acceptance test.
The service catalog and integration recipes publish that boundary in human- and machine-readable form.
CivicDataForge exists to reduce the hours lost between “the government publishes this somewhere” and “a person or system can use it without pretending it says more than it does.”
Our intention is to build a reusable evidence layer across public records: official-source first, inspectable, correction-friendly, and callable by both software and AI agents.
Our want is straightforward: make source quality and uncertainty visible enough that developers do not have to choose between a brittle one-off script and an opaque vendor answer.
We are open to collaboration and grateful for serious support, correction, and feedback. We are not asking for permission to build; we are inviting developers, public servants, data owners, researchers, and AI teams to make the contracts stronger with us.
What we need most from the developer community is adversarial input:
Those questions are more valuable than generic feature requests because they expose the real contract the system has to survive. If you bring one, we will meet it with respect, evidence, and a willingness to change what the facts prove should change.
If we started again, we would make three decisions earlier:
The code that fetches rows is often the smallest part of the system. The defensible work is keeping a source-linked statement truthful after paging, normalization, retries, source drift, and downstream automation.
That is the layer we are building.