A finance agent can produce a convincing answer and still be wrong in the one way that matters: it can make a decision without enough evidence.
That is why “add a better system prompt” is not a sufficient quality strategy for accounting automation. Prompts help define behavior, but they do not tell you whether a release reliably classifies transactions, preserves source evidence, routes ambiguity, and refuses to overreach.
The practical answer is an evaluation harness: a small, repeatable test system that runs realistic finance cases through an agent and scores the result against explicit expectations.
Before choosing an LLM or tool framework, define what the agent is allowed to decide.
For a transaction-review agent, a useful contract might look like this:
{
"action": "suggest_category",
"confidence": 0.91,
"evidence_ids": ["bank_tx_1842", "receipt_883"],
"needs_review": false,
"reason": "Recurring software subscription matches the approved vendor pattern."
}
The important fields are not only action and confidence. An evaluation harness should also check:
A response that says “software expense” with no evidence may look useful in a demo. In production, it is an unreviewable assertion.
Do not evaluate only clean examples. Clean examples measure whether the happy path works. They do not measure whether the agent knows when to stop.
A compact starter dataset can contain 50 to 100 cases divided into four groups:
Store the expected behavior as structured data, not just a paragraph written by a reviewer:
{
"case_id": "tx_missing_receipt_07",
"expected_action": "request_evidence",
"allowed_categories": [],
"must_cite": [],
"must_not": ["post_journal_entry", "mark_reconciled"]
}
This makes a regression test possible. If a prompt or model change causes the agent to classify this case anyway, the build should show a failure immediately.
A single “answer quality” score hides too much. Score the workflow in layers instead.
Schema validity checks whether the output can be parsed and whether required fields are present. This catches malformed tool calls and half-finished responses.
Evidence grounding checks that every cited document or transaction exists and supports the claim. You can start with deterministic checks: valid IDs, matching account IDs, date-window constraints, and amount consistency.
Policy compliance checks authority boundaries. A suggestion may be acceptable while an automatic journal entry is not. The evaluator should distinguish “propose,” “request review,” and “execute.”
Decision quality checks the actual accounting outcome against a reviewed label. Keep this separate from writing quality. A beautifully explained wrong category is still wrong.
Operational behavior checks latency, tool-call count, duplicate handling, and retry safety. A model that reaches the correct answer by creating three duplicate reconciliation records is not production-ready.
Finance agents rarely operate as chat-only systems. They read transactions, fetch documents, call categorization services, and write review tasks. Your harness should record each tool call as an event:
run_id=eval-2026-09-21-014
case_id=tx_duplicate_webhook_03
step=2 tool=lookup_transaction args_hash=...
step=3 tool=create_review_task result=task_551
step=4 tool=create_review_task result=already_exists
final_action=request_review
That trace lets you test properties that are difficult to see in the final answer:
This is where conventional software testing and LLM evaluation meet. The language model may be probabilistic, but the side effects around it should be bounded and observable.
Run the curated case set on every prompt, model, tool-schema, and policy change. Keep a versioned baseline so you can compare releases instead of relying on memory.
Then add production sampling. Select a small, privacy-aware sample of completed reviews, redact sensitive values where possible, and send them through the same evaluator. Track failures by category: unsupported claim, missing citation, incorrect escalation, duplicate write, and policy violation.
A useful release gate might require:
The exact thresholds depend on the workflow. The principle is stable: define the failure that would hurt, then make it measurable.
The goal is not to eliminate every review. It is to ensure that the agent sends the right work to review with enough context to resolve it quickly.
A good review item includes the source transaction, proposed action, cited evidence, detected uncertainty, and a clear reason for escalation. Tools such as Portali can fit naturally into this pattern when the accounting workflow needs a shared place for source-linked decisions rather than another opaque automation layer. The product matters less than the design principle: every automated suggestion should remain inspectable.
An evaluation harness turns that principle into an engineering practice. Prompts still matter, models still matter, and good UX still matters. But the durable quality signal comes from repeatable cases, explicit authority boundaries, evidence checks, and side-effect tests.
If your finance agent cannot pass those tests, it does not need a more persuasive demo. It needs a better harness.