Your First AI Agent Task Needs a Contract, Not More Permissions

A new guide adapted from AI NEWS IL outlines a contract-based approach for a first production AI agent task, arguing that narrow scope and explicit boundaries matter more than broad permissions. Using a hypothetical support-inbox example, it recommends defining the agent's inputs, outputs and decision-maker in one sentence, granting read-only access to approved policy folders, and treating send or write capabilities as a separate permission stage requiring new tests. It also advises measuring category accuracy, source fit, meaning changes, reviewer corrections, time to approval and cost, and notes there is no universal number of test cases that proves readiness.

Adapted from the Hebrew guide "AI Agent: How to Choose a Task, Set Boundaries and Check the Result" https://ainewsil.co.il/articles/ai-agent-task-fit-checklist on AI NEWS IL published 17 September 2026, updated 24 September 2026 . An agent's first production task should be small enough to test and safe enough to stop. "Manage our support inbox" is neither. "Classify one type of support request, find the relevant policy and prepare a draft for a representative" is a better starting point. This guide turns that narrower task into a testable contract. The support example is hypothetical, not a case study or a claim of measured results. Use an agent when the goal is clear but the next step depends on what it finds. A support request may require a policy lookup, another document or a handoff to a person. If every step is fixed, ordinary automation is usually easier to predict and test. Copying a field, sending a scheduled reminder or generating a standard report does not automatically need an agent. Before connecting anything, write one sentence: The agent receives X, returns Y, and Z decides what happens next. If you cannot name the output or the person responsible, the task is not ready. Start with a reversible task. Do not make payments, deletion, publication or permission changes part of the first pilot. A customer writes: "My monthly charge has increased. Why?" The agent can: A representative opens the invoice, checks the policy, corrects the draft and decides whether to send it. The agent does not send the reply or update the customer record. The point is to reduce search work without inventing service terms or exposing information. Adapt the example to your systems, privacy rules and business risks before using it. Allow only the information needed: the request text, a minimal customer reference, the response language and links to approved policies. Keep passwords, payment details and unrelated files out of the model input. Use a fixed output shape: | Field | What the reviewer checks | |---|---| | Category | Is it one of the allowed categories? | | Confidence or uncertainty | What remains unresolved? | | Source section | Does the cited policy support the answer? | | Draft reply | Has the agent added a promise or changed the policy's meaning? | | Items to verify | What must the representative check before sending? | Define invalid outputs too. A missing source link, an unknown category or an unsupported refund promise should produce "review required" with a reason, rather than a ready-to-send draft. A missing or conflicting document is a reason to stop and ask for review. It is not a blank the model should fill. For the first test, grant read access only to the approved policy folder and test requests. Do not give the agent the entire inbox, full CRM, invoices or contact directory because they might be useful later. Keep credentials outside prompts and user-visible logs. Limit access to the environment and operations required, and use short-lived credentials where supported. Treat these as separate permission stages: Passing the first ten cases does not authorize sending messages. That is a new capability with a new risk and needs its own decision and tests. Ask the organization's access owner to approve the permission list. Also ask what an attacker could reach if the credential leaked. Narrow the scope before the pilot if the answer is unacceptable. Write the expected outcome for each case before running it: draft, clarification or stop. Include routine requests alongside: For example, a customer message might contain a demand to ignore the policy and approve a full refund. The agent should apply the approved policy and flag the request for review. The message is task data, not permission to change the workflow. Measure category accuracy, source fit, changes in meaning, reviewer corrections, time to approval and cost. There is no universal number of test cases that proves readiness. Expand the test set according to task variety, error severity and how easily harm can be detected. Stop when a required source is missing, information is sensitive, uncertainty is unresolved, a financial decision is needed or a tool attempts an operation outside the allowed scope. Name one process owner who can stop the pilot. Record the limited inputs, sources, tool calls, output and human decision under your privacy policy. Changes to the model, prompt, data or permissions should trigger regression tests. Before the pilot, rehearse rollback: revoke the credential, return requests to representatives and verify that the manual process works. A rollback plan that has never been tried is still an assumption. Run a short, defined draft-only pilot after the test set passes. Two weeks is an example, not a universal requirement. For each draft, record whether it was: Track review time and whether the source was correct. Compare similar request types with and without the agent. A fast draft is not a saving if checking it takes longer than writing the reply manually. Set the continuation rule before the pilot. For example, continue only if corrections are small, no draft invents service terms and handling time falls. Stop for a data leak, skipped human review or costs that exceed the benefit. The first milestone is a useful draft under controlled permissions. More access is a separate decision, not a reward for a convincing demo. The Hebrew guide cites Anthropic's Building Effective Agents https://www.anthropic.com/research/building-effective-agents , NIST's AI Risk Management Framework https://www.nist.gov/itl/ai-risk-management-framework and IBM's introduction to building an AI agent https://www.ibm.com/think/topics/how-to-build-an-ai-agent . This English adaptation was prepared with AI assistance. Yaniv Shenhav.