Method not allowed
\")\n\n body = event.get(\"body\") or \"\"\n if event.get(\"isBase64Encoded\"):\n body = base64.b64decode(body).decode(\"utf-8\")\n form = urllib.parse.parse_qs(body)\n token = form.get(\"token\", [\"\"])[0]\n decision = form.get(\"decision\", [\"\"])[0]\n\n if not token or decision not in (\"approved\", \"rejected\"):\n return page(400, \"Bad request
\")\n\n try:\n sfn.send_task_success(taskToken=token, output=json.dumps({\"decision\": decision}))\n except (sfn.exceptions.TaskTimedOut, sfn.exceptions.TaskDoesNotExist, sfn.exceptions.InvalidToken):\n return page(410, \"This approval request has expired or was already answered.
\")\n\n print(json.dumps({\"event\": \"approval\", \"decision\": decision}))\n return page(200, f\"Recorded: {decision}
\")\n```\n\nThe task token works as a bearer credential. Anyone holding it can answer the request, so treat the email like a password reset link. For a real deployment, put the HTTP API behind an authorizer, or replace the email click with a Slack interactive message that verifies the signer. The sandbox version here is enough to prove the pattern. If you later host the page on a Lambda Web Function instead of API Gateway, our breakdown of [Lambda Web Functions pricing](https://shattered.io/lambda-web-functions-pricing-active-cpu-billing-math-2026/) shows how active-CPU billing changes the math.\n\n**Step 7: Write the finalize function.** When the loop ends, one last Lambda stores the outcome. It writes a summary item with a time-to-live attribute so old runs expire on their own, and it returns a trimmed version for the completion event. Keep the event small. Consumers downstream should look up the full record by run ID.\n\n``` python\nimport json\nimport os\nimport time\n\nimport boto3\n\ntable = boto3.resource(\"dynamodb\").Table(os.environ[\"TABLE_NAME\"])\nRETENTION_SECONDS = 90 * 86400\n\ndef handler(event, context):\n final = event[\"final\"]\n text = final.get(\"final_text\", \"\")\n table.put_item(Item={\n \"pk\": event[\"run_id\"],\n \"sk\": \"summary\",\n \"alert\": json.dumps(event[\"alert\"])[:4000],\n \"summary\": text[:4000],\n \"iterations\": final[\"iteration\"],\n \"ttl\": int(time.time()) + RETENTION_SECONDS,\n })\n print(json.dumps({\"event\": \"finalize\", \"run_id\": event[\"run_id\"], \"iterations\": final[\"iteration\"]}))\n return {\"run_id\": event[\"run_id\"], \"summary\": text[:1500], \"iterations\": final[\"iteration\"]}\n```\n\n## Step 8: The Step Functions State Machine\n\nThis file is the heart of the build. It reads like a flowchart, and that is the point. You can open the execution in the console and watch the agent move from Decide to Route to RunTool and back again, with the input and output of each state one click away.\n\nThe flow runs as follows. `Init` reshapes the EventBridge event into the working state: a run ID taken from the execution name, the alert detail, an empty transcript and a zero counter. `Decide` calls the Lambda with retries tuned for Bedrock throttling. `Route` sends final answers to `Finalize`, risky tool calls to `RequestApproval`, and everything else to a pass-through state that stamps the approval as not required. `RunTool` executes. `Carry` rebuilds the state for the next loop, which also keeps stale fields from piling up.\n\n```\nComment: Ops triage agent loop with a human approval gate\nStartAt: Init\nStates:\n Init:\n Type: Pass\n Parameters:\n run_id.$: $$.Execution.Name\n alert.$: $.detail\n messages: []\n iteration: 0\n Next: Decide\n\n Decide:\n Type: Task\n Resource: arn:aws:states:::lambda:invoke\n Parameters:\n FunctionName: ${DecideFunctionArn}\n Payload.$: $\n ResultSelector:\n decision.$: $.Payload\n ResultPath: $.step\n Retry:\n - ErrorEquals: [BedrockThrottled]\n IntervalSeconds: 2\n BackoffRate: 2.0\n MaxAttempts: 5\n JitterStrategy: FULL\n - ErrorEquals: [Lambda.ServiceException, Lambda.AWSLambdaException, Lambda.SdkClientException, Lambda.TooManyRequestsException]\n IntervalSeconds: 2\n BackoffRate: 2.0\n MaxAttempts: 3\n Catch:\n - ErrorEquals: [States.ALL]\n ResultPath: $.error\n Next: FailRun\n Next: Route\n\n Route:\n Type: Choice\n Choices:\n - Variable: $.step.decision.action\n StringEquals: final\n Next: Finalize\n - Variable: $.step.decision.needs_approval\n BooleanEquals: true\n Next: RequestApproval\n Default: SkipApproval\n\n SkipApproval:\n Type: Pass\n Result:\n decision: not_required\n ResultPath: $.approval\n Next: RunTool\n\n RequestApproval:\n Type: Task\n Resource: arn:aws:states:::lambda:invoke.waitForTaskToken\n TimeoutSeconds: 3600\n Parameters:\n FunctionName: ${RequestApprovalFunctionArn}\n Payload:\n token.$: $$.Task.Token\n run_id.$: $.run_id\n alert.$: $.alert\n tools.$: $.step.decision.tools\n ResultPath: $.approval\n Catch:\n - ErrorEquals: [States.Timeout]\n ResultPath: $.approval\n Next: ApprovalTimedOut\n - ErrorEquals: [States.ALL]\n ResultPath: $.error\n Next: FailRun\n Next: RunTool\n\n ApprovalTimedOut:\n Type: Pass\n Result:\n decision: rejected\n reason: timeout\n ResultPath: $.approval\n Next: RunTool\n\n RunTool:\n Type: Task\n Resource: arn:aws:states:::lambda:invoke\n Parameters:\n FunctionName: ${RunToolFunctionArn}\n Payload:\n run_id.$: $.run_id\n messages.$: $.step.decision.messages\n tools.$: $.step.decision.tools\n approval.$: $.approval\n ResultSelector:\n messages.$: $.Payload.messages\n ResultPath: $.tool_result\n Retry:\n - ErrorEquals: [Lambda.ServiceException, Lambda.AWSLambdaException, Lambda.SdkClientException, Lambda.TooManyRequestsException]\n IntervalSeconds: 2\n BackoffRate: 2.0\n MaxAttempts: 3\n Catch:\n - ErrorEquals: [States.ALL]\n ResultPath: $.error\n Next: FailRun\n Next: Carry\n\n Carry:\n Type: Pass\n Parameters:\n run_id.$: $.run_id\n alert.$: $.alert\n messages.$: $.tool_result.messages\n iteration.$: $.step.decision.iteration\n Next: Decide\n\n Finalize:\n Type: Task\n Resource: arn:aws:states:::lambda:invoke\n Parameters:\n FunctionName: ${FinalizeFunctionArn}\n Payload:\n run_id.$: $.run_id\n alert.$: $.alert\n final.$: $.step.decision\n ResultSelector:\n result.$: $.Payload\n ResultPath: $.summary\n Next: PublishCompleted\n\n PublishCompleted:\n Type: Task\n Resource: arn:aws:states:::events:putEvents\n Parameters:\n Entries:\n - EventBusName: ${BusName}\n Source: ops.agent\n DetailType: agent.completed\n Detail:\n run_id.$: $.run_id\n summary.$: $.summary.result\n End: true\n\n FailRun:\n Type: Fail\n Error: AgentFailed\n Cause: The agent loop hit an unrecoverable error. Check the execution history.\n```\n\nRead the `Decide` retry block closely. Throttling gets five attempts with exponential backoff and full jitter, because Bedrock capacity errors tend to clear in seconds and synchronized retries make them worse. Generic Lambda service errors get three attempts. Anything else falls through to `Catch`, which sends the run to `FailRun` and surfaces it to the failure alarm you will add in Step 12.\n\nThe approval timeout deserves a note. If nobody answers within an hour, the state does not fail the run. It records a rejection and lets the model see the denial, so the final summary can say the restart was recommended but never approved. A silent failure here would hide exactly the cases that most need a human follow-up.\n\n## Step 9: The SAM Template, the EventBridge Rule, and Least-Privilege IAM\n\nThe template ties everything together. It creates the bus, the table, the SNS topic, the HTTP API, five functions, the state machine and the rule that starts it. Each function gets only the permissions it uses. The decide function can call Bedrock on the specific model ARNs you pass in and nothing else. The run-tool function can read CloudWatch metrics and write to one table. The approve function can complete tasks, and that is all it can do.\n\nThe EventBridge rule filters on source, detail type and severity, so low-severity noise never starts a run. SAM’s `EventBridgeRule` event type builds the rule, the target role and a dead-letter queue in one block. The retry policy tells EventBridge how long to keep trying if the state machine is unreachable. Because the rule sits on a custom bus, nothing from your other AWS activity can trigger it by accident.\n\n```\nAWSTemplateFormatVersion: \"2010-09-09\"\nTransform: AWS::Serverless-2016-10-31\nDescription: Event-driven ops triage agent (EventBridge, Step Functions, Lambda, Bedrock)\n\nParameters:\n ModelId:\n Type: String\n Description: Bedrock model ID or inference profile ID copied from the console\n ModelArns:\n Type: CommaDelimitedList\n Description: ARNs for the inference profile and the foundation models it routes to\n ApprovalEmail:\n Type: String\n\nGlobals:\n Function:\n Runtime: python3.13\n Timeout: 60\n MemorySize: 512\n CodeUri: src/\n Tracing: Active\n Environment:\n Variables:\n TABLE_NAME: !Ref RunsTable\n\nResources:\n AgentBus:\n Type: AWS::Events::EventBus\n Properties:\n Name: ops-agent-bus\n\n RunsTable:\n Type: AWS::DynamoDB::Table\n Properties:\n BillingMode: PAY_PER_REQUEST\n AttributeDefinitions:\n - AttributeName: pk\n AttributeType: S\n - AttributeName: sk\n AttributeType: S\n KeySchema:\n - AttributeName: pk\n KeyType: HASH\n - AttributeName: sk\n KeyType: RANGE\n TimeToLiveSpecification:\n AttributeName: ttl\n Enabled: true\n\n ApprovalTopic:\n Type: AWS::SNS::Topic\n Properties:\n Subscription:\n - Protocol: email\n Endpoint: !Ref ApprovalEmail\n\n ApprovalApi:\n Type: AWS::Serverless::HttpApi\n Properties:\n DefaultRouteSettings:\n ThrottlingBurstLimit: 10\n ThrottlingRateLimit: 5\n\n DecideFunction:\n Type: AWS::Serverless::Function\n Properties:\n Handler: decide.handler\n Timeout: 90\n Environment:\n Variables:\n MODEL_ID: !Ref ModelId\n MAX_ITERATIONS: \"6\"\n Policies:\n - Statement:\n - Effect: Allow\n Action: bedrock:InvokeModel\n Resource: !Ref ModelArns\n\n RunToolFunction:\n Type: AWS::Serverless::Function\n Properties:\n Handler: run_tool.handler\n Policies:\n - DynamoDBCrudPolicy:\n TableName: !Ref RunsTable\n - Statement:\n - Effect: Allow\n Action: cloudwatch:GetMetricStatistics\n Resource: \"*\"\n\n RequestApprovalFunction:\n Type: AWS::Serverless::Function\n Properties:\n Handler: request_approval.handler\n Environment:\n Variables:\n TOPIC_ARN: !Ref ApprovalTopic\n APPROVAL_BASE_URL: !Sub \"https://${ApprovalApi}.execute-api.${AWS::Region}.amazonaws.com\"\n Policies:\n - SNSPublishMessagePolicy:\n TopicName: !GetAtt ApprovalTopic.TopicName\n\n ApproveFunction:\n Type: AWS::Serverless::Function\n Properties:\n Handler: approve.handler\n Policies:\n - Statement:\n - Effect: Allow\n Action:\n - states:SendTaskSuccess\n - states:SendTaskFailure\n Resource: \"*\"\n Events:\n Approval:\n Type: HttpApi\n Properties:\n ApiId: !Ref ApprovalApi\n Path: /approval\n Method: ANY\n\n FinalizeFunction:\n Type: AWS::Serverless::Function\n Properties:\n Handler: finalize.handler\n Policies:\n - DynamoDBCrudPolicy:\n TableName: !Ref RunsTable\n\n AgentLogs:\n Type: AWS::Logs::LogGroup\n Properties:\n RetentionInDays: 30\n\n AgentStateMachine:\n Type: AWS::Serverless::StateMachine\n Properties:\n Type: STANDARD\n DefinitionUri: statemachine/agent.asl.yaml\n DefinitionSubstitutions:\n DecideFunctionArn: !GetAtt DecideFunction.Arn\n RunToolFunctionArn: !GetAtt RunToolFunction.Arn\n RequestApprovalFunctionArn: !GetAtt RequestApprovalFunction.Arn\n FinalizeFunctionArn: !GetAtt FinalizeFunction.Arn\n BusName: !Ref AgentBus\n Tracing:\n Enabled: true\n Logging:\n Level: ERROR\n IncludeExecutionData: false\n Destinations:\n - CloudWatchLogsLogGroup:\n LogGroupArn: !GetAtt AgentLogs.Arn\n Policies:\n - LambdaInvokePolicy:\n FunctionName: !Ref DecideFunction\n - LambdaInvokePolicy:\n FunctionName: !Ref RunToolFunction\n - LambdaInvokePolicy:\n FunctionName: !Ref RequestApprovalFunction\n - LambdaInvokePolicy:\n FunctionName: !Ref FinalizeFunction\n - EventBridgePutEventsPolicy:\n EventBusName: !Ref AgentBus\n Events:\n HighSeverityAlert:\n Type: EventBridgeRule\n Properties:\n EventBusName: !Ref AgentBus\n Pattern:\n source:\n - ops.alerts\n detail-type:\n - alert.raised\n detail:\n severity:\n - high\n - critical\n RetryPolicy:\n MaximumRetryAttempts: 4\n MaximumEventAgeInSeconds: 3600\n DeadLetterConfig:\n Type: SQS\n\n FailedRunsAlarm:\n Type: AWS::CloudWatch::Alarm\n Properties:\n Namespace: AWS/States\n MetricName: ExecutionsFailed\n Dimensions:\n - Name: StateMachineArn\n Value: !Ref AgentStateMachine\n Statistic: Sum\n Period: 300\n EvaluationPeriods: 1\n Threshold: 1\n ComparisonOperator: GreaterThanOrEqualToThreshold\n TreatMissingData: notBreaching\n AlarmActions:\n - !Ref ApprovalTopic\n\nOutputs:\n StateMachineArn:\n Value: !Ref AgentStateMachine\n ApprovalBaseUrl:\n Value: !Sub \"https://${ApprovalApi}.execute-api.${AWS::Region}.amazonaws.com\"\n RunsTableName:\n Value: !Ref RunsTable\n```\n\nTwo lines in that file are intentional compromises. `cloudwatch:GetMetricStatistics` uses a wildcard resource because CloudWatch read actions do not support resource-level scoping. And `states:SendTaskSuccess` is allowed on every resource because the task token itself is the capability. If you want tighter control, constrain who can reach the HTTP API instead of trying to scope that action.\n\n## Steps 10-11: Deploy the Stack and Fire a Test Alert\n\n**Step 10: Build and deploy.** Run the build, then the guided deploy. Answer the prompts with a stack name of `ops-agent`, your Region, your model ID, the comma-separated ARNs and your email. SAM stores the answers in `samconfig.toml` so later deploys are one command. Within a minute of the stack finishing, AWS sends a subscription confirmation to your inbox. Click the confirm link, or the approval email will never arrive.\n\n```\nsam validate --lint\nsam build\nsam deploy --guided --stack-name ops-agent --capabilities CAPABILITY_IAM\n\naws cloudformation describe-stacks --stack-name ops-agent --query \"Stacks[0].Outputs\" --output table\n```\n\nThe stack output lists the state machine ARN and the base URL of the approval API. Keep both in a scratch file. If `sam validate --lint` prints warnings about the DefinitionUri, make sure you are running it from the project root and that the ASL file exists at the path in the template.\n\n**Step 11: Seed a metric and send an alert.** The agent needs something to find. Push a few synthetic CPU readings into a custom CloudWatch namespace, then publish an alert event to the bus. CloudWatch can take a couple of minutes to make fresh custom datapoints queryable, so wait before sending the alert.\n\n```\nfor v in 91 94 96 93 95\ndo\n aws cloudwatch put-metric-data --namespace Custom/Ops --metric-name CpuPercent \\\n --dimensions Service=checkout-api --value $v\n sleep 2\ndone\n\ncat > events/alert.json <<'EOF'\n{\n \"Entries\": [{\n \"EventBusName\": \"ops-agent-bus\",\n \"Source\": \"ops.alerts\",\n \"DetailType\": \"alert.raised\",\n \"Detail\": \"{\\\"severity\\\":\\\"high\\\",\\\"service\\\":\\\"checkout-api\\\",\\\"title\\\":\\\"CPU above 90 percent for 10 minutes\\\",\\\"alert_type\\\":\\\"high_cpu\\\",\\\"namespace\\\":\\\"Custom/Ops\\\",\\\"metric\\\":\\\"CpuPercent\\\",\\\"dimension_name\\\":\\\"Service\\\",\\\"dimension_value\\\":\\\"checkout-api\\\"}\"\n }]\n}\nEOF\n\naws events put-events --cli-input-json file://events/alert.json\n```\n\nA successful publish returns a zero failed-entry count. The execution should appear in the Step Functions console within seconds.\n\n```\n{\n \"FailedEntryCount\": 0,\n \"Entries\": [\n { \"EventId\": \"11111111-2222-3333-4444-555555555555\" }\n ]\n}\n```\n\nWatch the run in the console, or list it from the terminal. A typical path goes Init, Decide, Route, SkipApproval, RunTool for the metric read, around the loop again for the runbook, then to a third Decide that asks for `restart_service`. At that point the execution turns to `RequestApproval` and stays there. Check your inbox, open the link, and click Approve. The run resumes, records the action, and finishes with a summary.\n\n```\naws stepfunctions list-executions --state-machine-arn \"$STATE_MACHINE_ARN\" --max-results 3\naws stepfunctions describe-execution --execution-arn \"$EXECUTION_ARN\" --query \"{status:status,output:output}\"\naws dynamodb query --table-name \"$TABLE_NAME\" \\\n --key-condition-expression \"pk = :p\" \\\n --expression-attribute-values '{\":p\":{\"S\":\"'\"$RUN_ID\"'\"}}'\n```\n\nExpected output after approval looks like the sample below. Your wording will differ because the model writes the summary, and the exact number of iterations depends on how it chooses tools.\n\n```\n{\n \"status\": \"SUCCEEDED\",\n \"output\": \"{\\\"run_id\\\":\\\"a1b2c3d4-...\\\",\\\"summary\\\":{\\\"result\\\":{\\\"run_id\\\":\\\"a1b2c3d4-...\\\",\\\"summary\\\":\\\"Probable cause: sustained CPU saturation on checkout-api. Evidence: five-minute averages above 90 percent across the last window, runbook step 3 met. Action taken: restart requested and approved by an operator.\\\",\\\"iterations\\\":4}}}\"\n}\n\nItems in DynamoDB for the run:\n sk = action#tooluse_xxxxxxxx service = checkout-api reason = ...\n sk = summary iterations = 4\n```\n\nNow test the unhappy paths, because they are where the design earns its keep. Send the same alert again and click Reject. The run should finish with a summary that says the restart was denied. Send it a third time and ignore the email for an hour, or temporarily shorten `TimeoutSeconds` to 60 and redeploy. The run should end with the same denial and a timeout reason in the execution history.\n\n## Step 12: Logging, Alarms, and Cost Controls\n\n**Step 12: Watch the agent and cap what it can spend.** The template already ships three safety nets: an alarm on failed executions, a 30-day retention period on the state machine log group, and an iteration cap of six. The Lambda functions print one JSON line per decision with token counts, and CloudWatch Logs Insights can parse those lines without any extra setup. Run the query below against the decide function’s log group once a day during the first week.\n\n```\nfields @timestamp, run_id, iteration, in, out\n| filter event = \"decide\"\n| stats sum(in) as input_tokens, sum(out) as output_tokens, count() as model_calls by bin(1h)\n| sort @timestamp desc\n```\n\nBedrock tokens will be the biggest line on the bill, not Lambda. Bedrock pricing varies by provider, model and modality, and the pricing page lists four service tiers: Standard, Flex, Priority and Reserved. According to the page AWS updated on October 30, Flex and batch inference run at a 50 percent discount to Standard, while Priority reportedly carries a 75 percent premium. For triage that nobody is watching live, Flex is worth testing if your chosen model offers it. AWS also says selected models support batch inference at up to 50 percent below on-demand prices, which suits nightly sweeps better than live alerts.\n\nLambda cost is easy to estimate from the published list rates: $0.20 per million requests and $0.0000166667 per GB-second, after a monthly free tier of 1 million requests and 400,000 GB-seconds. The table below assumes about 12 Lambda invocations per alert, each running 1.5 seconds at 512 MB, which works out to 0.75 GB-seconds per call and 9 GB-seconds per alert. That is illustrative arithmetic on the x86 list rate in the example region, not a measured bill.\n\n| Alerts per month | Lambda invocations | GB-seconds | Request charge | Compute charge | Lambda total | \n|---|---|---|---|---|---|\n| 1,000 | 12,000 | Not listed | $0.00 | $0.00 | $0.00 | \n| 10,000 | 120,000 | Not listed | $0.00 | $0.00 | $0.00 | \n| 100,000 | 1.2 million | 900,000 | $0.04 | Not listed | Not listed | \n| 1,000,000 | Not listed | Not listed | $2.20 | Not listed | Not listed | \n\nEven at a million alerts a month, Lambda stays small beside the model bill. Step Functions adds roughly 20 state transitions per alert in this design. AWS prices Standard workflows per transition, retries included, and reportedly gives 4,000 free transitions per month, so check the current rate on the Step Functions pricing page before you extrapolate. EventBridge bills per event, and each 64 KB chunk of payload counts as one event, which is another reason to keep the completion event short.\n\nFor the broader picture of how these services fit together, the [cloud computing section](https://shattered.io/category/cloud-computing/) collects the AWS, Azure and [Google Cloud](https://shattered.io/google-cloud-run-deploy-docker-12-steps-2026/) coverage in one place.\n\nTwo more cost habits pay off early. Set a reserved concurrency limit on the decide function, so a flood of alerts cannot open hundreds of parallel model calls. And turn on EventBridge’s dead-letter queue alarm, because an alert that never reached the state machine costs nothing but can hurt a lot.\n\n## Common Pitfalls That Break Agent Builds\n\nThese six mistakes account for most of the failed first deployments. Each one is easy to avoid once you know it exists.\n\n1. **Approving on a GET request.** Email scanners and chat link previews fetch every URL they see. A GET that approves an action will approve it before a human reads the message. Show a form on GET and act only on POST.\n2. **Skipping a tool result.** If the model emits two tool calls and you answer only one, the next Converse call fails with a validation error. Always return one result per tool-use ID, even if that result is an error message.\n3. **Dropping the tool config on later turns.** Once the transcript contains tool-use or tool-result blocks, the Converse API expects the tool configuration on every call. Removing it to force a final answer breaks the request. Stop the loop in code instead, as the iteration cap does.\n4. **Letting the transcript grow without limit.** Step Functions caps the size of the data passed between states. A verbose tool that returns pages of JSON will push a long run over the edge. Truncate tool output, and move the transcript to DynamoDB if runs routinely exceed ten loops.\n5. **Retrying actions that are not idempotent.** Retries are a feature of the state machine, which means a tool can run twice after a timeout. Key every state-changing action on the run ID and the tool-use ID, and write a conditional record before the side effect.\n6. **Publishing to the bus that triggers you.** The completion event goes back to the same bus as the alert. That is safe only because the rule matches`ops.alerts` and not`ops.agent` . Loosen the pattern and you get an infinite loop with a bill attached. AWS reportedly extended recursive-loop detection to EventBridge custom buses in October, but do not rely on it as your only guard.\n\nA seventh trap sits outside the code. Teams grant broad Bedrock permissions during testing, then forget to narrow them. The template in Step 9 scopes the policy to specific model ARNs from the start, which costs five minutes now and avoids an audit finding later.\n\n## Troubleshooting: 10 Errors and Their Fixes\n\n1. **AccessDeniedException from Bedrock.** Check that the model is enabled for your account in this Region, and that the policy lists both the inference profile ARN and the foundation model ARNs behind it. A profile that routes across Regions needs the model ARNs in each Region.\n2. **ValidationException about toolResult blocks.** The previous assistant message contained a tool call that has no matching result. Compare the tool-use IDs in the transcript with the IDs in the next user message. The run-tool function must answer all of them.\n3. **ValidationException about message order.** The first message must come from the user, and roles must alternate. If you seed the transcript yourself, check that two user messages never sit side by side.\n4. **The execution sits on RequestApproval and no email arrives.** The SNS subscription is still unconfirmed. Look for the AWS confirmation email, including the spam folder, and confirm it. Then start a new run, since the earlier message was never delivered.\n5. **The approval page says the request has expired.** The run already timed out, or someone clicked twice. The token works once. Check the execution history for a recorded rejection with a timeout reason.\n6. **States.Runtime: the JSONPath could not be found.** A state read a field that the previous state did not write. The usual culprit is`$.approval` on the path that skips approval. Make sure`SkipApproval` writes its result to that exact path.\n7. **The EventBridge rule never fires.** Confirm you published to`ops-agent-bus` and not the default bus, and that source, detail type and severity match exactly. Patterns are case-sensitive. Check the rule’s`FailedInvocations` metric and the dead-letter queue.\n8. **The decide function times out.** A long transcript or a slow model can exceed the 90-second limit. Raise the timeout, shorten the transcript, or choose a faster model. Remember to keep the state machine’s own retry count in mind, since each retry repeats the call.\n9. **CloudWatch returns empty datapoints.** Custom metrics need a minute or two to appear, and the dimension name and value must match the ones in the alert detail exactly. Run the same query from the CLI to confirm.\n10. **Repeated ThrottlingException errors.** Your retry rule may not be matching. The Lambda must raise`BedrockThrottled` , and the Retry block must list that exact name. If throttling persists, lower concurrency, try a different tier, or request a quota increase.\n\nA few extra checks help when none of those fit. Open the failed execution in the console and read the exception state, since the error name usually points at the right layer. Run the failing Lambda alone with `sam local invoke` using the state that went into it, which you can copy from the execution input tab. And turn the state machine log level up to ALL for one test run, then back to ERROR so you do not log execution data longer than you need to.\n\n## Advanced Tips for Production Hardening\n\n### Move the Transcript Out of the State\n\nThe simplest change with the biggest payoff is storing the conversation in DynamoDB and passing only a pointer through the state machine. The decide function reads the transcript by run ID, appends the new message, writes it back, and returns just the action. Payload size stops mattering, runs of thirty loops become possible, and the execution history in the console gets easier to read because it no longer shows kilobytes of JSON per state. The cost is one extra read and one extra write per loop, both on-demand and tiny.\n\n### Expose the Workflow as an Agent Tool\n\nOnce the triage flow works, other agents can call it. AWS’s October 29 announcement says Step Functions workflows can be registered as tools behind Bedrock AgentCore Gateway, with standard Step Functions and gateway charges applying. A planning agent could then request a triage run for a given service and read back the summary, without knowing anything about EventBridge or approval emails. AWS also reported a Step Functions step powered by AgentCore for agentic reasoning inside a workflow, dated July 20, which suits cases where only one branch needs open-ended thinking. Both features are recent, so check Regional availability and current pricing before you design around them.\n\nIf you run coding agents alongside ops agents, the same event-driven habits carry over. Our guide to [building Cursor cloud agents](https://shattered.io/build-cursor-cloud-agents-2026/) shows how another vendor structures isolated environments and secrets, which is a useful contrast to the IAM-first approach used here.\n\n### Route by Cost, Guard by Policy\n\nNot every alert needs the same model. A cheap, fast model can classify the alert and decide whether it is worth a full investigation. Only the ambiguous cases escalate to a larger model. Add a classification state before Decide, put the model ID in the state input, and let a Choice state pick the path. Pair that with Amazon Bedrock Guardrails by adding a guardrail configuration to the Converse call, so unsafe or off-topic output is filtered before it reaches a tool. Keep the guardrail on the model’s input and output, and keep the approval gate on the tools. They protect different things.\n\nOther upgrades are worth a look as the system matures:\n\n- **Lambda durable functions.** AWS documents them as a Lambda capability for longer-running, event-driven workflows, including remediation flows that combine EventBridge and Bedrock. They suit teams that prefer to write the loop in code instead of ASL. The AWS announcement dated October 16 reportedly extended them to GovCloud.\n- **EventBridge Scheduler.** Use it for periodic sweeps, such as asking the agent to review open alerts every morning. AWS lists a free tier of 14 million invocations per month and then $1 per million, according to its pricing page.\n- **The enhanced Custom event bus.** Launched in September 2026, it adds organization-wide sharing, optional ordering, content-based deduplication, synchronous Lambda targets and built-in retention. It uses separate ingress and egress pricing, so model your volumes before migrating.\n- **Archive and replay.** Archive the alert stream, then replay last month’s alerts against a new model or prompt in a test stack. That is the cheapest regression test an agent can get.\n- **Edge placement.** If the approval page needs to load fast for operators worldwide, compare the options in our[Cloudflare Workers and Lambda@Edge comparison](https://shattered.io/cloudflare-workers-vs-aws-lambda-edge-2026/) before adding a CDN layer.\n- **Memory pricing awareness.** If you graduate to AgentCore Memory, note that AWS changed short-term memory billing on October 6 from $0.25 per 1,000 new events to usage-based rates of $1.00 per GB ingested, $0.20 per GB retrieved and $0.10 per GB-month stored.\n\n## The Complete Working Project\n\nThe finished repository has ten files. Every file appears in full above, so assembling it is a copy and paste job. The tree below shows where each piece lives, and the script after it runs the whole sequence from an empty folder to a tested stack.\n\n```\nops-agent/\n template.yaml Step 9: bus, table, topic, API, functions, state machine, rule, alarm\n samconfig.toml written by sam deploy --guided\n statemachine/\n agent.asl.yaml Step 8: the loop and the approval gate\n src/\n tools.py Step 3: tool specs, runbooks, dispatcher\n decide.py Step 4: one Bedrock Converse call per invocation\n run_tool.py Step 5: executes tools, enforces approval\n request_approval.py Step 6: emails the approval link\n approve.py Step 6: form page and SendTaskSuccess\n finalize.py Step 7: writes the run summary\n events/\n alert.json Step 11: test alert\nbash\n#!/usr/bin/env bash\nset -euo pipefail\n\nSTACK=ops-agent\nMODEL_ID=\"paste-your-model-or-profile-id\"\nMODEL_ARNS=\"paste-profile-arn,paste-model-arn\"\nEMAIL=\"[email protected]\"\n\nsam validate --lint\nsam build\nsam deploy --stack-name \"$STACK\" --capabilities CAPABILITY_IAM --resolve-s3 \\\n --parameter-overrides ModelId=\"$MODEL_ID\" ModelArns=\"$MODEL_ARNS\" ApprovalEmail=\"$EMAIL\"\n\necho \"Confirm the SNS subscription email, then press Enter to send a test alert.\"\nread -r _\n\nfor v in 91 94 96 93 95\ndo\n aws cloudwatch put-metric-data --namespace Custom/Ops --metric-name CpuPercent \\\n --dimensions Service=checkout-api --value \"$v\"\ndone\nsleep 120\naws events put-events --cli-input-json file://events/alert.json\n\nSM_ARN=$(aws cloudformation describe-stacks --stack-name \"$STACK\" \\\n --query \"Stacks[0].Outputs[?OutputKey=='StateMachineArn'].OutputValue\" --output text)\naws stepfunctions list-executions --state-machine-arn \"$SM_ARN\" --max-results 1\n\n# Teardown when finished:\n# sam delete --stack-name \"$STACK\" --no-prompts\n```\n\nDelete the stack when you finish testing. The approval endpoint is public, the SNS subscription keeps your inbox on the hook, and the table keeps its data until the time-to-live sweep removes it. A single `sam delete` removes every resource the template created.\n\n## Frequently Asked Questions\n\n**How much does a serverless AI agent on AWS cost to run?**\n\nAt low volume the Lambda portion is free, because the monthly free tier covers 1 million requests and 400,000 GB-seconds. The real cost is Bedrock tokens, which depend on the model, the service tier and how many loops each alert needs. Log token counts from day one and multiply by the current price for your model.\n\n**Should I use Step Functions or Bedrock AgentCore for the agent loop?**\n\nUse Step Functions when you want a visible, auditable flow with retries and human gates that you control step by step. Use AgentCore when you want a managed runtime with memory and tool access and you accept more platform surface. The two also combine, since workflows can be exposed to agents as tools.\n\n**Which Bedrock model works best for tool-calling agents?**\n\nAny model that supports tool use in the Converse API can run this loop. GPT-6.1 Terra became generally available on Bedrock on October 26, 2026, and other families are listed in the Bedrock console. Start with a small, fast model, measure how often it picks the right tool, and move up only if it fails.\n\n**Is it safe to let an AI agent restart production services?**\n\nOnly with guardrails. Keep a short list of risky tools, require a human decision for them, enforce that rule inside the tool runner as well as the workflow, and make every action idempotent. Start with read-only tools and add write tools one at a time.\n\n**Do I need the new EventBridge enhanced Custom event bus?**\n\nNo. The Classic bus used in this build works as before, and AWS says existing buses continue unchanged. The enhanced bus is worth evaluating if you need cross-account sharing, ordering or deduplication.\n\n**Why does the Lambda in this build count as an Event Function?**\n\nAWS renamed existing Lambda functions to Lambda Event Functions in October 2026, when it introduced Lambda Web Functions for direct HTTP handling. The rename needs no code changes, so every handler here keeps working as written.\n\n**How do I stop the agent from looping forever?**\n\nCap the iterations in code, as the decide function does, and cap concurrency at the function level. Add an alarm on failed executions and a budget alert on the account. Three independent limits mean no single mistake can run up a surprise bill.\n\n**Can I swap SNS email for Slack approvals?**\n\nYes. Replace the SNS publish in the request-approval function with a Slack message that carries the task token in a button payload, and verify Slack’s request signature in the approval handler. The state machine does not change.", "url": "https://wpnews.pro/news/serverless-ai-agent-on-aws-12-steps-90-min-2026", "canonical_source": "https://shattered.io/serverless-ai-agent-aws-lambda-bedrock-step-functions-12-steps-2026/", "published_at": "2026-11-02 07:13:59+00:00", "updated_at": "2026-11-02 07:19:16.842444+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "artificial-intelligence", "large-language-models"], "entities": ["AWS", "Amazon Bedrock", "Step Functions", "EventBridge", "Lambda", "DynamoDB", "OpenAI", "Gartner"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/serverless-ai-agent-on-aws-12-steps-90-min-2026", "markdown": "https://wpnews.pro/news/serverless-ai-agent-on-aws-12-steps-90-min-2026.md", "text": "https://wpnews.pro/news/serverless-ai-agent-on-aws-12-steps-90-min-2026.txt", "jsonld": "https://wpnews.pro/news/serverless-ai-agent-on-aws-12-steps-90-min-2026.jsonld"}}