When I started building Autowired.ai — an AI document extraction SaaS, one of the earliest decisions I had to make was which AWS messaging and orchestration services to use for the processing pipeline.
My first instinct was to reach for SQS everywhere. I knew it well. It's simple, it's cheap; it's reliable. But as the pipeline grew more complex — document uploads triggering extraction workflows, batches processing dozens of files in parallel, and webhook notifications delivered to customer endpoints, I kept running into the edges of what a queue alone can do.
I ended up using all three services: SQS, EventBridge, and Step Functions. Not because I wanted complexity, but because each one fits a specific job that the others don't.
This post discusses the decision-making process for each service, detailing what each one does, where it may fail, and the specific reasons why I selected each service for different components of the Autowired pipeline.
The Problem With Reaching for One Tool
Here's the antipattern I see a lot: an engineer learns SQS, it works for the first async use case, and then SQS becomes the default for everything async. Queue → Lambda → done. Repeat.
This works fine until your workflow has multiple steps. Then you start chaining queues: Queue A → Lambda B writes to Queue C → Lambda D writes to Queue E. You've now built a state machine out of SQS queues — without any of the state, visibility, error handling, or branching logic that a state machine provides.
When something breaks in that chain, you're reconstructing what happened from CloudWatch logs across five different Lambda invocations with different request IDs, trying to figure out which step failed and what the data looked like when it did.
I've been there. It's not fun.
The right answer isn't "use Step Functions for everything" either. Step Functions has overhead — cost per state transition, latency per step, and operational complexity that's overkill for simple async tasks.
The answer is using each service for what it's actually designed for.
The Three Services, Simply Put
SQS is a queue. It buffers work between a producer and a consumer, handles retries, and dead-letters failed messages. It knows nothing about workflow state — only whether a message was processed or needs to be retried.
EventBridge is an event router. It receives events and routes them to one or more consumers based on rules. Its superpower is loose coupling — the producer doesn't know who's listening, and you can add new consumers without touching the producer.
Step Functions is a workflow orchestrator. It manages multi-step processes with persistent state, branching, parallelism, and error handling. Every step's input, output, and failure is recorded in the execution history.
None of these is a substitute for the others. They solve different problems.
How Autowired Uses All Three
Here's the actual service topology in the Autowired processing pipeline:
Let me explain why each service is where it is.
EventBridge: For the Scheduled Trigger
Every 5 minutes, Autowired checks whether any batches have a scheduled execution time that's due. This is handled by a Lambda (ScheduledBatchLambda) triggered by an EventBridge rule:
const scheduledBatchRule = new events.Rule(this, "ScheduledBatchRule", {
ruleName: `autowire-scheduled-batch-${stage}`,
schedule: events.Schedule.rate(cdk.Duration.minutes(5)),
description: "Triggers scheduled batch processing check every 5 minutes",
});
scheduledBatchRule.addTarget(
new targets.LambdaFunction(scheduledBatchLambda, {
retryAttempts: 2,
})
);
Why EventBridge here and not a CloudWatch Events cron or a polling Lambda?
Because EventBridge is the native AWS scheduling primitive. The rule is declarative, versioned in CDK, has built-in retry logic (retryAttempts: 2), and integrates cleanly with Lambda. There's no infrastructure to manage, no polling loop to maintain.
The retryAttempts: 2 matters: if the Lambda has a cold start failure or a transient error, EventBridge retries twice before giving up. Without this, a Lambda cold start would silently skip a scheduled batch check.
Step Functions: For the Document Processing Workflow
When a document batch is submitted — either via S3 upload or scheduled trigger — it starts a Step Functions execution. This is the core of the pipeline, and it's where the complexity lives.
The state machine looks like this:
I want to highlight a few specific decisions here:
The Map state with maxConcurrency: 10. Each batch can have dozens or hundreds of documents. The Map state fans out to one execution per document, running up to 10 in parallel. Why 10 and not 50? Because Textract and Bedrock both have per-account concurrency limits. Unconstrained parallelism would exhaust those limits, trigger throttling, and ironically make the batch slower. 10 is a deliberate contract with AWS service quotas, not a performance guess.
Per-document error handling with addCatch. This was one of the most important design decisions. If one corrupted PDF crashes the document processor, it should not abort the other 49 documents in the batch. The
processDocument.addCatch(markDocumentFailed, {
errors: ["States.ALL"],
resultPath: "$.error",
});
Without this, a single bad document would fail the entire batch execution. That's the wrong failure mode.
Why not Lambda chains? I tried a simpler version of this early on — chaining Lambda invocations directly. The moment I needed to fan out across multiple documents and then aggregate results (to determine overall batch status), Lambda chains couldn't express it. There's no fan-in primitive. Step Functions' Map state handles this natively.
The other thing Lambda chains can't give you: execution history. When a batch fails, the first thing I do is open the Step Functions console and look at the execution. Every state, every input, every error is right there. That debugging visibility has saved me hours.
SQS: For Webhook Delivery
After a batch completes, if the customer has webhook delivery configured, Autowired sends a notification to their endpoint. This is handled via SQS, not via a direct Lambda call from inside the state machine, and not via EventBridge.
Here's why SQS specifically:
this.webhookDeliveryQueue = new sqs.Queue(this, "WebhookDeliveryQueue", {
queueName: `autowire-webhook-queue-${stage}`,
visibilityTimeout: cdk.Duration.seconds(60),
deadLetterQueue: {
queue: webhookDlq,
maxReceiveCount: 5,
},
});
The decoupling is the point. Customer webhook endpoints are unreliable — they go down, they time out, they return 500s for unrelated reasons. If webhook delivery is inside the Step Functions execution, a failed webhook delivery blocks or fails the batch. Those are completely unrelated concerns.
By routing to SQS, the Step Functions execution completes cleanly. Webhook delivery has its own retry budget (**maxReceiveCount: 5 **— more than the document processor queue's 3, because external endpoints are flakier than internal Lambdas). Failures dead-letter after 5 attempts and trigger a separate alarm.
** batchSize: 1 on the consumer**. The
webhookDeliveryLambda.addEventSource(
new lambdaEventSources.SqsEventSource(this.webhookDeliveryQueue, {
batchSize: 1,
})
);
If you process 10 webhook deliveries in one Lambda invocation and 3 fail, SQS can't partially acknowledge all 10 retry. With batchSize: 1, each webhook delivery retries independently. The blast radius of a failed delivery is exactly one customer, not ten.
The Decision Framework I Actually Use
After building this, here's how I'd frame the decision for any new async requirement:
Reach for SQS when:
Reach for EventBridge when:
Reach for Step Functions when:
And the pattern to avoid: using SQS to chain multi-step workflows. If you find yourself writing a Lambda that reads from Queue A and writes to Queue B to trigger the next step, stop and reach for Step Functions instead.
Wrapping Up
The combination of EventBridge for scheduling, Step Functions for orchestration, and SQS for decoupled delivery isn't overengineering. It's three services doing the jobs they were designed for.
The Autowired pipeline is more debuggable, more resilient, and operationally cleaner because each service is in its right place — not because I used the most services, but because I used the right ones.
Next I'm writing about the full event-driven document processing pipeline — how S3 uploads trigger the Step Functions execution, how the DLQs are wired, and the failure handling that makes it production grade.
Follow along if that's useful.
This is part of a 10-post series on the architecture behind Autowired.ai — an AI document extraction SaaS I built solo on AWS serverless.