AWS Built an Emulator for Everything. It Thinks Nobody Is Famous

AWS released AWS Emulator, a local emulator for nearly all of its cloud services, and published its full issues folder documenting 19,252 gaps found between October 21 and November 10 — 12,708 since fixed and 6,544 still open, including 805 marked CRITICAL. AWS said three teams of autonomous AI agents (test writers, issue finders and fixers) built the emulator, with the company stating "Our team hasn't written or reviewed any of the code" while humans built the tests, issue process, merge gate and release pipeline. The roughly 900 issues filed per day show the emulator diverging from real AWS behavior in places, such as Amazon Rekognition's RecognizeCelebrities API always returning an empty celebrity-faces array.

Writing AWS Built an Emulator for Everything. It Thinks Nobody Is Famous. On this page The world of cloud testing got a little bit stranger this week. Well actually a lot better. AWS released AWS Emulator https://github.com/aws/aws-emulator , a local emulator for pretty much all of its services. You point your SDK or CLI at it, and instead of talking to AWS you talk to a program running on your laptop. No account, no bill, no “who left that NAT Gateway running” Slack message on a Monday morning. But that is not what caught my eye. What caught my eye was the issues folder https://github.com/aws/aws-emulator/tree/main/issues . They published every gap they found along the way. Not a curated list. ALL OF THEM So of course I went and read them. All of them. Are you nuts? I had help, Claude of course. Nobody in their right mind reads 137 MB of markdown with their own eyes and stays sane. Let me explain what pulled out of what Claude found for me. First, some numbers As of the time I am writing this: 1. 19,252 issue files 2. 12,708 of them fixed things-that-now-work https://github.com/aws/aws-emulator/tree/main/issues/things-that-now-work 3. 6,544 still open things-that-do-not-work https://github.com/aws/aws-emulator/tree/main/issues/things-that-do-not-work 4. 805 marked CRITICAL 5. All filed between October 21 and November 10 That is roughly 900 issues a day . For three weeks. No human team files 900 issues a day. And no human team did. AWS wrote up how autonomous AI agents built AWS Emulator https://www.aboutamazon.com/news/aws/how-autonomous-ai-agents-built-aws-emulator , and it is worth your time. It started as a weekend experiment. The agent came back with a first version of SQS in about 15 minutes, so naturally the next step was… the rest of AWS. Over the weekend. Turning that into an actual product took a lot longer, to be fair. The setup is three teams of agents: 1. Test writers build scenarios from the public API models and verify each one against real AWS 2. Issue finders replay those tests against the emulator and file an issue for every failure 3. Fixers pick up an issue each, make a change, and have to get it past a merge gate compiles, coding standards, security, no fidelity regressions And the humans? In their own words: “Our team hasn’t written or reviewed any of the code.” The humans built the guardrails the tests, the issue process, the merge gate, the release pipeline and let the agents loose inside them. My favorite detail from that post: in an early version, the agents needed Python for Lambda, so they built their own Python interpreter . The team had to step in and tell them to please just use the existing libraries. I have worked with engineers like that. I may have been one. So that is why every test resource is named sim2real-something , and why the issues are addressed to “the issue fixer”. Robots finding bugs, robots fixing bugs, and robots writing very, very long descriptions of both. And because every test was checked against real AWS first, each issue compares what the emulator does to a recording of what real AWS did for the same request. Which means that the issues tell you two stories. One about the emulator. And one about AWS itself. Let’s start with the fun one. The emulator’s alternative reality Every emulator has to fake something. This one occasionally fakes it with a lot of confidence. Nobody is famous Amazon Rekognition has an API called RecognizeCelebrities . You give it a photo, it tells you which famous people are in it. The emulator always answers with an empty list https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261106T074501Z-rekognition-recognizecelebrities-always-answers-an-empty-celebrityfaces-array-so.md . Every face goes into the “unrecognized” pile. Brad Pitt? Unrecognized. Taylor Swift? Never heard of her. You? Same as Brad Pitt. Finally Everyone speaks American English Amazon Transcribe can detect which language is being spoken. In the emulator, the answer is always en-US https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261101T145548Z-transcribe-identifylanguage-transcription-jobs-always-report-languagecode-en-us.md . They fed it English audio and French audio with the same settings. Both came back as American English, with a confident score attached . My Hebrew would be classified as en-US . My kids would argue that my English should not be. Predicting the future from nothing Application Auto Scaling has predictive scaling. It looks at your load history and forecasts what is coming. For a service with zero history , real AWS honestly says “I don’t know” it returns NaN . The emulator produced nice, finite, numeric forecasts anyway https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261024T081305Z-application-autoscaling-application-auto-scaling-getpredictivescalingforecast-fabricates-numeric-forecasts-for.md . That is not a forecast. That is a horoscope. This one is fixed, by the way. Any key opens the door AWS Backup Gateway needs an activation key from a real appliance before it creates a gateway. The emulator accepted any string you gave it and created a gateway https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261024T214521Z-backup-gateway-creategateway-accepts-any-activationkey-and-fabricates-a-gateway.md . The issue calls it a “phantom gateway”, which is honestly a great name for a band. A role for a service that doesn’t exist IAM service-linked roles are created for AWS services. Real ones. The emulator happily created one for sim2real-not-a-real-service.amazonaws.com https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261025T000319Z-iam-10-net-new-behavioral-root-causes-four.md , complete with a trust policy for that nonexistent service. The service literally has “not a real service” in its name. And it still got a role. I have seen job interviews with lower bars. Hardware from a parallel universe DescribeInstanceTypes tells you the specs of every EC2 instance type. In the emulator, part of that catalog is computed from its own formulas https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261106T072955Z-ec2-describeinstancetypes-serves-an-emulator-authored-instance-type.md instead of AWS’s published values. The issue says capacity planning against it is “computed from fiction”. My favorite detail: the default TCP established timeout. AWS says 350 seconds. The emulator says 432,000 seconds. That is five days. Your TCP connection could go on vacation and come back to find the session still waiting for it. Never late, never anything In Kinesis, MillisBehindLatest tells you how far behind the stream you are. The emulator always says 0 https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261027T011654Z-kinesis-9-net-new-root-causes-cloudformation-aws.md . You are never behind. Ever. I want that for my inbox. Same issue, bonus round: a CloudFormation stack with a Kinesis stream reports CREATE COMPLETE and hands you a perfectly formed ARN. The stream does not exist. The very next call says so. Born to fail, or never born at all Some resources in the emulator have a predetermined destiny: 1. EMR job flow steps always end FAILED https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261108T093128Z-ec2-emr-emr-job-flow-steps-always-end-failed-a.md 2. VPC Reachability Analyzer analyses always end FAILED https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261106T015719Z-ec2-reachability-analyzer-analyses-always-end-failed-with.md 3. Elastic Beanstalk application versions stay UNPROCESSED forever https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261029T180136Z-elasticbeanstalk-elastic-beanstalk-application-versions-stay-status-unprocessed-forever.md Forever. The word is in the title of the issue. Somewhere there is an application version that is still waiting, like the dog in Futurama. “Which service did you mean?” Here is one that made me laugh out loud. Several services share signing names, so the emulator’s router sometimes could not work out which service you were calling. So it refused the request with an error called AmbiguousService . An error that does not exist in AWS . This happened to: - Amazon SES v2 https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261023T183615Z-sesv2-is-unreachable-every-request-is-refused-ambiguousservice.md every request - Amazon Transcribe https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261022T040822Z-transcribe-router-rejects-every-request-as-ambiguousservice.md every request - AWS IoT Greengrass v2 https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261022T092619Z-greengrassv2-every-request-rejected-with-ambiguousservice-because-greengrass.md every request - AWS Marketplace Discovery https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261022T002146Z-marketplace-discovery-router-refuses-every-request-as-ambiguousservice.md every request And then there is the Sustainability service, which refused every request with UnknownService https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261022T174638Z-sustainability-every-request-is-refused-with-unknownservice-the.md . Sustainability? Never heard of it. Very on brand for the industry, if you ask me. To be fair, all of these are fixed now. But I will be using “AmbiguousService” the next time someone asks me a vague question in a meeting. Issues with ambition Some of these issues are… thorough. 1. There is a 266 KB issue https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261102T071442Z-maps-geo-getstyledescriptor-serves-a-synthesized-style-document-whose-layers.md about the style document of a map. 266 KB. About what color the roads should be. I have written entire presentations that were smaller. 2. There is an issue whose title is over 570 characters long https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261027T172504Z-kinesis-10-net-new-root-causes-the-per.md . It lists ten separate Kinesis bugs. In the title. That is not a title, that is a release note. Plot twist: AWS is weird too And here it comes… Remember that every issue compares the emulator to a recording of real AWS . So to be faithful, the emulator has to reproduce AWS exactly. Including the bits nobody would ever write on purpose. So the robots are now carefully preserving AWS’s typos. The typos you are now required to reproduce - OpenSearch: “ReleaseVersion does not applies for this domain or release does not exist” https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261106T051520Z-opensearch-two-not-found-refusals-answer-emulator-authored.md - License Manager: “Valid range is range 1 to 100.” https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261029T234804Z-license-manager-license-configuration-validation-answers-the-wrong-error-type.md - IoT SiteWise: “Provided CSV file format contains must contain an identifier…” https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261029T190317Z-iotsitewise-refusal-paths-answer-emulator-authored-messages-and-the.md - Athena has a double space https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261104T163600Z-athena-conflict-not-found-and-output-location-refusals.md in one of its error messages. So does SSM GUI Connect https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261029T171307Z-ssm-guiconnect-3-net-new-behavioral-root-causes.md . The issues explicitly call out the doubled space as AWS’s own text. As a requirement. The issue for the OpenSearch one even says it is AWS’s own wording, “grammatical oddity included”. I love the restraint in that sentence. Somewhere, a very sophisticated AI is diligently making sure “does not applies” stays exactly as it is. This is fidelity. And it is hilarious. Placeholders as data Redshift has a catalog of event descriptions. Some of them literally contain <cluster name and <time in UTC https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261109T193340Z-redshift-reference-catalog-reads-serve-empty-or-synthesized.md as text. Not filled in. Just the template. And the emulator is expected to return the same. Who is Locke? AWS AppFlow’s DescribeConnector returns a list of destination connectors. In the recording, item number 5 is a token called Locke https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261108T013452Z-appflow-describeconnector-returns-an-unfaithful-connector-configuration.md . It is not the spelling in the enum. I have no idea what Locke is. If you do, please tell me. Lost https://en.wikipedia.org/wiki/John Locke %28Lost%29 fans, I see you. Error codes with a sense of humor GuardDuty, for a specific request, returns HTTP 400 your fault with the error code InternalServerErrorException our fault https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261029T183757Z-guardduty-lifecycle-state-is-wrong-missing-dry-run.md . The issue calls it GuardDuty’s “documented quirk”. So it is both of our faults. Happy to share the blame here. Very balanced. The documentation is a suggestion - Amazon Connect Campaigns: the API model says you can ask for at most 25 campaigns in one batch. Real AWS happily processes 500 https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261029T172833Z-connectcampaigns-v1-validation-and-not-found-refusals-diverge.md and only refuses at 501. The issue says, and I quote, “The model is wrong about the upper bound”. Tells it like it is. - Marketplace Catalog: the model the emulator ships with says a change name must match ^ a-zA-Z $ . That is one letter . The emulator believed it https://github.com/aws/aws-emulator/blob/main/issues/things-that-now-work/20261024T091731Z-marketplace-catalog-startchangeset-wrong-behavior.md and rejected CreateNewProduct . Real AWS accepts it, of course. Sometimes the problem with following the documentation is following the documentation. OK, one serious bit I promised myself this would be a fun post. But two things are worth knowing before you go and replace your test environment with this. 1. Your “local” test might not be local. Around 121 open issues are about the emulator handing back real AWS hostnames . For example, CodeArtifact’s GetRepositoryEndpoint https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261109T164005Z-codeartifact-getrepositoryendpoint-publishes-a-real-aws-d-codeartifact.md returns a real codeartifact.amazonaws.com URL. Your npm client follows the documented flow, leaves the emulator, and goes looking for real AWS. 2. Permissions are mostly not checked. For example, EC2 DryRun never refuses an unauthorized caller https://github.com/aws/aws-emulator/blob/main/issues/things-that-do-not-work/20261024T065715Z-ec2-iam-sts-ec2-dryrun-authorization-never-refuses-an-unauthorized-principal.md . If your test proves that a role cannot do something, it will pass here even when it would fail in your account. To their credit, the README https://github.com/aws/aws-emulator/blob/main/README.md says it straight: don’t use it for production workloads or to test security. Listen to the README. RTFM So, should you use it? Hell yes. For what it is meant for. I actually think publishing all of this was the best thing they could have done. Every emulator has gaps. Most of them hide those gaps until you trip over one at 2 AM in your CI pipeline. This one hands you a 19,000-entry map of exactly where the potholes are. That is honest. And honest beats polished, any day of the week. And the fact that the gaps between AWS and its own documentation are in there too? That is a bonus nobody asked for, and everybody should read. Credit where credit is due I have spent most of this post laughing at phantom gateways and five-day TCP timeouts. So let me be very clear about one thing. This is an amazing piece of work. Think about what was actually done here. An emulator for 438 AWS API models. Tens of thousands of fidelity tests, every single one checked against real AWS. Hundreds of agents working at the same time, and more than 12,000 issues found and fixed in about three weeks. And not a line of the code written by a human. I have been around long enough to remember when getting a decent local mock of one service just S3 was a project in itself. People built entire companies on emulating a fraction of this. And the part that impresses me most is not the agents. It is the guardrails. Anyone can tell an agent “go build AWS”. The hard part is building the system that keeps it honest: a test suite grounded in real behavior, an issue process that groups failures by root cause, and a merge gate that says no. That is real engineering, and it is a blueprint a lot of us are going to be copying in the next couple of years. The typos, the celebrities, the horoscope forecasts? Those are the funny bits you get when you look at 19,000 issues. The real story is that someone looked at all of AWS, decided it could be done, built the machine that did it, and then published every single wart along the way. Hats off. Truly. One small tip. If one of the links in this post stops working, the issue probably got fixed and moved from things-that-do-not-work to things-that-now-work . Which is kind of the whole point. Now if you will excuse me, I have to go explain to Rekognition that I am in fact, in fact famous. I would be very interested to hear your thoughts or comments, so please feel free to ping me on Twitter https://twitter.com/maishsk or LinkedIn https://www.linkedin.com/in/maishsk/ .