Apple broke iCloud sign-in again. This time a customer found it before I did, and a software factory had the fix before I woke up.
A month ago I wrote about Apple locking us out and it took two AI agents and 45 hours. This time it took one sentence in a feedback form and under a day. The interesting part is everything that happened while nobody was at the keyboard.
On Wednesday, 18 November, at 23:34 UTC, every new Apple ID sign-in on airtaghistory.com started failing. I was asleep. The last one that worked had gone through at 19:57.
Nothing paged me. Nothing in Grafana went red. The polling of already-connected accounts kept running, so the dashboards looked healthy and existing customers kept getting their location history. Only people trying to connect a new Apple ID hit the wall, and from the outside that wall looked like a wrong password (a user error).
Around noon the next day, a customer paid for a plan, tried to connect their Apple ID, and got an error. They opened the Feedback form I had added to the site a few weeks earlier and typed one sentence: they had paid, it did not connect, and the error number was 22410.
By the time I read that sentence, a Claude Code agent had already read it too, pulled the production logs, confirmed that every sign-in since 23:34 the night before had died with the same Apple error code, drafted a reply to them, and started on the fix.
This post is about that pipeline, because the pipeline is the thing worth copying. The Apple detail is at the end for the people who sign in to iCloud from code and are staring at -22410 right now.
What broke this time
If you read the October post, you know the shape. Apple’s sign-in requires “anisette” headers, a device attestation that says “this request comes from a real Apple device”. In October Apple started refusing anisette generated by an emulated Android library whenever the client claimed to be a Mac. My fix was an old 2017 MacBook Pro in a drawer, serving real macOS anisette headers over a Cloudflare tunnel.
On 18 November Apple tightened the same screw one more turn. Alongside the anisette headers, every sign-in sends a device id, X-Mme-Device-Id. Until Wednesday Apple did not care what it was. From Wednesday it has to be the hardware id of the exact Mac that produced the anisette. Mine was a random UUID, minted fresh for every sign-in, on purpose, so that the one Mac in my drawer would not be identifiable to every account signed in through it. Apple no longer allows that choice.
The error for a mismatch is -22410, “This action could not be completed. Try again.” Trying again does not help.
The customer was the alert
Here is the part I am least proud of, and the one I would most like you to take away.
In October I wrote: fix the message before the bug. My code had been mapping Apple’s -80045 to “bad credentials”, so for a day and a half users were told they had typed their password wrong. The first pull request that landed back then was the one that stopped blaming the user.
I fixed -80045. I did not fix the bucket. Every Apple error code my exporter did not recognise still fell into bad_credentials, and bad_credentials is the user’s fault, and the user’s fault does not page anyone. So when a brand new code showed up on Wednesday night, it went straight into the one bucket that is designed to be ignored. It stayed there until 16:40 the next day, when the first pull request of the day landed and taught the exporter that -22410 is Apple refusing, not the user mistyping.
What did fire was a human. They had just paid, and they were motivated.
I had built the Feedback form with exactly this in mind.
The customer writes whatever they want, usually one line. The form appends everything I would otherwise have to ask for: their user id, their plan, the git commit running in production at that moment, the page they were on, whether their Apple ID is connected and in what state, when each of their AirTags last reported, their time zone, and so on. All of that lands in the notification email and in a row in the feedback table. One row is a complete bug report, including the parts the customer does not know they know.
The lesson is not “add a feedback form”. It is that the feedback form is part of your monitoring, and it is the only part that catches the failures you did not know how to classify yet. Build it like a sensor, not like a mailbox.
The factory picks it up
I use the term “software factory” with some reluctance. It is established enough that I will not fight it.
Mine is small. Claude Code runs against the repository with a set of written procedures, which Claude Code calls skills. One of them watches for new feedback and works the case. It is a markdown file, about two pages, and most of it is rules about what the agent is not allowed to do.
The workflow for a problem report goes like this. Read the row from the production database, read-only, through one sanctioned script. Decide what kind of report it is. Then try to confirm it, and the skill is specific about what “confirm” means: a log line, a database row, a metric, or a reproduction. A plausible story is “not yet confirmed”, and “not yet confirmed” gets a different reply. Count how many other users hit the same thing. Draft the reply to the customer in their time zone, in plain words, and never send it. Only production sends email, and only I press send. If it is a confirmed bug, open a branch in an isolated worktree, write the failing test first, fix it, open a pull request, and stop. The skill does not merge.
For feedback number seven, the agent found the log lines in minutes. Every POLLING_LOGIN since 23:34 had ended in the same -22410 at the first step of Apple’s sign-in. One other user had hit it earlier that day and left without connecting. It wrote a reply that said we had confirmed the problem and were working on a fix, with the times in their time zone, and put it in my drafts. Then it hit the limit of what logs can tell you. Logs said Apple was refusing. They did not say why.
Bisection without a password
This is the part I would have done badly on my own, and the part where the agent earned its keep.
Apple’s sign-in is a two-step SRP exchange. The first step, init, sends the username and a public key and gets back a salt and Apple’s public value. The password is only used in the second step. The error we were getting came back from the first step.
That is a gift, for two reasons. First, you can hit the first step with a username and no password, which means I never had to type my Apple password into anything the agent could see. Second, a failed first step does not count as a failed sign-in attempt, so you can hit it two hundred times without locking the account.
The agent wrote a small probe that does only step one and prints Apple’s response code. Then it started bisecting, in the medical sense: change one thing, run the probe, keep or discard.
Different anisette host: my own laptop’s headers instead of the drawer Mac’s. Still -22410. So it was not the server.
Python instead of Rust, a different TLS stack entirely. Still -22410. So it was not the HTTP client.
User-Agent, client info string, context ids, the RINFO field, the local-user id, the various flags in the request body, the version string. Dropped them one at a time, replaced them with garbage one at a time. Garbage anisette gives -80044. No anisette at all gives -80009, “MID is invalid”. An account that does not exist gives -1. Every one of those codes was different from ours, which told us something useful: -22410 comes after Apple has validated the anisette and found the account. It is a check that runs late, on something we were sending.
That left the device id. Random UUID: -22410. The real hardware UUID of the Mac that produced the anisette, the one akd itself sends: 0. Pass. Lowercase version of the same id: -22410 again, which is the kind of detail that tells you that you have found the real check and not a coincidence.
A search of the community threads turned up one earlier sighting, from October: someone getting -22410 after pairing anisette from a Windows machine with a Mac client label. Nobody had named the cause. In hindsight it was the same check, seen from a different angle.
The proof that the fix was complete was almost comic. We ran a full sign-in with a deliberately wrong password. It failed with -22406, which is Apple’s code for “wrong password”. Getting told your password is wrong had never felt so good. It meant the first step passed and Apple had actually looked at the credentials.
The fix, and what still needed hands
The code change is small. The helper that runs on the drawer Mac now asks the system for its own device id and serves it as X-Mme-Device-Id alongside the anisette headers. The Rust exporter already preferred a served id over the random one, so it needed only a comment and a test. The Python sign-in path needed a few lines to use the served id and store it with the account, so that later polling presents the same identity.
Two pull requests, one changelog entry, one new test on each side. The agent wrote them, I read them, and I merged them.
The parts that needed a human were the parts I had told the agent it could not touch. The helper had to be rebuilt and reinstalled on the Mac in the drawer, which is a different machine the agent does not reach. The password test had to be run by me, because the password is mine. And the first deployment was skipped by the CI system because of a flaky test that had nothing to do with any of this, which the agent then found and fixed in a third pull request.
At 19:44 UTC on Thursday I connected my own Apple ID on the production site. The logs showed init ec=0; complete ec=0. Twenty hours after Apple flipped the switch. The customer got their reply from me, not from the agent, after I had confirmed it worked.
Learnings
Your feedback form is a sensor. Do not make the customer tell you their user id, their plan, or what version they were on. Attach it. The report that catches the failure you have not learned to classify yet will arrive through that form, not through your alerts, because your alerts can only watch for what you already know.
Write the rules before you write the automation. My skill is mostly prohibitions: confirm before you claim, draft but never send, open a PR but never merge, read production but never write it. Those rules are why I can let it start without me. An agent that can do anything needs supervision for everything. An agent that can do five things and knows it needs supervision for one.
Bisect from the cheapest side. The trick that cracked this was noticing that Apple’s first step is free: no password, no lockout, one HTTP request. Before you ask what changed, ask what you can test without consequences, and test that a hundred times.
Fix the bucket, not the code. In October I fixed one error code and left every future unknown code falling into the same silent place. This week I fixed one more code. The bucket is still there. The next Apple error I have never seen will land in “wrong password” again, and that is the pull request I should have written a month ago. Being told about a problem you cannot classify yet is the whole point of being told.
The first pull request to land this time, again, was not the fix. It was the one that made this error wake me up next time. The one that makes every unknown error wake me up is still to come.