On December 17th, 2029, the agentic transport system in London failed. The dynamic bus route system, launched to mass acclaim only a year earlier, routed 174 driverless buses to 29 Acacia Avenue a cul-de-sac in a small suburb in Croydon. The ensuing gridlock took days to clear and caused 1.5 million journeys’ to be lost.
The inevitable post-mortem finally returned almost a year later. So, what happened?
The Agentic Transport system
As self-driving transport became mainstream, static bus routes were an obvious optimisation for London’s mayor. Capacity of buses was rarely 100% and with smart devices sharing geo-location information and predictive models identifying flows, LaTe (London’s AI for Transport - shouldn’t have let humans vote on the name) felt sure there was an optimisation to be made.
LaTe was launched in late 2027 as a data-driven approach to improving transport decisions. Initially, a small team of humans were in the loop, analysing decisions with the help of LaTe and finding a series of wins. Bus routes changed, reflecting the different transport habits of Londoners.
Dynamic routing was introduced a year later. With programmable signage, and the evolution of the transport card to include current location and destination, a transport system emerged that was genuinely magical. Buses appeared when you needed; travel times in London plummeted.
As transport planners became more comfortable, the level of human oversight decreased. It began as waving through new ideas, and then the cutting of staff until finally in early 2029 the decision was made to grant full autonomy to LaTe, signed off by the mayor herself.
LaTe was set up to minimise transport time, maximise capacity and maximise profit.
The Post Mortem
Answers were needed, and popular AI critic Dame Sarah Stone was appointed to lead the investigation.
Her report ran to some 340 pages. I’ll try and save you the trouble of reading all of it.
The problems started in the morning. At 08:40 on the 17th December an over-eager workman took out a fibre cable, and this knocked out the primary pipeline for location data for south London. Of course, LaTe had seen this scenario many times. When live data failed, it simply fell back to historical patterns of Londoners’ movements. It might not be optimal, but it sure beat those quaint fixed routes.
Unfortunately, some of the learning data wasn’t really about Londoners at all. In 2027, the developers had built a simulation to test the routing engine. The model required a home address, and for reasons that no-one could explain this had been hard-coded to be 29 Acacia Avenue (rumours swirled that the agentic engineer was a Bananaman fan). The simulation ran for months of compute time, and everyone of the simulated journeys started there. As far as LaTe’s history was concerned, it was the busiest home in London.
29 Acacia Avenue happened to be a real place. At 9:05am, Joan Okafor took a photo of the first bus arriving. It couldn’t turn around, so it waited. By 09:30 there were eleven buses. A crowd started to form. By 10:00 there were more buses, and the local news had turned up.
LaTe hadn’t malfunctioned. One bit of bad data sent a bus or two. And then people started to turn up to see buses stuck in a cul-de-sac. Which added more.
Then the influencers arrived. An enterprising young chap with a few hundred followers made some terrible meme “you wait for a bus …” and before you know it that was re-yeeted and amplified across social networks.
As more people arrived, the fibre outage was fixed. And now all LaTe saw was signal around Acacia Avenue. The crowds travel card showed a location and no destination, which is exactly what someone waiting for a bus looks like.
Maximise capacity? Where better to send buses than a gathering of thousands?
Maximise profit? Dynamic pricing. Acacia Avenue was out in the suburbs AND now a genuine destination.
Minimise transport time? The crows weren’t going anywhere, so their journeys took no time at all. All three objectives agreed, and no-one human was left to do anything simple.
By the time central control had noticed (thanks to a phone call from a journalist) 174 buses were stuck and the batteries were drained. Croydon was gridlocked which took days to clear with an estimated cost to the taxpayer of millions.
The report cleared LaTe of any wrong-doing and the principal action item was to return to an era of human oversight (and of course, remove 29 Acacia Avenue from the test-data).
Back to Reality
Sooner than we think, we’re going to be in a world where agents control part of our lives, be it transport infrastructure, warehouse restocking or even producing the food we need to survive.
These systems will go wrong.
Oversight will gradually be removed because it’s seen as an overhead. Humans are slower than AI and they probably will agree with it most of the time. Unfortunately, overhead is a lot easier to measure than “# of catastrophic failures averted”.
If human’s had been involved, they could have created a guardrail. They might have even introduced deliberate friction to slow the system down. The pace of change should be no faster than the pace of understanding the impact. When we build agents, we have a responsibility to decide what they must never do, and to keep someone (human!) accountable for those limits long after they’ve stopped looking useful.