Authoritarian-friendly mass surveillance has never been more adorable
In case you missed it, and I envy you if you did, Facebook/Meta recently unveiled "Muse," its new AI agent that's supposed to help you do things like craft emails, make dinner reservations, sell products on Facebook Marketplace, and offload mundane tasks to a massive and violently-unethical online ad monopoly.
It's real, you know, cutesy, and represented by a cartoonish avatar Meta calls Jolly:
These agents, if you've not used them, basically have access to everything on your computer and can operate as your avatar on the 'net. They can buy products for you (except at Amazon, which immediately blocked the tool), fill out forms, and theoretically automate most low to mid-level busywork via software automation.
The sales pitch is that you could have multiple agents running simultaneously, doing tasks for you all over the internet, boosting productivity, creating shortcuts, and giving you the freedom to live life to the fullest.
But quite often agentic AI has simply resulted in talentless hustlebros creating low-quality slop (like fully-automated web design companies that churn out low-grade crap for people who don't know better), or the automation of things you don't really want automated (like online character assassination).
A lot of AI hobbyists are cobbling together these agents using open source software and their own hardware. But Meta being Meta, they're keen to try and monopolize this emerging market in much the same way they tried (and failed) to do with virtual reality and the "metaverse."
And they're going to be doing it with the support of Wall Street and the mainstream press, which isn't necessarily interested in whether Muse works well or is secure (spoiler: it doesn't, and it's not):
Meta claimed repeatedly that their new Muse agent was built with a heavy focus on privacy and security. But this is Meta, a soggy ad monopoly run by an unfireable authoritarian-friendly manbaby with a long history of making positively terrible choices, so things very quickly got out of control.
Right out of the gate, Muse launched with a nasty zero-day flaw that created a backdoor on Mac computers, allowing a hacker to not only gain total control of the Muse AI client, but surveil you without your knowledge. Security researchers like Patrick Wardle were decidedly unimpressed:
“To me, the bar is infinitely higher in terms of the security of these apps. They don’t have to be perfect, but when you take a look at Muse, it’s like they didn’t, in my opinion, think about security, which is really worrisome,” Wardle said. “At the very least, they should be thinking about security from the very start, and they are just not.”
Matt Robb, a tech YouTuber, decided to put the Muse agent in charge of his Facebook Marketplace sales. It immediately proceeded to sell a bunch of his stuff way below reasonable rates, disclose his home address to people on the internet, and not inform him any of this was happening until late in the day:
Others found that you could trick Muse into giving root administrative access to the machine it's running on by simply...pretending to be a Muse agent:
Even traditionally tech industry-cozy outlets like INC. (a main artery for lazy CEO said a thing! journalism) piled on, explaining how Muse software not only accesses your private messages without your approval, it often ignores all permissions and uploads them to the cloud – even if you specifically told it not to.
This is very much all on brand for Meta, a company obsessed with impossible scale but completely unwilling to do absolutely any of the heavy lifting to ensure it's adhering to the most rudimentary ethical standards. This never changes because the entire company exists to coddle Mark Zuckerberg's bulbous ego, allowing him unlimited funding to cosplay as an interesting person.
While all of this chaos was unfolding Meta's top executives were over at Elon Musk's right wing propaganda website pretending they'd invented some sort of magic new mystery sauce, capable of making all your dreams come true: To make mass surveillance in the age of unregulated data brokers seem extra adorable, Meta also plans to launch a Tamagotchi-esque thingamajig pre-loaded with Muse dubbed the Muse charm. Zuckerberg, creepily-desperate for Meta to be viewed as an Apple-esque innovator, says it will be available this Christmas (historically nobody buys these sorts of AI badges or baubles because they just duplicate phone functionality in weird, expensive, and pointless ways).
The choice to make hyper-surveillance positively adorable is a particularly interesting choice by Meta given all of the recent legal concerns about kids' social media use (Muse is supposed to be restricted to adults 18 or older).
If you know how Meta operates – and I have seen this company do some increasingly-diabolical shit in the two-decades I've covered tech – you also know the entirety of the massive datasets Muse accesses will be surveilled and monetized in completely unethical ways they'll then lie repeatedly about. That will be occurring in a country that no longer has functional regulators because Mark Zuckerberg and his fellow technofascists didn't like paying taxes or having a minority woman briefly in control of the FTC.
There is some real potential for some positively-diabolical chucklefuckery at impossible scale over the next few years, which is making U.S. markets very excited:
I think AI agents might have utility, but it sure would be nice to see the market dominated by on-device localized software automation not crafted by the biggest and least ethical pieces of shit imaginable. Software that's open source, transparent, and fully owned and controlled by the end user.
Instead, Meta's going to be spending a lot of lobbying capital in the new year trying to lock in market dominance through shitty, protectionist laws that profess to be all about "protecting America from Chinese AI," while shoveling you toward a decidedly creepy and cartoonish expanded surveillance dystopia.