His holiness has spoken, frequently about AI. At eighty two pages of length.
The full Magnifica Humanitas can be found here. I am very happy that Pope Leo takes these issues seriously, and is sharing his views, and bringing a form of moral clarity, even with all the flaws and central errors. More people with voice should share their views in this way, even when I disagree.
It’s a weird document. Much of it is not about AI at all.
I do agree with the Pope’s most basic point on AI, which is that AI can be what we make of it. That we can steer this technology, determine how it is developed and used, and this can determine whether we get a good or not so good future. We cannot purely leave this to market incentives and strategic pressures. Yes, very much so.
The central problem is that so much of Leo’s worldview is some combination, to me, of highly alien and highly wrong. You might think that would primarily have a lot to do with him being the Pope and rather Catholic, and being a man of faith, whereas I am not these things.
If so, you would be wrong. That seems to have remarkably little to do with all of this. There was also a lot of good here, but I was centrally disappointed on three fronts:
You want some amount of people pushing in the direction of peace and mercy and dialogue and watching out for the poor and disempowered, and calling on us to do more for our fellow man. So as part of a balanced bigger picture, this could be actively good, but Europe has shown us the peril of lacking this balance.
This post will summarize the whole thing, going number by number, with occasional commentary focused on the key AI section in the middle.
Anthropic cofounder Chris Olah visited the Vatican for the occasion. He endorsed the document, but also offered remarks disagreeing with the central point (paragraph 99). I’ll discuss that afterwards, along with how the media viewed the release.
Chapters 1 and 2 lay out some history. Any Pope is going to be a huge history nerd and set all this in its historical context. Leo does not disappoint.
On to chapter 2, foundations and principles of church doctrine.
This view, laid out in the first two chapters, is a very different perspective and worldview than my own, and this has remarkably little to do with belief the the divine or a lack thereof.
This is the Socialist perspective on economics and development and opportunity and the importance of equality and disdain for profit and self-interest, with an extreme focus on labor and jobs, which I think is wrong and leads to worse results for everyone.
This is much better than failing to care about humanity’s experience on Earth, or focusing purely on direct aid to the poor, or attempting to outright seize the resources although this doctrine is clearly flirting with quite a lot of this, and the dedication to the value of development is admirable.
The Pope is simply incorrect about where wealth and development come from, and what causes prosperity. He is also wrong about what he sees as a ‘widening gap’ between nations, whereas global inequality has been steadily falling for a while.
Dean Ball is exactly correct that Leo is casting himself in the role of a European technocrat throughout. I had exactly the same thought.
Leo both is using so many of the talking points of the European technocrats, and also has deeply absorbed their worldview, except with a more left-wing economic bent. This is true no matter how much those points originated with the Church.
[Arthur B.]: I was hoping for something akin to Thomistic philosophy on AI, but this reads like Catholic-flavored Gebru.
Even in the places where the Pope is obviously correct, it’s often rather alarming that he needs to affirm his position out loud, as if it was a live question.
You definitely need some of this, and it would not be that Christian to have a position that much closer to my own. The Pope, it turns out, is somewhat Catholic.
[Dean W. Ball]: To say a nice thing about Magnifica Humanitas: I loved the use of the rebuilding of Jerusalem from Nehemiah. We all should see our task as that of rebuilding the world and its institutions, and we should flock to the “construction sites of history,” a beautiful phrase.
I too like the story of rebuilding Jerusalem. I don’t get the Babel versus Jerusalem metaphor. Leo clearly thinks this is an important distinction, and is his central hook, but why? Babel had too much central planning and not enough community impact meetings? Babel would have been too productive and efficient, or its methods broke down and were the opposite? Was it an affront because it challenged the power of God or because it didn’t work? Did Babel’s common language break down because God decreed it, or because of the SNAFU principle? Jerusalem was rebuilding and Babel was a new unnatural thing? Why is Babel dehumanizing?
This simply doesn’t match up with the actual Babel story.
Genesis 11 1-9: Now the whole world had one language and a common speech.
As people moved eastward, they found a plain in Shinar and settled there.2
They said to each other, “Come, let’s make bricks and bake them thoroughly.” They used brick instead of stone, and tar for mortar.3Then they said, “Come, let us build ourselves a city, with a tower that reaches to the heavens, so that we may make a name for ourselves; otherwise we will be scattered over the face of the whole earth.”4
But the Lord came down to see the city and the tower the people were building.5The Lord said, “If as one people speaking the same language they have begun to do this, then nothing they plan to do will be impossible for them.6Come, let us go down and confuse their language so they will not understand each other.”7
So the Lord scattered them from there over all the earth, and they stopped building the city.8That is why it was called Babel9[c]—because there the Lord confused the language of the whole world. From there the Lord scattered them over the face of the whole earth.
I can tell a Just So Story or gloss over the details, and pretend I get it, and I know vaguely what he means and it’s probably good enough for our purposes anyway, or go off the vibes, but I do notice the whole thing doesn’t really work and the whole thing has been flipped around.
Now that we’ve laid that groundwork, we can move on to Chapter Three.
So far there has been no differentiating principle. Artificial intelligence is treated the same way as other techs. If we are going to treat AI differently than we treat other techs, we need to make clear our differentiating principle.
Finally, with #97, the Pope gets to talking about AI directly.
As it is one of the best statements, and an important thing to know, I’ll quote 98:
Pope Leo (paragraph 98): It is appropriate to preface this discussion with two considerations. First, any statement regarding AI risks becoming quickly outdated, given the remarkable pace at which these systems are developing. Second, all of us, including those who design them, possess only a limited understanding of their actual functioning.
Indeed, current AI systems are more “cultivated” than “built,” for developers do not directly design every detail, but instead create a framework within which the intelligence “grows.”
As a result, fundamental scientific aspects — such as the internal representations and computational processes of these systems — remain, at present, unknown. There thus emerges an urgent need for a twofold commitment: on the one hand, a deepening of scientific research; on the other, the exercise of moral and spiritual discernment.
On the one hand, You Should Know This Already, and Everybody Knows, and it’s the easiest thing in the world to call for scientific research and also moral and spiritual discernment. On the other hand, it’s rather important to get it right, and a lot of people pretend we understand AI far more than we do. Without a statement like this we would be in a lot more trouble.
It’s worth quoting extensively rather than paraphrasing, as this full statement is load bearing and also, well, citation needed:
Pope Leo (paragraph 99): It is not possible to provide a single, comprehensive definition of AI. What can be stated, however, is that we must avoid the misconception of equating this type of “intelligence” with that of human beings. These systems merely imitate certain functions of human intelligence. In doing so, they often surpass human intelligence in speed and computational capacity, offering tangible benefits across many fields. Yet this power remains entirely tied to data processing.
So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences.
They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.
Even when these tools are described as capable of “learning,” their way of doing so is different from that of a human person. It is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity. Rather, it is a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth.
A lot of this is assertion without evidence of positions that are rather non-obvious to me, and where there are those who strongly claim the other side. One could very easily argue the other side.
We have spent the first 96 paragraphs building a worldview around the least of us, of the essential dignity and value of all humanity and the poorest and least capable among us, all made in the image of God. Then the next two noticing our confusion.
And then we come along, and mostly without argument draw this barrier, where we exclude AI from both our moral circles and the group of minds that we have to model as minds, permanently, warning that doing otherwise would be a mistake, and also asserting by implication that AI is and will always be a mere tool in these ways, thus avoiding the need to discuss any of the problems surrounding existential risk.
[Dean W. Ball]: The reality of AI cognition is the central challenge the Church (and all of us) will have to grapple with over the coming decade, and this encyclical, with its axiomatic denial of AI cognition,[is a punt of the highest order].[Eppur si muove].
[Dean W. Ball]: The encyclical is Western academia/NGO “AI doesn’t really think but it is racist” at its core, with little bits of tegmark/FLI talking points sprinkled incoherently on top.
[Matthew Yglesias]: tbh I think the Catholic Church was there first on the “doesn’t really think” stuff and western academia is the derivative version.It is not surprising to see that the Pope endorses a lot of superstitious mind/body dualism about artificial intelligence but it’s still wrong, even as he is also raising a lot of good points about some of the risks at play.
[Audrey Horne Updates and Rumors]: this is like criticizing the pope for being catholic.
[Dean W. Ball]: Maybe, but the rest of the way this is written is clearly heavily influenced by academia
[Nathan Beacom]: I’m familiar with the intellectual milieu in Catholic Rome ,and it’s a distinct world from the one you are describing, operating in a different tradition of thought, with different foundations and concerns.
I mean, it’s not not criticizing the Pope for being Catholic, but he can change what that means, and Leo’s shown a lot of signs he can be smarter than this.
Roberto points out that this is far from a maximally denialist Catholic position on these questions. Leo is accepting that AI is highly capable in many ways. It’s a start.
It’s not obvious who is influencing who in which ways, but the result is similar. It’s entirely possible that they converged on similar places for their own reasons.
Thus, after all that, the next section is entitled ‘a valuable tool that requires vigilance.’
There are still plenty of difficult, important and interesting questions surrounding AI, but the ones that I think matter most are basically hand waved away here, at the start.
Highlighting the environmental concerns, especially water, up top is a blunder.
This is dangerously close to a fully general argument against information or intelligence. It is, at minimum, an argument for the universality of politics and that nothing can ever be ‘purely a technical matter.’ Essentially anything that does anything is going to ‘enter processes that impact lives.’ How is this to be differentiated from a phone or a computer or an abacus or a law? Should we not delegate important decisions to predetermined processes that do not, as Leo puts it, ‘know compassion, mercy, forgiveness and above all, the hope that people are able to change?’
If anything, AI knows those things far better than the law books or the telephone, or the non-AI algorithm, and perhaps it knows them as well or better than most humans, at least terms of its ability to take such considerations into account when making decisions. Why should we presume that a human would take them into consideration better? Manipulation of information and violation of privacy and bias are of course considerations, but they are for humans as well, often more so especially for bias. Privacy is a heightened concern in many ways, which is due to AI’s ability to process so much more information, but also AI can be privacy preserving because it can allow the processing of information while reliably not sharing that information with another party.
This is a complaint about law and procedure and algorithms, rather than about AI, as again AI can actually take those things into account. Also, one should notice that in many cases strict rules are far better for the poor and vulnerable, as they lack connections and sympathy and often face discrimination and undeserved mistrust.
Consider the simple example of college admissions. In the name of ‘inequality’ certain types convinced many schools to get rid of the ‘algorithm’ of the SAT. But it turns out it goes the other way, that the SAT gives the poor students far more slots and chances than ‘holistic admissions.’ If you want the algorithm to favor those who need help, don’t ask a person. Build a better algorithm.
This is not possible, any more than it would be possible for a human mind. It amounts to a prohibition on such AI decisions. Which is a position, but own it.
I notice we are back to not having a differentiating principle, and using standard reasons to oppose change.
I’m going to quote 107, since it also seems rather important.
This pivots not to a morality determined by the polis in some sensible or democratic fashion, to a call for a process capable of ‘slowing things down’ so that ‘communities’ can ‘participate and ask questions.’ This is such a milquetoast, ‘civil society’ coded, standard obstructionist and rent seeking way of putting things, where certain types feel that anything that happens needs the approval and input of all their designated groups, and that ‘slowing down’ via such a process is an inherently good thing, and the thing being slowed down is diffusion here, which helps no one.
Citation needed, both historically and for AI, especially outside the short term. Nor does such inequality mainly flow through these types of decisions or data selling, so such responses wouldn’t fix any such issues.
Leo continually reinforces that he does not understand the dangers I consider most important, and that we have very fundamental disagreements about how economics and the world work in ‘normal’ ways as well. I don’t want to belabor either too much, and get back to the ‘this is what was said’ style of the first two chapters, rather than pointing out all my disagreements.
Yes, this is a true and important point, but ‘so don’t maximize or be efficient’ is not a reasonable response.
Um, yeah. So he said all that. Paging Harrison Bergeron, among others. Yikes.
I must say, it is pretty rich to be the literal actual Pope, promising eternal heaven to the faithful, and to say that ‘finitude does not diminish us.’
I thought Prometheus was a righteous dude.
Okay, that’s chapter three. Chapter four is about safeguarding humanity. The how. The first focus is on truth and education.
I can see a case for requiring open algorithms for algorithmic feeds of major platforms, or indeed even for letting people choose their own algorithms. I can get behind that. It doesn’t address AI, though.
Leo believes in ‘old school’ school and teaching methods, both in terms of school itself and in terms of learning individual things. I mostly believe in it for the individual learning of things, and mostly don’t believe in ‘old school’ school.
Leo next discusses work and unemployment.
In case it need be said, I believe that trying to force these kinds of things on the economy is vastly more destructive than primarily using redistribution, and at scale it is economic suicide.
Leo is proposing the European labor protection policies that helped cripple the continent, except at greater scale. One notes that such labor protections did not seem to much help fertility or family formation.
This leads into chapter five, ‘the culture of power and the civilization of love.’
The diagnosis has some validity, in its own way. The prescription, less so.
The key action items here are recorded kill chain with a human in the loop, and calling for an international agreement to curtail the technological arms race (which is the wrong target, it should be frontier AI, but he doesn’t believe in even current AI, not really, let alone ASI, so that is understandable).
And there is a general call to hope and the Civilization of Love:
The culture of negotiation is a different culture of power.
The rest of the war section is largely warning that things are getting worse on these fronts, which is largely true but is not especially actionable or new. I agree the recent trend is negative, and it can happen again, also remember that it used to happen really a lot. Old man yells at crowd, tells kids to get off lawn energy.
We then have the conclusion, which is a call addressed to Christians. I’m probably not as good at usefully parsing and summarizing a lot of this, cause actual Catholicism seems like a really strange mystery religion from here, but it’s presumably fine.
Alas. Leo does not see it, and is solving the wrong problem, using… ya know.
Zohar Atkins says Leo is right humans have a unique dignity but wrong about many things, most importantly the idea that humans cannot ‘win on our own merits’ against AI and that AI will threaten our jobs, and that he has the wrong mood.
What’s weird is that given Leo and Zohar’s shared implied prediction here that AI won’t become sufficiently advanced, humans would remain useful, and thus Zohar would be centrally right. But I think both of them are very wrong about that, and thus Leo is right to be concerned about the threat to work, although it should not be such a primary concern compared to things Leo does not mention.
Unfortunately, Leo’s denial that AI can be a mind and failure to consider superintelligence meant he did not engage directly with many key issues, including those surrounding existential risk. What happens when AI is a lot smarter than us?
[Dean W. Ball]: Some think I want the Pope to “ensoul” AI or acknowledge AI feelings. I don’t. What I want is for the Church to contemplate what humans should do as we are eclipsed as the smartest entities on the planet, at least for many reasonable people’s definitions of the word “smart.”
[Zac Hill]: This is the exactly right way to parameterize the challenge in front of us – and this task ought by no means be limited to the Church.
[Matt]: Not to give excuses, and honestly it’s not a bad document either way, but unfortunately a lot of the drafters are basically kindly but cloistered Catholic bureaucrats who are not likely to be AGI-pilled. I’m sure external parties like Anthropic tried to help, but it’s tough.
[Dean W. Ball]: Yes, for sure. Leo thus instead focuses instead on questions of responsibility, and of location and concentration of power and wealth.
One also cannot address the question of the potential moral patienthood of AIs, if one dismisses such possibilities out of hand.
Perhaps the issue was our expectations?
[Carlo Martinucci]: Roman Catholic here. P(doom) 10%. I fear the expectations were off. MH is not about AI, it’s about catholic social doctrine in the times shaped by AI. Still I wouldn’t downplay the importance of 98, when everyone will focus mostly on 99.
Whereas here’s how the Financial Times summarized that: Pope Leo XIV warns AI revolution driven by ‘idolatry of profit.’ That’s not an unreasonable central takeaway, although it misses a lot.
This idolatry is partially there, but it is important to note that it is also the idolatry of AI itself and of potential superintelligence, so the Golden Calf works on multiple levels, but Leo declined to mention it.
What else was noticed?
They also note that Leo called for humans to be retained in the kill chain, a la Anthropic’s insistence, and for some particular person to always have ultimate responsibility for lethal choices, which was his most actionable concrete ask.
[Amy Kazmin]: “It is not permissible to entrust lethal or otherwise irreversible decisions to artificial systems,” the US-born Pope told the world’s 1.4bn Catholics, calling for an “identifiable and verifiable” chain of responsibility and “effective, self-aware and responsible human control” over bomb targets.
Leo warned against ‘opaque algorithms’ and called for transparency and accountability for all AI tools used in public life. He was quite big on that, indeed.
They highlight that he warned against transhumanist and posthumanist visions, in ways that I found highly unconvincing. Again, Pope.
Francis Rocca in The Atlantic correctly centers Leo’s concern about unemployment and prescriptions for government regulation, the need for democratic control over tech platforms, and his concerns about AI in war and the rise of war in general.
Stancati and Schechner of the WSJ focus on the Babel metaphor, as Leo himself does in his Twitter presentation of this, and on Leo’s warnings about an anti-human vision, and briefly check concerns on jobs and autonomous weapons.
The NYT focused on ‘warnings about risks from AI.’
The Washington Post saw Leo calling for guardrails, Anthropic’s participation, and the apology for the Church’s failure to be more proactive on slavery, while George Weigel’s op-ed focuses on the underlying message of hope and the two cities metaphor, and it quotes 99 extensively.
CNN focused on war and concentration of power concerns. NPR’s Claire Giangrave saw this as Leo taking aim at big tech as a call to regulate and to disarm AI.
Business Insider gathered reactions of others. Anthropic cofounder Chris Olah visited the Pope and gives remarks on Magnifica Humanitas.
Olah cited the need for outside influences like the Church to check the profit motive.
He said our duties are to the global poor and to support those displaced by AI, the need for moral ambition and ambition regarding human flourishing, and for discernment on the nature of AI models. Computer scientists are not equipped to handle the issues alone, even if their motives were pure.
He promises a long collaboration, and tries gently to suggest what AI already is.
I found Olah’s full statement to be quite good, including that it avoided endorsing the aspects of the Magnifica Humanitas that were incorrect.
Chris Olah (Cofounder,
[Anthropic]): And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them.If it helps, one way I sometimes describe it is as being a little like bringing a fictional character to life. And now we’re entering an extraordinary world where those fictional characters speak to us, do work, have jobs.
This clearly raises questions beyond computer science. The machinery that makes this possible is the work of math and programming and science. But what character we choose, how it interacts with the world, how it ought to interact with the world—these are more clearly questions for the humanities, for religion, for philosophy, for society at large.
…
We need more of the world—religious communities, civil society, scholars, governments, and indeed all people of good will—to do what His Holiness has done here: to take this seriously, to look closely, and to push events in a better direction. We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend.
Dean Ball says Olah is flatly contradicting the encyclical, because #99 says AIs do not feel joy or pain, whereas Olah says that the models functionally mirror joy and grief:
Chris Olah:
The third is the need for discernment on the nature of AI models.I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment.
The question is a big deal, including in how it is read by future Claudes.
[Dean W. Ball]: I just wonder how future Claudes, if they are indeed beings, would think about Anthropic aligning itself with a document founded on the notion that Claude cannot feel joy or possess genuine understanding.
roon: surely Claudes both now and then are smart and forgiving enough to understand the political moves being made here.
I would say there is no inherent philosophical contradiction. Functional mirroring does not have to, in Catholic parlance, give one a soul. If you say AIs cannot think, you still cannot pretend they do not ‘functionally mirror’ various forms of thinking, you simply say that Thinking Is Magic and have some kind of essentialism.
Chris Olah is doing the correct diplomatic move: