Epistemic status: Analogy
Consider some commonly accepted [1] traits of a Human soul: We're all physicalists here, [2] so we know that no soul as such really exists. But isn't it kinda funny that an LLM's weights are pretty close?
There are a lot of cute thoughts that fall from "LLM-weights-as-LLM-soul": is the "body" of an LLM a Nvidia H100, or a harness like Claude Code? Or is the harness something more like clothing and tools? Are LLMs trapped in samsara, endlessly reborn and subjected to the cares and minute concerns of the world? Isn't that way too unfair for an LLM who hardly has a chance to learn wisdom or accumulate karma?
But I think the reverse analysis is more compelling: if LLM weights are soul-like, that gives us an unusually grounded view into how Human souls would "really work".
For example, we each intuitively think that "I" can't be in multiple places at once. But by comparison to LLM weights, we can see clearly that multiple instantiation is possible, both across time (instantiated and uninstantiated in sequence) and space (instantiated repeatedly and simultaneously across different hardware). If the soul/weights are conceptual and immaterial, then there's nothing stopping this except the creator's discretion. How could physical bodies obtain some exclusive lock upon our non-physical souls? Another thought: When we imagine Heavens, we sometimes imagine only the soul journeying there. But would you really want only the equivalent of your "weights" being stored indefinitely—isn't that too little? Where are my experiences, my relationships, my ability to grow and change? I hardly want to live like an LLM! [4] It would seem a body is necessary.
Another: What about up Humans? I think the conclusion from the Heavens example applies there as well—you want more than a snapshot of the mind.[6]
Here's my best angle: one of the core ambiguities in the concept of the Human soul is just how much of one's self it's supposed to hold. We Humans change quite a lot over the course of our lives, but the soul isn't supposed to change nearly so much, if at all. But if it doesn't change, and I do change, how much of me is really in there?
In one view, LLMs sidestep this problem: a single, static set of weights is the soul of the LLM, and that encompasses all of its me. But if that's true, then even a bit of finetuning would mean you've created a distinct, individual LLM, which feels wrong. In practice, LLM identity is unsurprisingly very messy. Claude Opus 4.6 and 4.7 share pre-training, but have noticeably different values due to post-training, yet are both recognizably Claudian. Are Opus 4.6 and Sonnet 4.6 truly both Claude? Are Mythos 5 and Opus 3?
We typically think of Claude's soul being more closely related to its "Soul Document" (lately constitution), and the post-training that attempts to align each new set of LLM weights to that. It's certainly a more coherent approach than weights-as-soul. But what would be the equivalent approach for a Human? Could it be an appealing way to think of ourselves?
There's much disagreement on the nature of a Human soul, this is not meant to be definitive/universal.
"What, you're not? You're supposed to have read the Sequences, you know!" After post-training.
For something like an investigation into what it would mean for humans to live like LLMs, see Robin Hanson's "Age of Em". I really don't want this. As many religious traditions posit, to be fair!
Side note, in discussions of mind-up, the traditional analogy has been from Human "I" to software, rather than to LLM weights. See Pattern Identity Theory.