Here's a claim that might sound extreme at first: AI hallucinations will never be fully solved by software alone. I'm not saying this to be provocative — I'm saying it because the hardware itself may be part of the problem.
The Usual Suspects
Most discussions about AI hallucinations revolve around algorithms and data — the nature of statistical prediction, data contamination, reasoning failures. All valid. But I want to try a more fundamental angle.
The Core Observation
Digital computers built on the von Neumann architecture are physically discrete binary logic. Human natural language is continuous, fuzzy, and context-dependent.
The semantic signals we input must be encoded into discrete 0/1 sequences to be processed by hardware. The vast gray area of human cognition — uncertainty, doubt, conditional states — has no native physical representation at the hardware level. There is no "uncertainty register," no "suspend judgment" instruction.
All confidence estimation, self-reflection, and uncertainty quantification in today's LLMs are software simulations built on top of binary bits. At output time, it still picks the single most probable path.
The Hard Question
If this framing holds, could current alignment and interpretability research be treating symptoms at the software layer without addressing a deeper structural mismatch? I'm not dismissing this work — it's the most practical and effective direction we have. I'm just curious: if there is a fundamental gap between physical design and input modality, can software alone fully bridge it?
Objections I've Considered
Counterargument My response
"NNs use float32/float16, not raw binary" True, but those are still discrete approximations over finite precision. You're simulating continuity, not embodying it.
"Turing completeness implies computational equivalence across paradigms" Computational equivalence doesn't mean representational equivalence. The gap isn't about what you can compute — it's about what you can preserve during encoding.
"Biological neurons also use discrete action potentials, yet handle ambiguity fine" This is the strongest objection. My tentative response: the physical substrate matters. Neurons fire in analog circuits, not binary switches. The action potential is discrete, but the context it sits in is not.
What This Means for Alignment
If there is indeed an irreducible loss of semantic information when continuous meaning is compressed into discrete tokens, then: Alignment work is still valuable — it's the best tool we have right now.
But it may be treating symptoms — fixing outputs rather than addressing the encoding-level loss.
And it may hit a ceiling — if the loss is baked into the physical substrate, no amount of software-layer patching will fully recover it.
Not a Conclusion — an Invitation
This is not a settled argument. I'm not saying "this is the answer." I'm saying: is this a question we should be asking more seriously?
Engineers will keep building. Researchers will keep aligning. That's good. But if there's a fundamental gap between discrete hardware and continuous semantics, we should at least be aware of it — even if we don't yet know how to close it.
Related project I'm working on: a software-layer framework called Yellow Light Mechanism that tries to handle uncertainty in the output stage, as a practical step while these deeper questions remain open.
Originally posted as Discussion #24950 on the llama.cpp repository. This is an adapted version for a wider audience.