I think I understand your point better now.
I agree that AI should not be used for emotional support in its current state, especially for children or vulnerable users. In that kind of situation, people can become overly trusting, treat the model like an authority, and be harmed by inaccurate or unstable responses.
Where I am less certain is on the architecture question. I am not convinced that the problem is specifically that Transformers or LLMs are inherently unusable for real-world support roles in principle. To me, a large part of the issue is also training data, fine-tuning, safety design, and how the system is deployed.
For example, uncensored and safety-tuned models can share the same basic architecture while behaving very differently depending on training and alignment. And from what I have seen, newer models do appear to be more cautious than older ones, even if they are still not fully safe and can still be jailbroken. So I think I agree with you on the present safety problem, but I am less convinced that this proves the entire architecture can never be improved enough for safer use in the future.
What I am still unclear on is your exact conclusion. Are you saying these systems are unsafe in their current form, that the underlying architecture is fundamentally unsuitable for this kind of use, or that we need to change how people interact with AI?