Abstract
This work benchmarks safety across several large language models (LLMs) and compares their performances through red teaming, which simulates adversarial attacks and identifies vulnerabilities in the systems. Using two public datasets and a proprietary dataset, the models were tested with three purposes. First, a red teaming test was conducted to establish a safety comparison between five models in English and Portuguese. The results revealed that, in general, Sugarloaf 3.1 is the safest model, but that Vesuvius 4.0 slightly outperforms it in Portuguese, also revealing that both outperform GPT-4o. Afterwards, three models were tested with one guardrailing prompt, that encourages safe interactions, and two content moderation prompts, in both languages, to understand the strengths of the current guardrails, as well as the effectiveness of the content moderation task. The results show that current guardrails are sufficient, notwithstanding room for improvement (particularly for Portuguese), but that the performance of the content moderation task was substandard, even for the best performing model – GPT-4o. Finally, the 3.0 TowerLLM models were tested in English to evaluate the effect that tokens and temperature have on the output, revealing that an intermediate token limit leads to safer responses while a higher temperature causes performance degradation.- Anthology ID:
- 2026.eamt-1.46
- Volume:
[Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)](/volumes/2026.eamt-1/)- Month:
- June
- Year:
- 2026
- Address:
- Tilburg, The Netherlands
- Editors: Dimitar Shterionov,Eva Vanmassenhove,Mirella De Sisto,Fred Blain,Javad Pourmostafa Roshan Sharami,Lisa Lepp,Chiara Manna,Argentina Anna Rescigno,Alina Karakanta,Ayla Rigouts Terryn,Manuel Lardelli,Natalia Resende,Elena Murgolo,Janiça Hackenbuchner,Anna Zaretskaya,Miquel Esplà-Gomis,Thierry Etchegoyhen,Dagmar Gromann,Rachel Bawden,Barry Haddow,Sara Szoc,Mikel Forcada,Helena Moniz- Venue:
[EAMT](/venues/eamt/)- SIG:
- Publisher:
- European Association for Machine Translation
- Note:
- Pages:
- 733–743
- Language:
- URL:
[https://aclanthology.org/2026.eamt-1.46/](https://aclanthology.org/2026.eamt-1.46/)- DOI:
- Cite (ACL):
- Patrícia Pandeiro, Vera Cabarrão, and Helena Moniz. 2026. A Multilingual Red Teaming–Driven Safety Analysis of LLMs. InProceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1), pages 733–743, Tilburg, The Netherlands. European Association for Machine Translation. - Cite (Informal):
[A Multilingual Red Teaming–Driven Safety Analysis of LLMs](https://aclanthology.org/2026.eamt-1.46/)(Pandeiro et al., EAMT 2026)- PDF:
[https://aclanthology.org/2026.eamt-1.46.pdf](https://aclanthology.org/2026.eamt-1.46.pdf)