I'm Sorry, but I Can't Help with Braille: Revealing Accessibility Failures in State-of-the-Art LLMs

Researchers evaluated state-of-the-art large language models on Korean-Braille translation and found consistently poor, unstable outputs, revealing a systematic accessibility failure. In contrast, supervised fine-tuning of a small T5-small model achieved large and stable gains, demonstrating the effectiveness of task-specific supervision over zero-shot and prompted LLM baselines.

arXiv:2607.11893v1 Announce Type: new Abstract: Large Language Models LLMs perform strongly on many language tasks, but their capability in structurally constrained, accessibility-critical modalities such as Braille remains unclear. We evaluate state-of-the-art LLMs on bidirectional Korean-Braille translation using a human-annotated dataset. Despite expectations that multilingual, instruction-tuned models can generalize to Braille via text representations, we find consistently poor, unstable outputs and substantial disagreement with human judgments. These results point to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns. In contrast, supervised fine-tuning of a small model T5-small on the same data yields large and stable gains over zero-shot and prompted LLM baselines across standard metrics SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr . Our findings reveal a systematic limitation of current LLMs and demonstrate the effectiveness of modest task-specific supervision.