cd /news/large-language-models/on-robustness-of-prompt-based-semant… · home › topics › large-language-models › article
[ARTICLE · art-170400] src=machinebrief.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

A new arXiv paper (2301.12868v4) reports the first empirical study of the adversarial robustness of Codex, a large prompt-based code language model, finding that state-of-the-art code-language models remain vulnerable to carefully crafted adversarial examples on natural-language inputs in semantic parsing. The authors note that adversarial training, which has improved robustness in smaller semantic parsers, is impractical for large language models because it requires substantial computational resources and expensive human annotation of in-domain semantic parsing data. The paper proposes methods for improving robustness without significant amounts of labeled data or heavy computational resources.

by read1 min views1 publishedNov 16, 2026

arXiv:2301.12868v4 Announce Type: replace Abstract: Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models trained on code have demonstrated superior performance in generating these representations compared to traditional unimodal language models, which are trained on downstream tasks. Despite these advancements, existing fine-tuned neural semantic parsers are susceptible to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, \codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples. To address this challenge, we propose methods for improving robustness without the need for significant amounts of labeled data or heavy computational resources.

── more in #large-language-models 4 stories · sorted by recency
── more on @codex 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/on-robustness-of-pro…] indexed:0 read:1min 2026-11-16 · —