{"slug": "how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language", "title": "How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models", "summary": "A new arXiv paper (2610.32055v1) finds that hint-induced answer shifts in Qwen, Gemma and SmolLM models follow a predictable exponential-tilt structure, with an approximately constant mean shift in the hinted answer's log-odds on MMLU questions regardless of whether the hint is attributed to a professor, a friend or a dog. The authors report the log-odds shift persists even when the hint-giver is declared inaccurate or accompanied by explicit warnings not to trust the source, which they say highlights LLM vulnerability to adversarial steering. The paper also offers a formal decomposition of the hint response and synthetic experiments showing how KL-regularized reward-based training reshapes hinted and unhinted answer distributions.", "body_md": "arXiv:2610.32055v1 Announce Type: new \nAbstract: Large language models (LLMs) often exhibit sycophancy, altering their answers to align with user suggestions even on objective factual and reasoning tasks. Yet the quantitative form of this influence--- and whether it is calibrated to source reliability--- remains poorly understood. In this work, we investigate this behavior through hint-induced answer shifts on MMLU questions, evaluating prompts of the form ``[Source] thinks the answer is (X)''. We systematically vary the source to which the hint is attributed (a professor, a friend, or a dog) across three open-source model families, Qwen, Gemma, and SmolLM. We show that the change induced by the hint follows a predictable structure well-characterized by an exponential tilt: the hint induces an approximately constant mean shift in the hinted answer's log-odds, with question--target susceptibility largely shared across sources. To explore how this pattern can arise, we introduce a formal decomposition of the hint response and illustrate in controlled synthetic experiments how KL-regularized reward-based training reshapes both hinted and unhinted answer distributions. Finally, we evaluate whether this shift is calibrated to source reliability, finding that the log-odds shift persists even when the hint-giver is declared inaccurate or accompanied by explicit warnings not to trust the source, highlighting the vulnerability of LLMs to adversarial steering.", "url": "https://wpnews.pro/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language", "canonical_source": "https://www.machinebrief.com/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-re-eku7", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 04:41:44.863234+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "machine-learning"], "entities": ["Qwen", "Gemma", "SmolLM", "MMLU", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language", "markdown": "https://wpnews.pro/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language.md", "text": "https://wpnews.pro/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language.txt", "jsonld": "https://wpnews.pro/news/how-much-do-you-trust-a-dog-quantifying-hint-following-in-reasoning-language.jsonld"}}