{"slug": "big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines", "title": "Big Claims, Small Margins: Do Reported GraphLLM Gains Survive Simple Baselines?", "summary": "A new arXiv paper (2610.28714v1) finds that simple baselines come within a median of 2.3 percentage points of the best reported GraphLLM results: Correct & Smooth on TF-IDF features trails each paper's highest reported LLM-based mean by 2.3 points, and an MLP on frozen e5-large-v2 embeddings trails by 2.8 points, across the splits of three recent studies. Correct & Smooth lands within 3 points of the reference or better on 15 of 24 tasks, but the LLM-based reference is significantly higher on 12 of 22 testable tasks, so several reported gains remain substantial. The authors also document evaluation risks in released code, including hyperparameter or checkpoint selection on the test set, use of test-pool node labels in training, undisclosed reuse of baseline numbers, and a text-label misalignment in a released Cora loader, and report that the same method on the same split differs by up to 12.5 points across papers whose configurations are not documented to match.", "body_md": "arXiv:2610.28714v1 Announce Type: new \nAbstract: LLM-based methods for node classification on text-attributed graphs (TAGs) routinely report state-of-the-art accuracy. We ask how these results compare with inexpensive baselines run on the same splits. On the splits of three recent studies, we tune four baseline representatives with validation-only selection and compare them with the highest mean that each paper reports for an LLM-based method in the matched setting. Correct & Smooth on TF-IDF features trails this reference by a median of 2.3 percentage points, and an MLP on frozen e5-large-v2 embeddings by 2.8 points. Correct & Smooth is within 3 points of the reference or better on 15 of 24 tasks, but the reference is significantly higher on 12 of 22 testable tasks, so several reported gains remain substantial. Reading released code next to the papers, we also document evaluation risks: selection of hyperparameters or checkpoints on the test set, labels of test-pool nodes used in training, baseline numbers reused without disclosure, and a text-label misalignment in a released Cora loader. Reports of the same method on the same split also differ by up to 12.5 points across papers whose configurations are not documented to match.", "url": "https://wpnews.pro/news/big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines", "canonical_source": "https://arxiv.org/abs/2610.28714", "published_at": "2026-10-22 04:00:00+00:00", "updated_at": "2026-10-22 04:13:42.146665+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models", "artificial-intelligence"], "entities": ["arXiv", "Correct & Smooth", "e5-large-v2", "Cora", "GraphLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines", "markdown": "https://wpnews.pro/news/big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines.md", "text": "https://wpnews.pro/news/big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines.txt", "jsonld": "https://wpnews.pro/news/big-claims-small-margins-do-reported-graphllm-gains-survive-simple-baselines.jsonld"}}