A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification

Researchers at ACL 2026 introduced a method to reuse LLM hidden states for classification tasks, training lightweight probes on token- and layer-selective representations to eliminate separate guard models. Their two-stage aggregator, tested on Llama-3.2-3B, GPT-OSS-20B, and Qwen3-30B-A3B, achieved competitive accuracy with lower latency and VRAM usage compared to traditional pipelines.

Abstract Production LLM systems often rely on separate models for safety and other classification-heavy steps, increasing latency, VRAM footprint, and operational complexity. We instead reuse computation already paid for by the serving LLM: we train lightweight probes on its hidden states and predict labels in the same forward pass used for generation. We frame classification as representation selection over the full token×layer hidden-state tensor, rather than committing to a fixed token or fixed layer e.g., first-token logits or final-layer pooling . To implement this, we introduce a two-stage aggregator that i summarizes tokens within each layer and ii aggregates across layer summaries to form a single representation for classification. We instantiate this template with direct pooling, a 100K-parameter scoring-attention gate, and a downcast multi-head self-attention MHA probe with up to 35M trainable parameters. Across safety and sentiment benchmarks our probes improve over logit-only reuse e.g., MULI and are competitive with substantially larger task-specific baselines, while preserving near-serving latency and avoiding the VRAM and latency costs of a separate guard-model pipeline. Multi-backbone experiments on dense and mixture-of-experts architectures Llama-3.2-3B, GPT-OSS-20B, Qwen3-30B-A3B confirm that these findings generalize beyond a single model family.- Anthology ID: - 2026.acl-long.1955 - Volume: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers /volumes/2026.acl-long/ - Month: - July - Year: - 2026 - Address: - San Diego, California, United States - Editors: Maria Liakata /people/maria-liakata/ , Viviane P. Moreira /people/viviane-p-moreira/unverified/ , Jiajun Zhang /people/jiajun-zhang/unverified/ , David Jurgens /people/david-jurgens/ - Venue: ACL /venues/acl/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 42226–42239 - Language: - URL: https://aclanthology.org/2026.acl-long.1955/ https://aclanthology.org/2026.acl-long.1955/ - DOI: - Cite ACL : - Gonzalo Ariel Meyoyan and Luciano Del Corro. 2026. A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification https://aclanthology.org/2026.acl-long.1955/ . In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers , pages 42226–42239, San Diego, California, United States. Association for Computational Linguistics. - Cite Informal : A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification https://aclanthology.org/2026.acl-long.1955/ Meyoyan & Del Corro, ACL 2026 - PDF: https://aclanthology.org/2026.acl-long.1955.pdf https://aclanthology.org/2026.acl-long.1955.pdf