Signals and Queries: Token-Efficient Agentic Context Discovery

A paper on arXiv (2610.29326v1) introduces Signals and Queries (S&Q), a first-stage relevance filter built for Parlant 4.0 that removes at least 30% of candidate policies before downstream LLM evaluation while preserving high recall. S&Q has an LLM generate activating examples and hard negatives offline for each policy, then trains an independent logistic classifier on their embeddings so relevance is predicted via embeddings and vector dot products with no per-turn LLM calls. Tested against 9,144 LLM-judge relevance labels covering 254 τ²-bench conversation prefixes and 36 airline policies, S&Q achieved higher precision than top-K and cosine-threshold retrieval at every evaluated recall level, with consistent results across two embedding providers.

arXiv:2610.29326v1 Announce Type: new Abstract: User-facing LLM agents must identify which behavioral policies apply at each turn, but evaluating every policy with an LLM is costly. We introduce Signals and Queries S&Q , a lightweight first-stage relevance filter developed for Parlant 4.0 that removes at least 30% of candidate policies before downstream evaluation while preserving high recall. For each policy, an LLM generates activating examples and hard negatives offline. An independent logistic classifier trained on their embeddings then predicts relevance using embeddings and vector dot products, with no per-turn LLM calls. This shifts expensive relevance evaluation to a smaller candidate set, while allowing each policy's classifier to be updated independently. Evaluated against 9,144 LLM-judge relevance labels covering 254 $\tau^2$-bench conversation prefixes and 36 airline policies, S&Q achieves higher precision than top-K and cosine-threshold retrieval at every evaluated recall level, with consistent results across two embedding providers. These findings support inexpensive, per-policy classification as a practical first stage for reducing downstream evaluation workload in customer-facing agents.