{"slug": "tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag", "title": "TIS 2.0: Token Importance Scoring Now Eliminates Position Bias in RAG", "summary": "Token Importance Scoring (TIS) 2.0 eliminates position bias in retrieval-augmented generation, reducing the lost-in-the-middle gap to zero and improving exact match accuracy by 5 percentage points over baseline, according to developer oldman-dev. The existing tis-stage3-ert checkpoint, originally trained for KV cache compression, achieves identical passage reordering performance without retraining, demonstrating transferable importance patterns. New features include a query-aware architecture, speculative decoding integration with 12.5% speedup, and open-source checkpoints on HuggingFace.", "body_md": "****An update to the Token Importance Scoring series — new passage reordering capability, transfer learning findings, and open-source checkpoints.****\n\nFollowing the initial [TIS release]([Presenting TIS (Token Importance Scoring) - A new way to compress KV cache](https://discuss.huggingface.co/t/presenting-tis-token-importance-scoring-a-new-way-to-compress-kv-cache/177429)), we’ve significantly expanded TIS capabilities. The biggest finding: ****a learned importance head trained for KV cache compression also eliminates position bias in retrieval-augmented generation without any retraining**** — suggesting TIS learns generalizable importance patterns.\n\n**## What’s New in TIS 2.0**\n\n**### 1. Position Bias Elimination (Lost-in-the-Middle)**\n\nThe core discovery: reordering retrieved passages by TIS importance scores completely eliminates the LITM position gap while improving overall accuracy.\n\n| Pipeline | EM (early) | EM (middle) | EM (end) | LITM Gap |\n\n|—|—|—|—|—|\n\n| Baseline (no reordering) | 18.3% | 13.3% | 18.3% | ****0.050**** |\n\n| Lexical (TF-IDF) | 16.7% | 18.3% | 18.3% | −0.008 |\n\n| ****TIS (passage reranker)**** | ****21.7%**** | ****21.7%**** | ****21.7%**** | ****0.000**** |\n\n| Oracle | 18.3% | 18.3% | 18.3% | 0.000 |\n\n****Result****: TIS reduces the LITM gap to ****zero**** (matching Oracle) while improving EM by +5pp over baseline. Evaluation on 60 MS-MARCO queries × 3 positions = 180 test cases.\n\n**### 2. Zero-Shot Transfer Finding**\n\nEven more surprising: the existing `tis-stage3-ert` checkpoint (trained for KV cache compression on synthetic data) achieves identical passage reordering performance:\n\n| Checkpoint | LITM Gap | EM (all positions) | Training |\n\n|—|—|—|—|\n\n| `tis-stage3-ert` (KV eviction) | ****0.000**** | ****21.7%**** | None (existing) |\n\n| `tis-passage-reranker` (contrastive) | ****0.000**** | ****21.7%**** | MS-MARCO passage pairs |\n\nThis suggests TIS learns abstract “importance” patterns that generalize across task granularities (tokens ↔ passages) and objectives (compression ↔ ranking).\n\n**### 3. Query-Aware Architecture**\n\n`QueryAwareImportanceHead` extends the base `ImportanceUpdateHead` with:\n\n- Cross-attention query-context interaction\n\n- Per-passage relevance scoring\n\n- RMSNorm output stabilization\n\n- InfoNCE contrastive training on passage pairs\n\n**### 4. Speculative Decoding Integration**\n\nTIS importance bias applied to drafter predictions improves acceptance:\n\n| Condition | Accept Length | Speedup | Notes |\n\n|—|—|—|—|\n\n| No TIS | 5.80 / 8 | 0.644 | Baseline |\n\n| ****TIS depth-scaled**** | ****6.57 / 8**** | ****0.730**** | +12.5% |\n\n(LLaMA-3.1-8B target + LLaMA-3.2-1B drafter, n=30)\n\n**### 5. Improved Benchmarking**\n\nNew evaluation framework with 4 concurrent pipelines:\n\n- `baseline`: original retrieval order\n\n- `lexical`: TF-IDF cosine similarity reordering\n\n- `tis`: learned passage importance\n\n- `oracle`: gold passage always first\n\nEnables direct comparison of ranking quality vs. baseline and oracle bounds.\n\n**## Checkpoints**\n\n****HuggingFace****:\n\n- [`tis-stage3-ert`]([oldman-dev/tis-stage3-ert · Hugging Face](https://huggingface.co/oldman-dev/tis-stage3-ert)): KV compression + passage reordering (no additional training needed)\n\n- ****[`tis-passage-reranker**`]([oldman-dev/tis-passage-reranker · Hugging Face](https://huggingface.co/oldman-dev/tis-passage-reranker))** (new): dedicated passage reordering head\n\n- [`tis-v8b-hard-anchor`]([oldman-dev/tis-v8b-hard-anchor · Hugging Face](https://huggingface.co/oldman-dev/tis-v8b-hard-anchor)): 82% NIAH @ 25% budget\n\n****GitHub****: [GitHub - nitroxido/token-importance-scoring: Public repository for TIS system · GitHub](https://github.com/nitroxido/token-importance-scoring)\n\n**## Key Code Changes**\n\n****New modules:****\n\n- `QueryAwareImportanceHead` — contrastive query-aware scoring\n\n- `CrossAttentionImportanceScorer` — per-passage relevance\n\n- `TISCompositeLoss` — 5-component objective (task + KL + budget + churn + saliency)\n\n- `TISDrafter` — speculative decoding adapter\n\n- Evaluation: `run_litm_with_baselines.py` (4-pipeline comparison)\n\n****Updated:****\n\n- `ImportanceUpdateHead.direct_score()` — explicit named scorer path (evaluation-only)\n\n- `PatchedCausalLM` — Transformers 5 SDPA compatibility warning + generation prefix fix\n\n- `eval/benchmarks.py` — full LITM pipeline with scoring policies\n\n**## Technical Insights**\n\n1. ****Generalizable importance****: Token-level and passage-level importance share underlying patterns. A head trained for KV compression can score entire passage relevance.\n\n2. ****Position bias root cause****: The position effect is strongest at positions where passage content is sparse (middle positions lose context) — not just a positional encoding issue. Reordering addresses this directly.\n\n3. ****Trade-offs****: Improved passage ordering does not guarantee proportional answer quality gains. ~76% of test cases remain incorrect after reordering, indicating answer extraction (not ranking) is the bottleneck in this domain.\n\n**## Reproducibility**\n\n``` bash\n\ngit clone [GitHub - nitroxido/token-importance-scoring: Public repository for TIS system · GitHub](https://github.com/nitroxido/token-importance-scoring.git)\n\ncd token-importance-scoring\n\npython -m venv .venv && source .venv/bin/activate\n\npip install -e .\n\n# Download checkpoint\n\nhf download oldman-dev/tis-stage3-ert --local-dir checkpoints/stage3_ert\n\n# Run 4-pipeline LITM evaluation\n\npython scripts/run_litm_with_baselines.py \\\n\n```\n--checkpoint checkpoints/stage3_ert \\\\\n```\n\n--n-examples 60 \\\n\n```\n--seed 42\nExpected: baseline LITM gap 0.050 → TIS gap 0.000, baseline EM 16.7% → TIS EM 21.7%\n\n**## Open Questions for the Community**\n\n1. ****Does this transfer to other domains?**** We validated on MS-MARCO. Does passage reordering work on e.g., NaturalQuestions, HotpotQA, or multi-document summarization?\n\n2. ****Beginning-position ceiling****: All learned scorers (including ours) achieve 0% on beginning-position tokens due to causal masking. Our oracle `key_match` scorer (text parsing) reaches 67%. Can we close this gap with a non-causal architecture?\n\n3. ****Cross-model transfer****: Can a TIS head trained on Mistral-7B transfer to Llama-3-8B or Qwen-72B without retraining?\n\n4. ****Inference cost****: TIS adds ~276ms per query (score computation). Is this acceptable for real-time RAG, or should we optimize further?\n\n**## Links**\n\n- ****GitHub****: [GitHub - nitroxido/token-importance-scoring: Public repository for TIS system · GitHub](https://github.com/nitroxido/token-importance-scoring)\n\n- ****HuggingFace****: [oldman-dev (Old Man Developer)](https://huggingface.co/oldman-dev)\n\n- ****Reproducibility guide****: [REPRODUCIBILITY-GUIDE.md]([token-importance-scoring/REPRODUCIBILITY-GUIDE.md at main · nitroxido/token-importance-scoring · GitHub](https://github.com/nitroxido/token-importance-scoring/blob/main/REPRODUCIBILITY-GUIDE.md))\n\n- ****Architecture & specs****: [ARCHITECTURE-TECHNICAL-SPECS.md]([token-importance-scoring/ARCHITECTURE-TECHNICAL-SPECS.md at main · nitroxido/token-importance-scoring · GitHub](https://github.com/nitroxido/token-importance-scoring/blob/main/ARCHITECTURE-TECHNICAL-SPECS.md))\n\n**—**\n\n****We’d love to hear from the community:**** Have you tested TIS on your domain? Do the transfer findings hold for other model families? What would make passage reordering more useful for your RAG pipeline?", "url": "https://wpnews.pro/news/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag", "canonical_source": "https://discuss.huggingface.co/t/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag/178296#post_1", "published_at": "2026-07-29 20:33:44+00:00", "updated_at": "2026-07-29 20:37:18.661543+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-research", "ai-tools"], "entities": ["oldman-dev", "HuggingFace", "GitHub", "MS-MARCO", "LLaMA-3.1-8B", "LLaMA-3.2-1B", "tis-stage3-ert", "tis-passage-reranker"], "alternates": {"html": "https://wpnews.pro/news/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag", "markdown": "https://wpnews.pro/news/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag.md", "text": "https://wpnews.pro/news/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag.txt", "jsonld": "https://wpnews.pro/news/tis-2-0-token-importance-scoring-now-eliminates-position-bias-in-rag.jsonld"}}