{"slug": "looking-for-contributors-trie-based-memory-efficient-llm-runner", "title": "Looking for contributors – trie based memory efficient LLM runner", "summary": "SALT, a trie-based memory-efficient LLM runner, solves the 'theme collapse' problem in long-document compression by mapping recurring themes into a keyword trie and spreading the token budget across theme branches. The system, now defaulting to a coverage/CELF selector, indexes a document once and reuses the trie across conversation turns, cutting compute and memory costs while preserving minor themes.", "body_md": "Note\n\n**What is next**\n\n**MCP server**- a`salt-mcp`\n\nentry point so AI clients can use SALT as their conversation memory without the REPL.**Tail-aware memory selection**- stop spending the memory budget on text the model is already reading in the recent messages.** Incremental compression**- reuse the previous turn's work instead of rebuilding the whole selection every turn.** Graduating the memory switches**- decide which off-by-default memory behaviors become defaults, using numbers from real sessions.** Scripted conversation runs**- richer tooling around`--turns`\n\nfor driving and scoring long canned conversations.\n\nSALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost.\n\n**The problem.** When a prompt is too long, existing compressors give each\nsentence a single relevance score and keep the top-scoring ones until the budget\nruns out. Under a tight budget this lets the document's main topic swallow the\nwhole budget, so smaller but still important points get dropped - a failure called\n*theme collapse* (in multi-hop questions, for example, it can keep\npassages about the main entity yet lose the one sentence that links it to a\nsecond).\n\n**The solution.** SALT first maps the document's recurring themes by organizing\neach sentence's keywords into a trie, a small keyword tree ordered by how often\nthose keywords recur, then spreads the budget across those theme branches\nbefore choosing sentences, so minor themes keep their share instead of being\ncrowded out. Because the theme map is built once, it can be reused across the\nturns of a conversation without re-reading the document.\n\nThe previous\n\nlegacyselector as described in the paper release is tagged[;]`v1.0.0`\n\n`main`\n\nnow defaults to the coverage/CELF selector described below.\n\nTwo phases. **Indexing** reads the document once and builds a keyword trie - a\nreusable map of its recurring themes. **Selection** then picks a sentence subset\nunder a token budget, returned in original document order, query-biased or not.\n\n```\n INDEXING  ── once per document, reused across turns and budgets\n   document → split + junk filter\n           → per-sentence keywords   (BGE-small [CLS] attention + knee cutoff)\n           → theme salience          (SF = #sentences keeping a word, top quantile)\n           → keyword trie            (each sentence's themes, SF-ordered, form a\n                                      root-to-leaf path, leaves hold sentence ids)\n\n SELECTION ── per budget, with or without a query\n   maximize theme coverage with CELF lazy-greedy: a pick's value shrinks as its\n   theme branches fill, so budget spreads across themes instead of collapsing\n   onto the dominant one. A query re-weights the trie (lexical + BGE-semantic)\n   without rebuilding it. → compressed prompt, original order, ≤ budget\n```\n\nThe whole system as a blueprint - this map is kept current as SALT grows, so it is the fastest way to find where a change belongs:\n\n```\n┌──────────────────────────────────────────────────────────────────────────┐\n│                                   SALT                                   │\n│                                                                          │\n│ ┌────────────────────┐  ┌────────────────────┐  ┌────────────────────┐   │\n│ │      Indexing      │  │    Keyword Trie    │  │     Selection      │   │\n│ │                    │  │                    │  │                    │   │\n│ │ BGE-small encoder  │  │ SF-ordered paths   │  │ coverage (CELF)    │   │\n│ │ attention keywords │  │ theme branches     │  │ branch discounting │   │\n│ │ knee cutoff        │  │ §file: doc branches│  │ multi-anchor query │   │\n│ │ junk filter        │  │ rebuilt cheaply    │  │ ≤ word budget      │   │\n│ └────────────────────┘  └────────────────────┘  └────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────┐  ┌────────────────────┐  ┌────────────────────┐   │\n│ │    Session Trie    │  │  Prompt Assembly   │  │    Chat Runner     │   │\n│ │                    │  │                    │  │                    │   │\n│ │ per-conversation   │  │ stable prefix first│  │ HF streaming       │   │\n│ │ lives in DRAM      │  │ append-only tail   │  │ vLLM + APC (opt-in)│   │\n│ │ grows every turn   │  │ memory + question  │  │ vllm-serve client  │   │\n│ │ cross-turn coverage│  │ instructions.md    │  │ model registry     │   │\n│ │ + half-life decay  │  │                    │  │ GPU-pinned models  │   │\n│ │ + near-dup gate    │  │                    │  │                    │   │\n│ │ + background ingest│  │                    │  │                    │   │\n│ └────────────────────┘  └────────────────────┘  └────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────────────────────────────────────────────────────┐   │\n│ │              Document ingest (salt@ files, salt --doc)             │   │\n│ │ pypdf extract · furniture scrub · paragraphs rejoined across floats│   │\n│ │ tables + pseudocode grouped under captions · footnotes isolated    │   │\n│ │ headings, panel labels and equations kept · reference list dropped │   │\n│ └────────────────────────────────────────────────────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────────────────────────────────────────────────────┐   │\n│ │            Trie shape - the root binds the conversation            │   │\n│ │                               ● root - the conversation bind       │   │\n│ │         ┌─────────────────────┼─────────────────────┐              │   │\n│ │  §file:paper.pdf       §file:notes.txt        conversation         │   │\n│ │         │                     │              ┌──────┴──────┐       │   │\n│ │   keyword paths         keyword paths     theme A       theme B    │   │\n│ │         │                     │              │             │       │   │\n│ │     sentences             sentences      sentences     sentences   │   │\n│ │                                                                    │   │\n│ │ each turn: ≤ budget spread across branches (CELF discounting)      │   │\n│ │ the untrie - the verbatim tail - sits OUTSIDE the trie, as the     │   │\n│ │ prompt's stable recent-history window                              │   │\n│ └────────────────────────────────────────────────────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────────────────────────────────────────────────────┐   │\n│ │                  Prompt layout (KV-cache shaped)                   │   │\n│ │ [system: instructions · file inventory · attach@ full documents]   │   │\n│ │ → [tail: recent exchanges - append-only, block-wise compaction]    │   │\n│ │ → [newest user message: SALT memory (≈20% selection) + question]   │   │\n│ │ stable prefix = reusable KV ──── fresh suffix = per-turn prefill   │   │\n│ └────────────────────────────────────────────────────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────────────────────────────────────────────────────┐   │\n│ │                    kvtrace - per-turn KV ledger                    │   │\n│ │ read (reused) / write (fresh) / output · events.jsonl + tokens.npy │   │\n│ │ usage keys: input (write) · input_cached_tokens (read) · output    │   │\n│ │ apc fields: engine-measured prefix-cache reuse (vllm + vllm-serve) │   │\n│ └────────────────────────────────────────────────────────────────────┘   │\n│                                                                          │\n│ ┌────────────────────────────────────────────────────────────────────┐   │\n│ │                            Entry points                            │   │\n│ │ salt (one-shot: --data / --doc) · saltChat · saltServe · eval.py   │   │\n│ │ salt@ trie attach · attach@ full text · /doc /model /budget /stats │   │\n│ └────────────────────────────────────────────────────────────────────┘   │\n└──────────────────────────────────────────────────────────────────────────┘\n```\n\nWhere each stage lives:\n\n| Stage | Code |\n|---|---|\n| Split + junk filter | `salt/engine/embedder.py` , `salt/engine/sentence_filter.py` |\n| Keywords, BGE embedding, theme profiling | `salt/engine/trie_core.py` |\n| Coverage selection (default) | `salt/engine/celf.py` |\n| Prose pipeline runner | `salt/engine/compressor.py` |\nFew-shot bypass (`trec` , `triviaqa` , `samsum` ) |\n`salt/engine/fewshot.py` |\nDataset adapters (`--synthetic` , `--code` ) |\n`salt/engine/dataset_modes.py` |\n| Multi-turn session store | `salt/engine/session_trie.py` |\n| Chat text handling (verbatim storage, short turns) | `salt/engine/chat_text.py` , `salt/chat/shortturn.py` |\n| Background ingest worker (chat) | `salt/chat/ingest.py` |\nDocument ingest (PDF/text cleanup, `salt@` , `--doc` ) |\n`salt/chat/pdfio.py` |\n| Chat REPL + model registry | `salt/chat/` , `salt/models/` |\nPersistent serving (`saltServe` , serve client) |\n`salt/chat/serve.py` , `salt/chat/runner_serve.py` |\nMulti-GPU placement (`--gpu` list) |\n`salt/chat/runner.py` , `salt/chat/serve.py` |\n| CLI entry points | `salt` (`salt/compress.py` ), `eval.py` , `saltChat` , `saltServe` |\n\nRequires Python 3.10 and a CUDA GPU (CPU works for compression, just slower).\n\n**1. Clone the repository**\n\n```\ngit clone https://github.com/oteomamo/SALT.git\ncd SALT\n```\n\n**2. Create the environment**\n\nWith conda:\n\n```\nconda env create -f environment.yml\nconda activate salt\n```\n\nOr with venv:\n\n```\npython3.10 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n**3. Install SALT in editable mode**\n\n```\npip install -e .\n```\n\nThis also installs the two console commands: `salt`\n\n(one-shot compression,\nsee [Usage](https://oteomamo.github.io/SALT/usage/)) and `saltChat`\n\n(interactive chat, see\n[Chatbot mode](#-chatbot-mode)).\n\n**4. Authenticate with Hugging Face** - the eval model\n(`meta-llama/Llama-3.1-8B-Instruct`\n\n) is gated:\n\n```\nhf auth login\n```\n\nOr skip the CLI and export the token directly: `export HF_TOKEN=hf_...`\n\n.\n\n**5. (Optional) vLLM backend.** `eval.py`\n\ndefaults to vLLM,\n`saltChat --backend vllm`\n\nuses it for prefix caching, and `saltServe`\n\nlaunches a persistent model server with it. Install it into the `salt`\n\nenv:\n\n```\npip install \"vllm==0.11.0\" \"prometheus-fastapi-instrumentator>=8.0.1\"\n```\n\nThe second pin keeps the server's routes healthy next to newer fastapi\nreleases. Skip this and run `eval.py --backend hf`\n\nfor a portable run\nthat needs no vLLM. `saltChat`\n\nalready defaults to its HF backend.\n`saltServe`\n\ncan also run a vLLM installed in a separate environment\nthrough `--vllm-bin`\n\n.\n\n`bash scripts/setup_env.sh`\n\ndoes steps 2–3 in one shot (add`WITH_VLLM=1`\n\nto include vLLM).\n\nFetch the LongBench data, then compress every task present and evaluate the outputs in one command:\n\n```\npython salt/datasets/download_longbench.py\nbash scripts/run_datasets.sh\n```\n\nSmoke test (5 samples per task, compression only):\n\n```\nMAX_SAMPLES=5 RUN_EVAL=0 bash scripts/run_datasets.sh\n```\n\nThe script routes each dataset to the right mode automatically, then scores\nthe results. Every command, flag, and knob is documented on the\n[Usage](https://oteomamo.github.io/SALT/usage/) and\n[Datasets](https://oteomamo.github.io/SALT/datasets/) pages.\n\n`saltChat`\n\nis an interactive chat REPL where SALT is the conversation memory:\none persistent trie per conversation grows with every exchange (and any\nattached documents), and each turn it compresses the accumulated history into\na query-biased context block under the token budget.\n\n```\nsaltChat --model qwen05 --conversation-id demo1 --doc report.txt\n```\n\nA persistent server started with `saltServe`\n\nkeeps the model loaded and\nits cache warm between chats, so a resumed conversation picks up without\nre-reading its documents. The\n[Chatbot mode guide](https://oteomamo.github.io/SALT/chatbot/) covers\nthe concepts, the [Serving](https://oteomamo.github.io/SALT/serving/)\npage covers the server, and the\n[Options](https://oteomamo.github.io/SALT/options/) page lists every\nflag in one line each, including the off-by-default switches that make\nlong sessions better.\n\nSALT (coverage/CELF selector) reaches an overall **44.60** LongBench average\nwith Llama-3.1-8B-Instruct at a 20% token budget. The full per-dataset table\nis on the [Results](https://oteomamo.github.io/SALT/results/) page.\n\nIn progress:\n\n**MCP server**- a`salt-mcp`\n\nentry point exposing compression and session memory as tools, so AI clients (Claude Code, Claude Desktop, Cursor) can use SALT as their conversation memory without the REPL.**Tail-aware memory selection**- skip sentences the model is already reading verbatim in the recent messages, so the memory budget buys new context instead of repeating what is on screen.**Incremental compression**- carry the previous turn's selection work forward on an append-only conversation, instead of redoing all of it every turn.**Graduating the memory switches**- several memory behaviors ship off by default (see the[Options](https://oteomamo.github.io/SALT/options/)page) while`/stats`\n\nnumbers from real sessions decide which become defaults.**Scripted conversation runs**- richer tooling around`--turns`\n\n, so canned conversations can drive long sessions and be scored afterward.\n\nNext:\n\n**Summarization coverage**- extend the theme-coverage objective to better serve summarization, where recall across many minor themes matters most.\n\nPRs welcome - see [CONTRIBUTING.md](/oteomamo/SALT/blob/main/CONTRIBUTING.md).\n\nIf you find this project useful for your research, please consider citing our paper：\n\n```\n@misc{mamo2026saltsalienceawarelexicaltrie,\n      title={SALT: Salience-Aware Lexical Trie for Long-Context Compression}, \n      author={Oteo Mamo and Hyunjin Yi and Joydhriti Choudhury and Shangqian Gao and Weikuan Yu},\n      year={2026},\n      eprint={arXiv:2607.17486}\n}\n```\n\nSALT is released under the [MIT License](/oteomamo/SALT/blob/main/LICENSE).", "url": "https://wpnews.pro/news/looking-for-contributors-trie-based-memory-efficient-llm-runner", "canonical_source": "https://github.com/oteomamo/SALT", "published_at": "2026-07-21 04:45:07+00:00", "updated_at": "2026-07-21 04:52:44.042207+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "natural-language-processing"], "entities": ["SALT", "BGE-small", "CELF"], "alternates": {"html": "https://wpnews.pro/news/looking-for-contributors-trie-based-memory-efficient-llm-runner", "markdown": "https://wpnews.pro/news/looking-for-contributors-trie-based-memory-efficient-llm-runner.md", "text": "https://wpnews.pro/news/looking-for-contributors-trie-based-memory-efficient-llm-runner.txt", "jsonld": "https://wpnews.pro/news/looking-for-contributors-trie-based-memory-efficient-llm-runner.jsonld"}}