{"slug": "analyze-trends-measure-sentiment-predict-what-comes-next", "title": "Analyze trends, Measure sentiment, Predict what comes next", "summary": "HeronLoom, an open-source pipeline from developer korries, combines iterative clustering, LLM-based analysis, graph construction, and analytical search to turn social-media posts or documents into a structured, explorable 3D graph with citations and confidence scores. The tool is built for corpora of up to 50,000 posts or documents and was demonstrated on 43,976 Reddit WallStreetBets posts from September 29, 2020 to April 1, 2021 covering the January 2021 GameStop short squeeze, with no predefined categories, keyword seeds, or manual annotations. HeronLoom requires Python 3.11+, Ollama for embedding and cluster-naming models, and an OpenAI- or Anthropic-compatible API endpoint for its heavy model, with an NVIDIA GPU available for acceleration.", "body_md": "Turn social-media posts or documents into a structured, explorable 3D graph —\nand ask analytical questions in natural language, with citations and a\nconfidence score. **No predefined categories, no manual annotation.**\n\n[Install](#-install) ·\n[Quick start](#-quick-start) ·\n[How it works](#-how-it-works) ·\n[Documentation](#-documentation)\n\n**HeronLoom** combines iterative clustering, LLM-based analysis, graph\nconstruction, and analytical search in a single pipeline. It tracks how topics\nand narratives evolve over time, identifies the signals driving those changes,\nand makes every conclusion traceable to the underlying corpus.\n\nBuilt for corpora of up to 50,000 posts or documents. The pipeline favours clustering quality and analytical depth over raw throughput — larger corpora may work but have not been systematically validated.\n\n- **Reconstructs how ideas change** — tracks topics, arguments, turning\npoints, and changing interpretations across the corpus, instead of just\nproducing a static list of topics.\n- **Explains change over time** — goes beyond what people discussed to show\nwhat changed, when it changed, and what happened afterward.\n- **Preserves conflicting interpretations** — keeps competing explanations\nvisible instead of reducing them to a single narrative.\n- **Separates claims from verified facts, and signals from outcomes** —\nrepeated claims, theories, and interpretations aren't treated as evidence\njust because they are frequent or highly engaged with. When a signal is used\nto anticipate an outcome, the prediction is kept separate from the evidence\nof what actually happened.\n- **Evidence-based confidence** — every conclusion shows what supports it,\nwhat remains uncertain, and where the evidence comes from, down to the\noriginal posts, timestamps, and quotes.\n- **Built for noisy short-form text** — fragmentation, repetition, slang,\nsarcasm, mixed sentiment, and rapidly changing context make short-form\ncontent difficult to analyze reliably with text similarity or aggregate\nscores alone.\n\nA complete run on [Reddit WallStreetBets Posts](https://www.kaggle.com/dsv/2530155) by Gabriel Preda, using raw posts from September 29, 2020 to April 1, 2021. After date filtering, the pipeline processes 43,976 posts covering the January 2021 GameStop short squeeze. No categories, keyword\nseeds, or manual annotations are provided to the system. Every cluster, name,\nnarrative chain, and pool shown below is produced by the pipeline.\n\n**Two real questions asked to the system using this corpus — full answers with confidence scores below, no editing:**\n\n- **Analyze trends + measure sentiment:**\n [What were the main topics and sentiment in the 48h after Robinhood's trading restrictions?](https://github.com/korries/HeronLoom/blob/main/examples/robinhood-restrictions-sentiment.md)\n- **Predict what comes next:**\n [What do you think will happen next, after March 2021, to GameStop's stock, to Robinhood and Citadel, and to the hedge funds that bet against GameStop?](https://github.com/korries/HeronLoom/blob/main/examples/after-March-2021.md)\n\n*Note: Because the corpus (r/wallstreetbets) may contain offensive language,\nsource quotes have been removed from these two examples. The quotes are\nretrieved from the corpus and are not part of the LLM's response.*\n\n**Requirements**\n\n- Python 3.11+\n- [Ollama](https://ollama.com/) , for the embedding and cluster-naming models.\n- An OpenAI- or Anthropic-compatible API endpoint for the `heavy` model,\nunless a local endpoint is configured.**For large corpora, a model with\na 1M-token context window is recommended.**\n- An NVIDIA GPU can accelerate computation:\n\n```\ngit clone https://github.com/korries/HeronLoom.git\ncd HeronLoom\npython -m venv .venv\nsource .venv/bin/activate\n```\n\nOn Windows:\n\n```\n.venv\\Scripts\\activate\npython -m pip install -U pip\npython -m pip install -r requirements.txt\n```\n\nOptional, for GPU acceleration — installing this later, in a new terminal?\nActivate the virtual environment again first (step 2): you should see `(.venv)`\nat the start of your prompt.\nExample for CUDA 13.2:\n\n```\npython -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132\npython -m pip install cupy-cuda13x\ncp .env.example .env\nollama pull qwen3-embedding:8b\nollama pull qwen3.5:9b\n```\n\nConfigure model slots in `config/config.yaml` before the first run. API keys go\nin `.env`, never in configuration files. Provider setup, model selection, and\nparallelism: [Configuration](https://github.com/korries/HeronLoom/blob/main/docs/CONFIGURATION.md).\n\nPlace your files in `data/raw/` — see [Input format](#input-format) below.\n\nThen launch the dashboard:\n\n```\npython dashboard.py\n```\n\nOpen `http://127.0.0.1:8000` and select **New run**. From there: launch the run,\nask questions, and open the 3D graph, all on the same page. See\n[Dashboard](https://github.com/korries/HeronLoom/blob/main/docs/DASHBOARD.md).\n\nFor scripting or automation, use the CLI instead — same `data/raw/` folder, no\ndashboard needed:\n\n```\npython pipeline.py --input data/raw/\npython reload.py search <run_id> \"What were the main topics of discussion and the overall sentiment in the 48 hours following Robinhood's trading restrictions in late January 2021?\"\n```\n\nA `clustering_only` mode is also available, for clustering purposes exclusively\n— it skips the LLM analysis stages (Nova & ADEPT).\n\nFor the full flag reference, run `python pipeline.py --help` for the pipeline\nor `python reload.py --help` for resume, restart, relabel, and search. See\n[CLI reference](https://github.com/korries/HeronLoom/blob/main/docs/CLI.md) for details.\n\n1. **Ingestion & embeddings** — Input files are normalized into a common\ncorpus format with automatic field detection and embedded with**Qwen3-Embedding** .\n2. **Cluster-count estimation** — The number of clusters is determined\nautomatically with GMM-BIC or set manually.\n3. **Iterative refinement (ADR)** — The initial clustering is refined over\nseveral passes using discriminant projection and GMM re-clustering\n([details](#clustering) ).\n4. **Cluster naming** — Labels are generated by the configured naming\nmodel, with a c-TF-IDF fallback.\n5. **Nova & ADEPT***(full mode only)* — Nova organizes each cluster\ninto subtopics and narrative structure. ADEPT groups unassigned\ncontent into pools using Density Peak Clustering. See[Nova & ADEPT](https://github.com/korries/HeronLoom/blob/main/docs/NOVA_ADEPT.md) .\n6. **Graph construction** — Edges capture temporal, semantic, and\nanalysis-derived relationships within and between clusters\n([edge definitions](https://github.com/korries/HeronLoom/blob/main/docs/ALGORITHMS.md#edges) ).\n7. **3D layout** — ForceAtlas2 renders the graph as an interactive\nThree.js scene.\n8. **Ask anything** — Once a run is finished, ask a natural-language\nquestion in any language and get answers with relevant corpus evidence,\ncitations, and a confidence score. See[Search](https://github.com/korries/HeronLoom/blob/main/docs/SEARCH.md) .\n\nA stage-flow diagram is in\n[Architecture](https://github.com/korries/HeronLoom/blob/main/docs/ARCHITECTURE.md#pipeline-stages).\n\nAccepted: `.json`, `.csv`, `.tsv`, `.txt`, `.md`, `.pdf`. Only text content is\nrequired.\n\n```\n{\n  \"id\": \"post_001\",\n  \"content\": \"Text of the post.\",\n  \"timestamp\": \"2026-06-01T14:30:00Z\",\n  \"engagement\": 142\n}\n```\n\n`id`, `timestamp`, and `engagement` are optional; timestamps and engagement\nenrich the graph when present. Non-standard field names can be mapped via\n`field_mapping` in `config.yaml`.\n\nHeronLoom's refinement loop is adapted from **TopiCLEAR**'s ADR loop (Fujita\net al., 2026): clusters are repeatedly refined instead of being fixed after a\nsingle pass. TopiCLEAR itself builds on the **Adaptive Dimension Reduction\n(ADR)** framework introduced by Ding & Li (2007), which alternates between an\nLDA-based discriminant projection and k-means/GMM clustering within that\nsubspace.\n\nHeronLoom replaces the original closed-form LDA step with **LDA-GO** (Shen &\nDong, 2025), which learns the discriminant subspace through gradient-based\noptimization instead. This matters most on high-dimensional text embeddings —\nand especially when the number of clusters is large relative to the sample\nsize — where the closed-form LDA solution routinely hits an invertibility\nfailure and becomes unreliable. LDA-GO avoids that failure mode and makes the\nloop converge reliably in production.\n\nThe integration of LDA-GO into the TopiCLEAR-style refinement loop, along with the surrounding implementation and stability engineering, represents HeronLoom's own contribution.\n\nFull mathematical treatment: [Algorithms](https://github.com/korries/HeronLoom/blob/main/docs/ALGORITHMS.md).\n\nThe dashboard listens on `127.0.0.1` by default. Binding it to `0.0.0.0` to\nexpose it on a network does not add authentication on its own — read\n[Security](https://github.com/korries/HeronLoom/blob/main/SECURITY.md) first.\n\nThanks to the authors of TopiCLEAR and LDA-GO for their open-source contributions.\n\nHeronLoom is licensed under the GNU Affero General Public License v3.0\n(AGPL-3.0). See the [LICENSE](https://github.com/korries/HeronLoom/blob/main/LICENSE) file.\n\nThird-party code and dependencies keep their own licence terms. See\n[Third-party notices](https://github.com/korries/HeronLoom/blob/main/THIRD_PARTY_NOTICES.md).\n\nFor questions, suggestions, or collaboration, please feel free to reach out:\n[korprotech@gmail.com](mailto:korprotech@gmail.com)", "url": "https://wpnews.pro/news/analyze-trends-measure-sentiment-predict-what-comes-next", "canonical_source": "https://github.com/korries/HeronLoom", "published_at": "2026-09-19 20:24:29+00:00", "updated_at": "2026-09-19 20:54:21.272324+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "natural-language-processing", "ai-agents"], "entities": ["HeronLoom", "korries", "Ollama", "OpenAI", "Anthropic", "Reddit", "WallStreetBets", "GameStop"], "alternates": {"html": "https://wpnews.pro/news/analyze-trends-measure-sentiment-predict-what-comes-next", "markdown": "https://wpnews.pro/news/analyze-trends-measure-sentiment-predict-what-comes-next.md", "text": "https://wpnews.pro/news/analyze-trends-measure-sentiment-predict-what-comes-next.txt", "jsonld": "https://wpnews.pro/news/analyze-trends-measure-sentiment-predict-what-comes-next.jsonld"}}