LLM-Based Hierarchical Topic Modeling Tool

A new LLM-based hierarchical topic modeling tool normalizes inconsistent raw strings into a reusable taxonomy for JSONL data, steering with plain-language instructions. The tool maps values like 'bad', 'neg', and 'negative' to 'negative' or 'search' to 'Features|Search', and works with any JSONL file containing strings, nested objects, and arrays of objects. It requires Python 3.9, an API key for a LiteLLM-supported model provider, and internet access while the LLM builds or extends the taxonomy.

Hierarchical topic modeling groups related labels into a taxonomy: broad topics at the top and increasingly specific subtopics below them. This tool is a practical, LLM-assisted version for JSONL data: it turns inconsistent raw strings into a reusable hierarchy that you steer with plain-language instructions. For example, it can map bad , neg , and negative to negative , or map search to Features|Search . It is designed for extracted Reddit-study data, but works with any JSONL file containing strings, nested objects, and arrays of objects. The input is a JSONL file with raw string values. The output is another JSONL file with the same records and structure , except that mapped values have been replaced by their normalized category paths. run normalization.py — the interactive normalizer. normalization.py — field discovery, hierarchical mappings, config merging, and JSONL transformation. ai helper.py — Helper for interacting with LiteLLM config.py — model IDs, API keys, and rate limits. examples\reddit extraction.jsonl — a small Reddit-shaped input example. - Windows, macOS, or Linux. - Python 3.9 probably works with other versions . - An API key for a LiteLLM-supported model provider pretty much all of them . - Internet access while the LLM is building or extending the taxonomy. Most installations use prebuilt Python packages and do not compile anything. If pip reports that Rust , rustc , or cargo is required, install Rust first rather than trying to bypass the error. On Windows, download and run the Rust installer from the official Rust installation page https://rust-lang.org/install.html . Accept the default stable toolchain. If it offers to install the Visual Studio C++ Build Tools, allow it: Rust’s Windows MSVC toolchain may need them. Close and reopen PowerShell, then verify: rustc --version cargo --version On macOS or Linux, install Rust with Rustup: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Choose the default installation, then open a new terminal or run source "$HOME/.cargo/env" and verify: rustc --version cargo --version On Linux, a native compiler/linker may also be required when building packages from source. Install your distribution's development tools first if the Rust installation or pip build reports a linker error—for example, build-essential on Debian/Ubuntu or base-devel on Arch Linux. Rustup is the official recommended installer and installs both rustc and cargo . Rust installation documentation https://rust-lang.org/install.html Open PowerShell in this folder: cd "C:\reddit download script\normalization tool" Use your installed Python directly: python -m pip install --upgrade pip python -m pip install -r requirements.txt An isolated environment is recommended if you work on several Python projects, but it is not required. With Miniconda or Anaconda installed: conda create -n topic-normalizer python=3.11 pip -y conda activate topic-normalizer cd "C:\reddit download script\normalization tool" python -m pip install --upgrade pip python -m pip install -r requirements.txt Activate the same environment whenever you use the tool: conda activate topic-normalizer Open config.py . Its three important settings are: AI MODEL KEYS — model ID → one or more API keys. MODELS — the priority order of models to try. AI MODEL LIMITS — rate limits for each model/key. The model ID must be a valid LiteLLM model identifier for your provider. The included example uses OpenRouter. If you downloaded this tool from Git, create your private configuration first: Copy-Item config.example.py config.py This is the quickest setup. Replace the example list with your actual key: AI MODEL KEYS = { "openrouter/google/gemini-2.5-flash-lite": "sk-or-v1-your-real-key-goes-here", , } MODELS = "openrouter/google/gemini-2.5-flash-lite" Do not commit or share this edited file. Treat it like a password. This preserves the same os.getenv ... logic in config.py , but supplies those variables automatically whenever the tool starts. - .env.example in this folder and rename the copy to exactly .env . - Open .env in a text editor and fill in the key for the provider you use: OPENROUTER API KEY=sk-or-v1-your-real-key-goes-here - Ensure config.py reads the same variable name. The existing OpenRouter pattern already does this: AI MODEL KEYS = { "openrouter/google/gemini-2.5-flash-lite": key for key in os.getenv "OPENROUTER API KEY" if key , } The tool loads .env before it evaluates AI MODEL KEYS . Values already set in PowerShell, Conda, or the operating system take priority over .env . The .env file is listed in .gitignore , but it is still a plain-text secret file: do not share it, upload it, or commit it manually. The direct-paste option remains available. If you paste a key directly into AI MODEL KEYS , that direct value is used; .env is simply ignored for that entry. To use a different provider, change the model ID, the environment-variable name, and MODELS together. If you have several keys for the same model, list them all under that model. The helper rotates between them and tracks the limits configured in AI MODEL LIMITS . The input must be valid JSON Lines .jsonl : one complete JSON object per non-empty line. Values that you want to normalize should still be raw strings. {"id":"kdnmrkk","overall platform sentiment":"bad","categories discussed": {"target feature":"search","sentiment":"neg"} } {"id":"abc123","overall platform sentiment":"negative","categories discussed": {"target feature":"Search","sentiment":"bad"} } Nested arrays work automatically. In this example, categories discussed.sentiment refers to the sentiment value of every object in the categories discussed array. python -m run normalization The script asks for: Input JSONL path — for example examples\reddit extraction.jsonl . Output JSONL path — press Enter to save beside the input as <input-name normalized.jsonl . Normalization config path — press Enter for normalization config.json in this folder. Reuse the same file for future datasets to retain the taxonomy. The script then discovers every string path in the JSONL and starts the interactive normalization loop. Each round shows: - the number of configured fields; - the current ignored fields list; - every field with unmapped values; - the total number of unmapped values for each field and a preview ending in ... . Give the LLM one instruction for the current round. Be explicit about both normalization and exclusions. For example: Normalize overall platform sentiment and categories discussed.sentiment to positive, negative, or neutral. Ignore id, url, created at, opinion summary, and user niche. The model returns a complete configuration. The tool merges it into the saved configuration without discarding existing mappings. It then checks the JSONL again and repeats until every discovered string value is either mapped or belongs to an ignored field. Tell the LLM to ignore fields that are identifiers, URLs, dates, long free-text summaries, or any values you do not want changed. The model adds their dot paths to ignored fields in normalization config.json . Example: Ignore id, url, created at, and opinion summary. Normalize user persona and all sentiment fields. Ignored fields are not checked for missing mappings and are preserved unchanged in the output. The list is shown at the start of every round so you can see exactly what the script will skip. Choose how many unmapped values per field are sent to the LLM in a round. Press Enter for 50. A smaller number reduces prompt size and cost; a larger number gives the model more examples. Answer y to Autocomplete until all values are mapped or ignored? to reuse the same instruction automatically on subsequent rounds. This is useful after giving a broad instruction such as “normalize all values into a concise hierarchy and ignore technical fields.” Autocomplete stops when all values are mapped or ignored. If an LLM call fails, it retries the loop; stop the program manually if a provider outage persists. Answer n when you want to inspect and guide each round yourself. Type exit at the instruction prompt to save the current configuration and stop before creating the normalized JSONL output. The configuration supports any depth of hierarchy. A mapping at a nested level is written to the output as a pipe-delimited path. { "fields": { "field path": "categories discussed.target feature", "categories": { "normalized value": "Features", "raw variations": , "subcategories": { "normalized value": "Search", "raw variations": "search", "Search function" , "subcategories": } } } , "ignored fields": "id", "url" } Here, search becomes Features|Search in the output JSONL. For complex taxonomies, you can prepare the configuration outside the script before continuing: - Stop the script or let it save normalization config.json . - Copy the entire JSON configuration. - Paste it into a chat UI that can handle large pasted documents, such as DeepSeek. - Ask it to review the taxonomy, merge duplicates, improve category names, add hierarchy, or make the exact changes you want. - Copy the full corrected JSON object back into normalization config.json . - Restart python -m run normalization so the tool reloads the edited config. This gives the normalizer a stronger starting taxonomy for its next interactive round. A future version is intended to make this external review step unnecessary; for now, it is a useful way to fine-tune a large configuration. - This is instruction-driven taxonomy construction, not a high-coverage or fully automated research pipeline. It only sees the raw string values and samples you provide; it does not have broad context about the original posts unless that context is itself present in the field values. - LLMs can hallucinate, misclassify values, choose an overly broad category, or incorrectly ignore a field. Review the configuration and normalized output, especially before analysis or publication. - A single raw value can currently map to only one normalized path. Values that genuinely belong to two separate categories are not represented well by the current configuration format. - Long, highly unique free text is usually a poor fit for this tool. Ignore it or preprocess it into labels first. - API cost, model quality, context size, and rate limits affect speed and consistency. The tests use mock JSONL records, mocked interactive input, and mocked LLM responses—no API calls are made. python -m unittest discover -s tests -v | Problem | What to do | |---|---| No configured API keys | Set the environment variable used in config.py , or paste a key into AI MODEL KEYS . Check that the model also appears in MODELS . | rustc or cargo is required | Install Rust with Rustup, reopen PowerShell, verify rustc --version , then rerun the pip install. | | Provider/model error | Check the LiteLLM model identifier, your provider key, and the selected model’s availability. | | Invalid JSON line | The error includes a line number. JSONL must have one complete JSON object per non-empty line. | | Values remain unmapped | Give a more specific instruction, increase sample size, or tell the LLM to add the field to ignored fields . |