LLM-Based Hierarchical Topic Modeling Tool

Hierarchical topic modeling groups related labels into a taxonomy: broad topics at the top and increasingly specific subtopics below them. This tool is a practical, LLM-assisted version for JSONL data: it turns inconsistent raw strings into a reusable hierarchy that you steer with plain-language instructions.

For example, it can map bad

, neg

, and negative

to negative

, or map search

to Features|Search

. It is designed for extracted Reddit-study data, but works with any JSONL file containing strings, nested objects, and arrays of objects.

The input is a JSONL file with raw string values. The output is another JSONL file with the same records and structure, except that mapped values have been replaced by their normalized category paths.

run_normalization.py

— the interactive normalizer.normalization.py

— field discovery, hierarchical mappings, config merging, and JSONL transformation.ai_helper.py

— Helper for interacting with LiteLLMconfig.py

— model IDs, API keys, and rate limits.examples\reddit_extraction.jsonl

— a small Reddit-shaped input example.

• Windows, macOS, or Linux.
• Python 3.9 (probably works with other versions).
• An API key for a LiteLLM-supported model provider (pretty much all of them).
• Internet access while the LLM is building or extending the taxonomy.

Most installations use prebuilt Python packages and do not compile anything. If pip reports that Rust

, rustc

, or cargo

is required, install Rust first rather than trying to bypass the error.

On Windows, download and run the Rust installer from the official Rust installation page. Accept the default stable toolchain. If it offers to install the Visual Studio C++ Build Tools, allow it: Rust’s Windows MSVC toolchain may need them. Close and reopen PowerShell, then verify:

rustc --version
cargo --version

On macOS or Linux, install Rust with Rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Choose the default installation, then open a new terminal (or run source "$HOME/.cargo/env"

) and verify:

rustc --version
cargo --version

On Linux, a native compiler/linker may also be required when building packages from source. Install your distribution's development tools first if the Rust installation or pip build reports a linker error—for example, build-essential

on Debian/Ubuntu or base-devel

on Arch Linux.

Rustup is the official recommended installer and installs both rustc

and cargo

. Rust installation documentation

Open PowerShell in this folder:

cd "C:\reddit download script\normalization_tool"

Use your installed Python directly:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

An isolated environment is recommended if you work on several Python projects, but it is not required. With Miniconda or Anaconda installed:

conda create -n topic-normalizer python=3.11 pip -y
conda activate topic-normalizer
cd "C:\reddit download script\normalization_tool"
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Activate the same environment whenever you use the tool:

conda activate topic-normalizer

Open config.py

. Its three important settings are:

AI_MODEL_KEYS

— model ID → one or more API keys.MODELS

— the priority order of models to try.AI_MODEL_LIMITS

— rate limits for each model/key.

The model ID must be a valid LiteLLM model identifier for your provider. The included example uses OpenRouter.

If you downloaded this tool from Git, create your private configuration first:

Copy-Item config.example.py config.py

This is the quickest setup. Replace the example list with your actual key:

AI_MODEL_KEYS = {
    "openrouter/google/gemini-2.5-flash-lite": [
        "sk-or-v1-your-real-key-goes-here",
    ],
}

MODELS = ["openrouter/google/gemini-2.5-flash-lite"]

Do not commit or share this edited file. Treat it like a password.

This preserves the same os.getenv(...)

logic in config.py

, but supplies those variables automatically whenever the tool starts.

.env.example

in this folder and rename the copy to exactly.env

. - Open

.env

in a text editor and fill in the key for the provider you use:

OPENROUTER_API_KEY=sk-or-v1-your-real-key-goes-here

Ensure

config.py

reads the same variable name. The existing OpenRouter pattern already does this:

AI_MODEL_KEYS = {
    "openrouter/google/gemini-2.5-flash-lite": [
        key for key in [os.getenv("OPENROUTER_API_KEY")] if key
    ],
}

The tool loads .env

before it evaluates AI_MODEL_KEYS

. Values already set in PowerShell, Conda, or the operating system take priority over .env

. The .env

file is listed in .gitignore

, but it is still a plain-text secret file: do not share it, upload it, or commit it manually.

The direct-paste option remains available. If you paste a key directly into AI_MODEL_KEYS

, that direct value is used; .env

is simply ignored for that entry.

To use a different provider, change the model ID, the environment-variable name, and MODELS

together.

If you have several keys for the same model, list them all under that model. The helper rotates between them and tracks the limits configured in AI_MODEL_LIMITS

.

The input must be valid JSON Lines (.jsonl

): one complete JSON object per non-empty line. Values that you want to normalize should still be raw strings.

{"id":"kdnmrkk","overall_platform_sentiment":"bad","categories_discussed":[{"target_feature":"search","sentiment":"neg"}]}
{"id":"abc123","overall_platform_sentiment":"negative","categories_discussed":[{"target_feature":"Search","sentiment":"bad"}]}

Nested arrays work automatically. In this example, categories_discussed.sentiment

refers to the sentiment

value of every object in the categories_discussed

array.

python -m run_normalization

The script asks for:

Input JSONL path— for exampleexamples\reddit_extraction.jsonl

.Output JSONL path— press Enter to save beside the input as<input-name>_normalized.jsonl

.Normalization config path— press Enter fornormalization_config.json

in this folder. Reuse the same file for future datasets to retain the taxonomy.

The script then discovers every string path in the JSONL and starts the interactive normalization loop.

Each round shows:

• the number of configured fields;
• the current ignored_fields

list; - every field with unmapped values;

• the total number of unmapped values for each field and a preview ending in ...

.

Give the LLM one instruction for the current round. Be explicit about both normalization and exclusions. For example:

Normalize overall_platform_sentiment and categories_discussed.sentiment to positive, negative, or neutral. Ignore id, url, created_at, opinion_summary, and user_niche.

The model returns a complete configuration. The tool merges it into the saved configuration without discarding existing mappings. It then checks the JSONL again and repeats until every discovered string value is either mapped or belongs to an ignored field.

Tell the LLM to ignore fields that are identifiers, URLs, dates, long free-text summaries, or any values you do not want changed. The model adds their dot paths to ignored_fields

in normalization_config.json

.

Example:

Ignore id, url, created_at, and opinion_summary. Normalize user_persona and all sentiment fields.

Ignored fields are not checked for missing mappings and are preserved unchanged in the output. The list is shown at the start of every round so you can see exactly what the script will skip.

Choose how many unmapped values per field are sent to the LLM in a round. Press Enter for 50. A smaller number reduces prompt size and cost; a larger number gives the model more examples.

Answer y

to Autocomplete until all values are mapped or ignored? to reuse the same instruction automatically on subsequent rounds. This is useful after giving a broad instruction such as “normalize all values into a concise hierarchy and ignore technical fields.”

Autocomplete stops when all values are mapped or ignored. If an LLM call fails, it retries the loop; stop the program manually if a provider outage persists. Answer n

when you want to inspect and guide each round yourself.

Type exit

at the instruction prompt to save the current configuration and stop before creating the normalized JSONL output.

The configuration supports any depth of hierarchy. A mapping at a nested level is written to the output as a pipe-delimited path.

{
  "fields": [
    {
      "field_path": "categories_discussed.target_feature",
      "categories": [
        {
          "normalized_value": "Features",
          "raw_variations": [],
          "subcategories": [
            {
              "normalized_value": "Search",
              "raw_variations": ["search", "Search function"],
              "subcategories": []
            }
          ]
        }
      ]
    }
  ],
  "ignored_fields": ["id", "url"]
}

Here, search

becomes Features|Search

in the output JSONL.

For complex taxonomies, you can prepare the configuration outside the script before continuing:

• Stop the script or let it save normalization_config.json

. - Copy the entire JSON configuration.

• Paste it into a chat UI that can handle large pasted documents, such as DeepSeek.
• Ask it to review the taxonomy, merge duplicates, improve category names, add hierarchy, or make the exact changes you want.
• Copy the full corrected JSON object back into normalization_config.json

. - Restart python -m run_normalization

so the tool reloads the edited config.

This gives the normalizer a stronger starting taxonomy for its next interactive round. A future version is intended to make this external review step unnecessary; for now, it is a useful way to fine-tune a large configuration.

• This is instruction-driven taxonomy construction, not a high-coverage or fully automated research pipeline. It only sees the raw string values and samples you provide; it does not have broad context about the original posts unless that context is itself present in the field values.
• LLMs can hallucinate, misclassify values, choose an overly broad category, or incorrectly ignore a field. Review the configuration and normalized output, especially before analysis or publication.
• A single raw value can currently map to only one normalized path. Values that genuinely belong to two separate categories are not represented well by the current configuration format. - Long, highly unique free text is usually a poor fit for this tool. Ignore it or preprocess it into labels first.
• API cost, model quality, context size, and rate limits affect speed and consistency.

The tests use mock JSONL records, mocked interactive input, and mocked LLM responses—no API calls are made.

python -m unittest discover -s tests -v

Problem	What to do
No configured API keys
Set the environment variable used in config.py , or paste a key into AI_MODEL_KEYS . Check that the model also appears in MODELS .
rustc or cargo is required
Install Rust with Rustup, reopen PowerShell, verify rustc --version , then rerun the pip install.
Provider/model error	Check the LiteLLM model identifier, your provider key, and the selected model’s availability.
Invalid JSON line	The error includes a line number. JSONL must have one complete JSON object per non-empty line.
Values remain unmapped	Give a more specific instruction, increase sample size, or tell the LLM to add the field to ignored_fields .

source & further reading

github.com — original article