{"slug": "show-hn-data-enrichment-with-ai-for-pandas-dataframe", "title": "Show HN: Data enrichment with AI for Pandas DataFrame", "summary": "A new open-source Python library called enrichment lets users add AI-generated columns to pandas DataFrames with a single function call, using plain-English prompts to classify, extract, translate, or summarize data. The library, showcased on Hacker News, caches duplicate rows to avoid repeated API costs and automatically batches large jobs, with OpenAI batch processing offering a 50% discount. It supports both single-column and multi-column inputs and returns a new DataFrame without modifying the original.", "body_md": "**Add AI columns to your pandas DataFrame.**\n\nYou have a table. You want a new column that only a human could fill in — like \"is this review happy or angry?\" or \"what industry is this company in?\".\n\nWrite what you want in normal English. `enrichment`\n\nasks an AI model for every row and gives you back a new table.\n\n**Before:**\n\n| review |\n|---|\n| I loved the product! |\n| It arrived broken. |\n| It was okay. |\n\n**Your instruction:** *\"Classify sentiment as positive, negative, or neutral\"*\n\n**After:**\n\n| review | sentiment |\n|---|---|\n| I loved the product! | positive |\n| It arrived broken. | negative |\n| It was okay. | neutral |\n\nThat is the whole idea. One function, one sentence of instructions, one new column.\n\n``` python\nimport pandas as pd\nfrom enrichment import enrich\n\ndf = pd.DataFrame(\n    {\n        \"review\": [\n            \"I loved the product!\",\n            \"It arrived broken.\",\n            \"It was okay.\",\n        ]\n    }\n)\n\nresult = enrich(\n    df,\n    input_col=\"review\",       # which column the AI reads\n    output_col=\"sentiment\",   # name of the new column\n    prompt=\"Classify sentiment as positive, negative, or neutral\",\n)\n\nprint(result)\n```\n\nThat's it. Four things to fill in: your table, the column to read, the column to create, and what you want.\n\n**Two things worth knowing:**\n\n- Your original\n`df`\n\nis**not changed**. You always get a new DataFrame back. - If the same text appears twice, it is sent to the AI\n**once**. You don't pay twice for the same row.\n\nThe prompt is just plain English, so you are not limited to sentiment. A few ideas:\n\n```\n# Pull a value out of messy text\nenrich(df, input_col=\"address\", output_col=\"city\",\n       prompt=\"Extract the city name\")\n\n# Sort things into groups\nenrich(df, input_col=\"ticket\", output_col=\"team\",\n       prompt=\"Route to one team: billing, technical, or sales\")\n\n# Yes / no questions\nenrich(df, input_col=\"email\", output_col=\"is_spam\",\n       prompt=\"Answer yes or no: is this email spam?\")\n\n# Translate\nenrich(df, input_col=\"comment\", output_col=\"english\",\n       prompt=\"Translate to English\")\n\n# Clean up untidy data\nenrich(df, input_col=\"job_title\", output_col=\"clean_title\",\n       prompt=\"Rewrite as a standard job title, e.g. 'Software Engineer'\")\n\n# Summarize\nenrich(df, input_col=\"article\", output_col=\"summary\",\n       prompt=\"Summarize in one short sentence\")\n```\n\nSometimes one column is not enough context. Use `input_cols`\n\n(note the **s**) and pass a list:\n\n```\nresult = enrich(\n    df,\n    input_cols=[\"company_name\", \"website\"],\n    output_col=\"industry\",\n    prompt=\"Determine the company's industry\",\n)\n```\n\nThe AI sees both values together, with their column names, so it knows which is which.\n\nUse either `input_col`\n\nor `input_cols`\n\n— not both at the same time.\n\nThe prompt is the most important part. Small changes make a big difference.\n\n| Instead of | Try |\n|---|---|\n| \"sentiment\" | \"Classify sentiment as positive, negative, or neutral\" |\n| \"what is this about\" | \"Return the main topic in 1-3 words\" |\n| \"clean this\" | \"Return only the phone number, digits only\" |\n\nThree simple rules:\n\n**List the allowed answers.**\"positive, negative, or neutral\" gives you a tidy column. \"How does this person feel?\" gives you paragraphs.** Say how long the answer should be.**\"in one word\", \"in one short sentence\".** Test on a few rows first.**Run`df.head(10)`\n\nbefore running 10,000 rows.\n\n```\n# Try it small first\nsample = enrich(df.head(10), input_col=\"review\", output_col=\"sentiment\",\n                prompt=\"Classify sentiment as positive, negative, or neutral\")\nprint(sample)\n```\n\nYou don't have to do anything special for large tables — just call `enrich()`\n\nas usual.\n\n**Small jobs** are sent as several requests at the same time, so they finish faster.**Big jobs**(50 or more unique values) are automatically sent as one batch, if your provider supports it. On OpenAI, batches cost** 50% less**. They can take up to 24 hours, but usually much less.\n\nRows always come back in the original order, even when the provider returns them mixed up.\n\nWant to decide yourself?\n\n```\nresult = enrich(df, ..., use_batch=True)   # always batch\nresult = enrich(df, ..., use_batch=False)  # never batch\nresult = enrich(df, ..., use_batch=None)   # default: decide for me\n```\n\nOpenAI batch limits: up to 50,000 unique rows and 200 MB per batch.\n\nNetwork problems happen. `enrichment`\n\nhandles the common ones for you: rate limits (HTTP 429), timeouts, and temporary server errors are retried automatically with a growing wait time.\n\n**Empty rows** are skipped and get `pd.NA`\n\n— no API call, no cost.\n\n**Failed rows:** by default the whole job stops if a row keeps failing. If you'd rather finish the job and mark the bad rows, use `on_error=\"keep\"`\n\n:\n\n```\nresult = enrich(\n    df,\n    input_col=\"text\",\n    output_col=\"topic\",\n    prompt=\"Return the main topic\",\n    on_error=\"keep\",   # failed rows become pd.NA instead of stopping everything\n)\n```\n\nYou can also slow down or speed up the requests:\n\n```\nresult = enrich(\n    df,\n    input_col=\"text\",\n    output_col=\"topic\",\n    prompt=\"Return the main topic\",\n    max_concurrency=10,   # requests at the same time (default 5)\n    max_retries=3,        # tries per row before giving up\n)\n```\n\nAdd `return_report=True`\n\nto get a small report along with your table. Useful for checking cost and errors.\n\n```\nresult, report = enrich(\n    df,\n    input_col=\"text\",\n    output_col=\"topic\",\n    prompt=\"Return the main topic\",\n    return_report=True,\n)\n\nprint(report.completed)         # how many rows were filled\nprint(report.unique_requests)   # how many calls were actually sent\nprint(report.retries)           # how many retries were needed\nprint(report.input_tokens, report.output_tokens)   # usage, for cost\nprint(report.errors)            # what failed, if anything\n```\n\nNote that `enrich()`\n\nnow returns **two** things, so you need two variables on the left.\n\nThe default OpenAI model is `gpt-5-nano`\n\n. It is fast and cheap, and it's a good fit for sorting and extracting data — which is most of what people use this for.\n\nNeed better quality on a hard task? Pick another model:\n\n```\nresult = enrich(df, ..., model=\"gpt-5.4\")\n```\n\nYou are not locked into OpenAI. Anything that speaks the OpenAI Chat Completions format works — including models running on your own machine.\n\n**A local model (nothing leaves your computer):**\n\n``` python\nfrom enrichment import OpenAICompatibleProvider, enrich\n\nprovider = OpenAICompatibleProvider(\n    base_url=\"http://127.0.0.1:1234/v1\",\n    model=\"local-model\",\n)\n\nresult = enrich(\n    df,\n    input_col=\"review\",\n    output_col=\"sentiment\",\n    prompt=\"Classify sentiment\",\n    provider=provider,\n)\n```\n\n**A hosted provider with a key:**\n\n```\nprovider = OpenAICompatibleProvider(\n    base_url=\"https://provider.example/v1\",\n    api_key=\"your-api-key\",\n    model=\"provider/model-name\",\n    headers={\"X-App\": \"My application\"},\n)\nenrich(\n    df,\n    input_col=None,          # column to read\n    output_col=None,         # new column to create\n    prompt=None,             # what you want, in plain English\n    model=None,              # model name, provider default if empty\n    api_key=None,            # key, if you don't use an env variable\n    show_progress=True,      # show a progress bar\n    input_cols=None,         # several columns to read (instead of input_col)\n    provider=None,           # custom provider object\n    max_concurrency=5,       # parallel requests\n    max_retries=3,           # tries per row\n    retry_base_delay=0.5,    # seconds before the first retry\n    use_batch=None,          # True / False / None (automatic)\n    on_error=\"raise\",        # \"raise\" to stop, \"keep\" to fill pd.NA\n    return_report=False,     # also return an execution report\n)\n```\n\n**Do I need to know anything about AI?**\nNo. If you can write a sentence and use pandas, you're ready.\n\n**Is my data sent to the internet?**\nYes, to whichever provider you choose. If your data cannot leave your machine, run a local model and pass it through `OpenAICompatibleProvider`\n\n.\n\n**How much does it cost?**\nIt depends on your provider and model, not on this package. Two things keep it low: repeated values are only sent once, and big jobs use cheaper batches. Run `return_report=True`\n\nto see your exact token usage.\n\n**Will it change my original DataFrame?**\nNo. You always get a new one back.\n\n**What happens to empty cells?**\nThey are skipped and filled with `pd.NA`\n\n. No API call is made for them.\n\n**Can I get the same answer every time?**\nMostly, but AI models can vary a little. For anything important, check a sample of the output yourself.\n\n```\npip install enrichment\n```\n\nYou also need pandas, which comes along with the install.\n\n`enrichment`\n\ndoes not have its own AI. It talks to an AI provider for you. The easiest one to start with is OpenAI.\n\n- Go to\n[platform.openai.com](https://platform.openai.com/api-keys)and create an account. - Add a payment method (you pay for what you use — usually cents for small tables).\n- Create an API key and copy it. It looks like\n`sk-...`\n\n. - Tell your computer about it:\n\n**Mac / Linux:**\n\n```\nexport OPENAI_API_KEY=\"your-api-key\"\n```\n\n**Windows (PowerShell):**\n\n```\n$env:OPENAI_API_KEY=\"your-api-key\"\n```\n\nOr, if you prefer, pass it straight to the function:\n\n```\nenrich(df, ..., api_key=\"your-api-key\")\n```\n\nUsing MLJAR Studio?You can skip this whole section. Studio signs you in and picks the provider for you. No API key needed.\n\n**Writing your own provider**\n\nThe provider interface is small and synchronous:\n\n``` python\nfrom enrichment import CompletionResult, Provider\n\nclass MyProvider(Provider):\n    name = \"my-provider\"\n    default_model = \"my-model\"\n\n    def complete(self, request):\n        value = call_my_service(\n            instructions=request.instructions,\n            input_data=request.input_data,\n            model=request.model or self.default_model,\n        )\n        return CompletionResult(content=value)\n```\n\nAdd automatic batch support by also implementing `BatchProvider`\n\n.\n\n**Registering a runtime provider**\n\nApplications that embed `enrichment`\n\ncan register a provider so users never configure anything:\n\n``` python\nfrom enrichment import register_provider\n\nregister_provider(\"application-runtime\", provider, priority=100)\n```\n\nProviders are chosen in this order:\n\n`provider=`\n\npassed to`enrich()`\n\n- OpenAI configured explicitly with\n`api_key=`\n\n- Highest-priority registered runtime provider\n- MLJAR account token from\n`MLJAR_RUNTIME_TOKEN_FILE`\n\n- OpenAI configured through\n`OPENAI_API_KEY`\n\n**Development**\n\n```\npython -m pip install -e \".[dev]\"\npython -m pytest\n```\n\nLive OpenAI tests are skipped by default because they make paid API requests:\n\n```\nRUN_LIVE_API_TESTS=1 OPENAI_API_KEY=\"your-api-key\" python -m pytest -m live\n```\n\nThe live Batch API test opts in separately, because it can take several minutes:\n\n```\nRUN_LIVE_BATCH_TESTS=1 OPENAI_API_KEY=\"your-api-key\" \\\n  python -m pytest -m live -k batch\n```\n\nApache 2.0. See [LICENSE](/mljar/enrichment/blob/main/LICENSE).\n\nMade by [MLJAR](https://mljar.com). Found a bug or have an idea? [Open an issue](https://github.com/mljar/enrichment/issues).", "url": "https://wpnews.pro/news/show-hn-data-enrichment-with-ai-for-pandas-dataframe", "canonical_source": "https://github.com/mljar/enrichment", "published_at": "2026-08-20 09:48:41+00:00", "updated_at": "2026-08-20 10:15:34.512108+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["enrichment", "pandas", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-data-enrichment-with-ai-for-pandas-dataframe", "markdown": "https://wpnews.pro/news/show-hn-data-enrichment-with-ai-for-pandas-dataframe.md", "text": "https://wpnews.pro/news/show-hn-data-enrichment-with-ai-for-pandas-dataframe.txt", "jsonld": "https://wpnews.pro/news/show-hn-data-enrichment-with-ai-for-pandas-dataframe.jsonld"}}