Lumen 1.4.0 Release HoloViz released Lumen 1.4.0, an open-source framework for conversational data exploration in which an LLM proposes SQL, transformations and charts that stay visible, editable and re-runnable. The release lets `lumen-ai serve` run without an API key using the Claude Code, Codex and GitHub Copilot CLIs, Kilo's free models or a local Ollama, and adds choropleth maps from geometry columns, GeoJSON loading, an Annotate button that writes report summaries with Word export, per-call token and dollar tracking, and native Polars and PyArrow support. Lumen 1.4 requires Python 3.11 or later, Panel 1.9 or later and Bokeh 3.9 or later. Lumen 1.4.0 Release What is Lumen? Lumen https://lumen.holoviz.org/ is an open-source framework for conversational data exploration and analysis . You ask a question about your data, and an LLM proposes the SQL, the transformations and the chart. Every step stays visible, editable and re-runnable. Lumen is built on Panel https://panel.holoviz.org/ and the rest of the HoloViz https://holoviz.org ecosystem, so the results are interactive tables, plots and dashboards. If you are new to it, the Lumen 1.0 announcement ../../posts/lumen 1.0/index.html explains the design. New Release We are excited to announce the 1.4.0 release of Lumen This release makes Lumen easier to start and cheaper to run well. You no longer need an API key to try it, answers rest on better table context, and a report can write its own story. Here’s a high-level overview of the highlights: - Run it without an API key run-lumen-without-an-api-key : lumen-ai serve works with the Claude Code, Codex and GitHub Copilot CLIs you are already signed in to, with Kilo’s free models, or with a local Ollama. With nothing set up, it tells you which of these to try. - Maps from geometry columns maps-and-geospatial-data : a table’s own geometry column now draws as a choropleth, and GeoJSON files load end to end. - Reports with a story reports-with-a-story : an Annotate button writes a summary and a note under each section, and a report exports to Word. - Answers you can check answers-you-can-check : agents get a compact, exact profile of each table, tool errors go back to the model instead of ending the attempt, and every SQL result is checked for data-quality problems. - Usage and cost tracking usage-and-cost-tracking : see tokens and dollars per call, and trace every model and tool call. - Polars and PyArrow polars-and-pyarrow : sources can return polars or pyarrow frames, and filters and most transforms run on them natively. Lumen 1.4 requires Python 3.11 or later, Panel 1.9 or later and Bokeh 3.9 or later. Many thanks to the contributors to this release: @ghostiee-11 https://github.com/ghostiee-11 , @philippjfr https://github.com/philippjfr , @ahuang11 https://github.com/ahuang11 , @vignesh-chaturvedi https://github.com/vignesh-chaturvedi , @SajalDevX https://github.com/SajalDevX , @DevForge255 https://github.com/DevForge255 , @7487 https://github.com/7487 , @rimsha152007-dotcom https://github.com/rimsha152007-dotcom , @fly1d https://github.com/fly1d , @Vamshi1404 https://github.com/Vamshi1404 , @hoxbro https://github.com/hoxbro , @AmirF194 https://github.com/AmirF194 , @yiboban https://github.com/yiboban , @raashish1601 https://github.com/raashish1601 , @nightcityblade https://github.com/nightcityblade , @mayuriphad https://github.com/mayuriphad , @mahdibani https://github.com/mahdibani , @latent-9 https://github.com/latent-9 , @Zayn329 https://github.com/Zayn329 , @SuMayaBee https://github.com/SuMayaBee , @Mr-Neutr0n https://github.com/Mr-Neutr0n , @Coderambling https://github.com/Coderambling , @umi008 https://github.com/umi008 , @jadhavgaurav https://github.com/jadhavgaurav . A special welcome to our new contributors: @vignesh-chaturvedi https://github.com/vignesh-chaturvedi , @SajalDevX https://github.com/SajalDevX , @DevForge255 https://github.com/DevForge255 , @7487 https://github.com/7487 , @rimsha152007-dotcom https://github.com/rimsha152007-dotcom , @fly1d https://github.com/fly1d , @Vamshi1404 https://github.com/Vamshi1404 , @AmirF194 https://github.com/AmirF194 , @yiboban https://github.com/yiboban , @raashish1601 https://github.com/raashish1601 , @nightcityblade https://github.com/nightcityblade , @mayuriphad https://github.com/mayuriphad , @mahdibani https://github.com/mahdibani , @latent-9 https://github.com/latent-9 , @Zayn329 https://github.com/Zayn329 , @Mr-Neutr0n https://github.com/Mr-Neutr0n , @umi008 https://github.com/umi008 , @jadhavgaurav https://github.com/jadhavgaurav . Run Lumen Without an API Key Without an API key, lumen-ai serve used to start against a provider that was not there. In 1.4 there are four ways to get going without one. Coding CLIs If you are signed in to Claude Code, Codex or the GitHub Copilot CLI, Lumen can use it as its LLM: lumen-ai serve penguins.csv --provider claude-code The other values are codex-cli and copilot-cli , and antigravity-cli is available too. The Claude Code and Copilot providers run the CLI with its own tools switched off and in an empty temporary directory, so it answers Lumen’s prompts without reading the folder you launched from. One limit: Lumen’s own tools are not forwarded to these CLIs. An action that needs the model to call a tool, such as fetching a URL from the chat, cannot run, so pass your data on the command line. Free models with Kilo lumen-ai serve penguins.csv --provider kilo Kilo’s free tier needs no account. It is a shared service, so expect it to be slow and to have limits. In our tests a question that ended in a chart took five to six minutes, and after roughly 40 minutes of steady use it answered with a 429 that points to a free sign-up. The free router also changes which model it uses over time, and some of those models reject the structured requests the planner makes. When Kilo answers with an upstream error, Lumen now falls back to asking for JSON in plain text and keeps doing that for the session. For a local model, Lumen reads OLLAMA HOST , and the local providers now default to a 9B model, which you can fetch with ollama pull qwen3.5:9b . If nothing is set up, Lumen no longer guesses. It picks a provider only when its API key is set, and otherwise prints what would work on your machine: ERROR: No LLM provider found: no API key is set. Pick one of these to get started: - Free models, no signup: --provider kilo - Use a coding CLI you are signed in to: --provider claude-code - Sign in to one of these coding CLIs first: --provider copilot-cli, --provider antigravity-cli - Set an API key, e.g. OPENAI API KEY, ANTHROPIC API KEY or GEMINI API KEY - Use an OpenAI-compatible server: --provider openai --provider-endpoint URL - Ollama is running with no models: ollama pull qwen3.5:9b Docs: https://lumen.holoviz.org/configuration/llm providers/ Maps and Geospatial Data A table that has a geometry column can now be drawn as a map directly. Start Lumen on a GeoJSON file, a WKT file or a zipped shapefile, and ask for the map: lumen-ai serve countries.geojson --provider claude-code Map each country’s estimated population as a choropleth on a log colour scale using the Equal Earth projection, with a legend. The chart is a plain Vega-Lite spec, shown above the map. If something is off, edit the YAML or ask for the change. Maps are interactive: hover a state to see its values, and the view follows the light or dark theme. Around that feature, geospatial support got a round of fixes: - GeoJSON uploads load end to end 1996 https://github.com/holoviz/lumen/pull/1996 , and the coordinate reference system of a GeoDataFrame survives ingest into DuckDB 2077 https://github.com/holoviz/lumen/pull/2077 . - A choropleth whose lookup named an unregistered dataset no longer renders blank 2001 https://github.com/holoviz/lumen/pull/2001 , and point layers on a basemap are fed the table 2151 https://github.com/holoviz/lumen/pull/2151 . - The DeckGL prompt knows GeoJsonLayer and data-driven colours 1999 https://github.com/holoviz/lumen/pull/1999 , and datetime columns in GeoJSON layers serialize 1997 https://github.com/holoviz/lumen/pull/1997 . - Geometry now renders in Vega-Lite views, both from a table’s own geometry and when a spec joins external boundaries 2232 https://github.com/holoviz/lumen/pull/2232 . - Gridded data handles cftime dimensions on xarray-sql 0.6, and a to dataset result that is not a grid is no longer gridded 2144 https://github.com/holoviz/lumen/pull/2144 , 2049 https://github.com/holoviz/lumen/pull/2049 . Charts From a Sentence Maps are one case of a general ability: describe a chart and Lumen writes the SQL and the spec. Here is a prompt with several moving parts, run on the Gapminder data: For the ten most populous countries in 2005, draw a connected scatter plot of fertility against life expectancy from 1955 to 2005, with bubbles sized by population at each point and the country name at the 2005 end of each path. A language model writes each chart, so a rerun gives a different one, and a first pass sometimes needs a follow-up. In this one, a few labels near the bottom right overlap. The specs and SQL behind this figure are in the repro/ folder of this post. Follow-ups work on the chart you already have. Here Lumen draws UK electricity per capita from fossil fuels, nuclear and renewables as a stacked area, and a follow-up can redraw it, for example as a line chart comparing just fossil fuels and renewables: The Vega-Lite agent also got better at stacking and ordering: asked for the US athletes with the most Olympic medals, it draws a stacked bar by medal type 1992 https://github.com/holoviz/lumen/pull/1992 . Two more prompts on data-centre electricity demand: a slope chart that highlights the United States in red, and a ranking bar chart that labels each bar with its value. Reports With a Story A Lumen report collects the tables and charts from your explorations. In 1.4 you can ask it to write the story around them. The Annotate button uses an LLM to write an overall summary at the top and a short note under each section. The narrative shows in the report and flows into the Notebook and HTML exports. A report can now also be exported as a Word document. The .docx carries the headings, the narrative, native tables and the charts as images, respects the section selection, and includes the story and notes 1924 https://github.com/holoviz/lumen/pull/1924 . The story’s prose is editable with Panel’s TextEditor 2017 https://github.com/holoviz/lumen/pull/2017 , so you can fix a sentence instead of regenerating the whole thing. The Word export needs python-docx , which Lumen now installs. Answers You Can Check Most of the engine work in this release is about giving the model better facts and catching mistakes cheaply. A compact, exact view of your tables The table context agents see used to be built from a shuffled 100-row sample with no SQL types or keys. It could list wrong ranges and incomplete value lists, it dropped NULLs, and it changed on every run, which defeated prompt caching. It is now profiled with the cheapest accurate strategy for each source: engine metadata first, one exact pass for tables up to a million rows, and a seeded sample above that 2105 https://github.com/holoviz/lumen/pull/2105 . This is what the model sees for the penguins data: python import asyncio from lumen.ai.schemas import get metaset from lumen.sources.duckdb import DuckDBSource url = "https://datasets.holoviz.org/penguins/v1/penguins.csv" source = DuckDBSource uri=":memory:", tables={"penguins": f"read csv '{url}', nullstr='NA' "}, print asyncio.run get metaset source , "penguins" .compact context penguins 344 rows species VARCHAR {Adelie:152, Gentoo:124, Chinstrap:68} island VARCHAR {Biscoe:168, Dream:124, Torgersen:52} bill length mm DOUBLE 32.1..59.6 nulls <1% bill depth mm DOUBLE 13.1..21.5 nulls <1% flipper length mm BIGINT 172..231 nulls <1% body mass g BIGINT 2700..6300 nulls <1% sex VARCHAR {male:168, female:165} nulls 3% year BIGINT 2007..2009 Types, value counts, ranges and null shares are in one line per column, and the output is the same on every run. On the 11 BIRD Mini-Dev databases, the rendered schema context shrank from 22.4k to 16.1k tokens while carrying more facts, according to the pull request that introduced it. Tool errors go back to the model A tool call that failed, named a tool that does not exist, or sent invalid arguments used to end the attempt. It is now returned to the model as the tool’s result, so the model can correct itself in the same loop. Identical calls to read-only tools are answered from a cache, and SQLAgent submits its query as a tool call, so a rejected query is fixed inside the loop 2104 https://github.com/holoviz/lumen/pull/2104 . Data-quality checks on every result Every SQL result is profiled for missing values, duplicate rows, placeholder numbers, untrimmed text, numbers stored as text and outliers. The profile needs no LLM call. Only when something is worth fixing does SQLAgent spend one more call rewriting the query 2000 https://github.com/holoviz/lumen/pull/2000 . python import pandas as pd from lumen.ai.data quality import lint data readings = pd.DataFrame { "station": "north ", "south", "east", "west", "north ", "south" , "temp c": 12.5, 13.0, -9999, 14.0, -9999, 15.0 , } for finding in lint data readings : print "-", finding print "mean with the placeholders:", readings "temp c" .mean - Placeholder numbers -9999.0 appear as data in "temp c" 2 rows ; they almost certainly encode missing values. - Untrimmed or empty text in "station" 2 padded, 0 empty . mean with the placeholders: -3323.9166666666665 Without the check, the placeholders would have gone straight into an average and a chart. Pass SQLAgent clean data=False to turn the pass off. Usage and Cost Tracking LLM providers now count the tokens they report, and, if you give them prices, what those tokens cost. This example needs no API key, because it uses a free model on Kilo: python import asyncio from lumen.ai.llm import Kilo MODEL = "nvidia/nemotron-3-ultra-550b-a55b:free" async def main : llm = Kilo model kwargs={"default": {"model": MODEL}}, usage pricing={MODEL: {"input": 0.0, "output": 0.0}}, USD per million tokens with llm.capture usage as usage: await llm.invoke {"role": "user", "content": "Name one penguin species."} print usage.input tokens, usage.output tokens, usage.cost usd asyncio.run main 22 39 0.0 The token counts vary from call to call. usage pricing takes prices in USD per million tokens, keyed by the model name the provider returns. capture usage collects everything inside its scope, including tool rounds, retries and calls from child tasks, while llm.usage totals the lifetime of the instance 2100 https://github.com/holoviz/lumen/pull/2100 . To see what happened inside a request, with llm.trace as events: records every provider round trip and completed tool call in order, with arguments, responses, errors and durations 2101 https://github.com/holoviz/lumen/pull/2101 . Two opt-in building blocks for spending less landed as well: model routing, where a model kwargs entry with a "routing" key picks a model per call by task complexity 2015 https://github.com/holoviz/lumen/pull/2015 , and a DecisionModel interface for closed-choice questions such as follow-up classification 2109 https://github.com/holoviz/lumen/pull/2109 . Nothing in the coordinator uses the decision model yet. Polars and PyArrow A Source no longer has to return pandas. Any dataframe library that narwhals https://narwhals-dev.github.io/narwhals/ supports is accepted, and schema generation, Filter and the common transforms run on it natively 2008 https://github.com/holoviz/lumen/pull/2008 : python import polars as pl from lumen.pipeline import Pipeline from lumen.sources.base import InMemorySource from lumen.transforms import Aggregate, Filter url = "https://datasets.holoviz.org/penguins/v1/penguins.csv" penguins = pl.read csv url, null values="NA" pipeline = Pipeline source=InMemorySource tables={"penguins": penguins} , table="penguins", transforms= Filter conditions= "island", "Biscoe" , Aggregate method="mean", by= "species" , columns= "body mass g" , with index=False , , print type pipeline.data print pipeline.data