A read-only-first Databricks FinOps toolkit and Agent Skill for finding cost spikes, explaining their workload impact, and applying only explicitly approved optimizations.
It works in both Codex and Claude Code:
- Codex discovers
.agents/skills/optimize-databricks-costs/
. - Claude Code discovers
.claude/skills/optimize-databricks-costs/
. - Both copies use the same open
SKILL.md
format and are kept identical by CI.
- SQL polling intervals that repeatedly beat warehouse auto-stop
- Oversized or continuously active SQL warehouses
- Always-running Databricks Apps with little demonstrated use
- Failed, cancelled, duplicate, or canary Job runs
- High-cost users and service principals, resolved to their real workloads
- Missing cost-allocation tags and weak budget coverage
- Savings proposals that silently create freshness, latency, or availability impacts
The default workflow is read-only. It separates work into four explicit stages:
- Audit current cost and usage.
- Propose optimizations.
- Identify every affected workload and obtain approval for the impact.
- Apply only approved changes, with rollback commands and live verification.
The bundled audit script uses Databricks system tables and inventory APIs. It does not edit warehouses, stop Apps, Jobs, change permissions, or create budgets.
- Python 3.10+
- Databricks CLI 0.229 or newer; the live test used 0.296
- A configured Databricks CLI profile
- Access to a running SQL warehouse
- Permission to read the relevant
system.billing
,system.query
,system.lakeflow
, andsystem.compute
tables
Account-admin access gives the broadest audit, but the toolkit reports whatever the current identity is allowed to see.
git clone https://github.com/kylehuirevvision/databricks-cost-optimizer.git
cd databricks-cost-optimizer
codex
Then invoke:
$optimize-databricks-costs audit my Databricks account from a cost perspective
Codex loads repository skills from .agents/skills
according to the official OpenAI skill documentation.
git clone https://github.com/kylehuirevvision/databricks-cost-optimizer.git
cd databricks-cost-optimizer
claude
Then invoke:
/optimize-databricks-costs audit my Databricks account from a cost perspective
Claude Code loads project skills from .claude/skills
according to the official Claude Code skill documentation.
Find a warehouse ID:
databricks warehouses list
Run the read-only audit:
python3 .agents/skills/optimize-databricks-costs/scripts/audit.py \
--warehouse-id YOUR_WAREHOUSE_ID \
--recent-days 14 \
--output-dir reports/latest
Use a non-default CLI profile with --profile PROFILE
. Preview every SQL statement without connecting to Databricks with --dry-run
.
The command writes JSON evidence and a starter Markdown summary under reports/
. That directory is ignored by Git because reports commonly contain account IDs, emails, query metadata, and resource names.
If any system-table query fails, the collector still writes the evidence it obtained but exits nonzero and marks the summary incomplete. Missing evidence is never reported as zero cost.
Sanitized Databricks billing view. August 10 is a partial day and is excluded from the comparison below.
In the first account where this workflow was used, completed daily effective-list-price usage fell from $343 on August 5 to $108 on August 9, a 68.5% reduction. SQL usage fell from $257 to $42, an 83.7% reduction, and accounted for most of the measured improvement.
This is an observed before-and-after result, not a controlled experiment or a universal savings claim. Workload volume can vary, recent billing can be partial or restated, and negotiated invoice or underlying cloud-infrastructure costs may differ.
An anonymized engagement found tiny cached queries running approximately every 7.5 minutes against a warehouse with a 10-minute auto-stop. The queries performed almost no compute, but their cadence kept the warehouse billable for every hour in a two-week window. After stopping the recurrence, reducing warehouse size, shortening auto-stop, and stopping unused Apps, the following day's cost dropped materially.
See the sanitized case study for the reasoning pattern and caveats.
python3 -m unittest discover -s tests -v
python3 scripts/sync_skill.py --check
python3 scripts/check_public_safety.py
Maintainers who have Codex's built-in skill-creator
may additionally run its quick_validate.py
against both skill directories. Public users do not need that internal helper.
The audit estimates USD cost using the effective list price in system.billing.list_prices
. This is not necessarily the amount on a negotiated invoice. Recent billing records can arrive late or be restated, so equal-period comparisons exclude the current day and a configurable ingestion-lag window by default.