DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation

Researchers from HKUST introduced DS-Lighting, a unified harness toolkit that makes agent harness design explicit for data-science automation, decomposing it into four reusable layers: data, workflow, execution, and evaluation. The toolkit integrates multiple open-source benchmarks into an MLE-Bench-style format, and experiments show that explicit harness design improves reproducibility, comparability, and reliability while reducing system-level failures. The code is available on GitHub.

arXiv:2608.28590v1 Announce Type: new Abstract: Large Language Model LLM agents have shown promise for automating data-science workflows, yet their end-to-end performance depends critically on the agent harness that represents tasks, manages execution state, constrains output artifacts, and provides evaluation feedback. Existing data-science agents often leave this harness implicit, making results difficult to reproduce, compare, and attribute across heterogeneous tasks. We introduce DS-Lighting, a unified harness toolkit that makes harness design explicit for data-science automation. DS-Lighting decomposes the harness into four reusable layers: data, workflow, execution, and evaluation, and represents diverse agents as executable operator programs that support both predefined pipelines and adaptive search. We further integrate multiple open-source data-science benchmarks into an MLE-Bench-style task format, enabling controlled comparison under a shared task interface, sandboxed runtime, and metric protocol. Experiments across agents, harnesses, models, and ablations show that explicit harness design improves reproducibility, comparability, and reliability, while reducing avoidable system-level failures in end-to-end data-science workflows. Our code is available at https://github.com/usail-hkust/dslighting