SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

Researchers introduced SQuaD-SQL, a method that uses knowledge distillation from large language models to train small language models for text-to-SQL tasks, achieving 86.9% execution accuracy on WikiSQL with reduced computational requirements.

arXiv:2607.08161v1 Announce Type: new Abstract: Text-to-SQL is a fundamental task in natural language processing that enables users to interact with structured databases using natural language. While large language models LLMs have demonstrated remarkable performance on this task, their substantial computational requirements hinder deployment in resource-constrained settings. In this paper, we introduce SQuaD-SQL Small-Qualified and Distilled for SQL , a novel approach that empowers small language models SLMs to approach the performance of LLMs on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation. Our method comprises three key components: 1 LLM-based synthetic data generation, where structured knowledge is extracted from LLMs via carefully designed prompting strategies; 2 parameter-efficient fine-tuning, enabling full model training on a single consumer-grade GPU; and 3 domain-adaptive fine-tuning, where domain-specific synthetic data further enhances performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieves an execution accuracy of 86.9% on the test set, approaching the performance of LLMs while offering faster inference and lower memory usage. These results suggest that, with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments.