Using Large Language Models for Hyperparameter Optimization

A new study from researchers demonstrates that large language models (LLMs) can match or outperform traditional hyperparameter optimization (HPO) methods such as Bayesian optimization within constrained search budgets. By prompting LLMs with dataset and model descriptions, the methodology iteratively refines hyperparameter configurations based on model performance, and also treats model code as a hyperparameter for greater flexibility.

This paper explores the use of foundational large language models LLMs in hyperparameter optimization HPO . Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.