Catalyst: Building an AI Scientist for Theory Discovery

Imbue AI released Catalyst, an open-source framework that implements a discovery loop for AI agents to autonomously propose, test, and refine formal theories from data. The system uses a state-machine architecture that iterates through hypothesis generation, verification via Python scripts, and refinement based on error traces, requiring 10-20 LLM calls per task. Catalyst is designed for researchers seeking to automate hypothesis testing but faces constraints including high compute costs and dependency on available libraries.

Catalyst: Building an AI Scientist for Theory Discovery For those of us into LLM agents, the real value here isn't just the "scientist" label—it's the architecture. It's designed to handle the "discovery loop": observing a phenomenon, proposing a formal theory, testing that theory via code, and updating the theory based on the results. Getting Started with Catalyst To get this running locally, you'll need a Python environment 3.10+ recommended and an API key for your chosen LLM Claude /en/tags/claude/ 3.5 Sonnet is typically the best performer for this kind of reasoning . 1. Clone the repository and install dependencies: git clone https://github.com/imbue-ai/catalyst.git cd catalyst pip install -r requirements.txt 2. Configure your environment variables. Create a .env file in the root directory: ANTHROPIC API KEY=your api key here OPENAI API KEY=your api key here 3. Run a basic discovery task. You can point the agent at a dataset and a specific question you're trying to solve: python main.py --dataset path/to/your/data.csv --query "Find the relationship between X and Y" Technical Deep Dive: The Discovery Loop The core of the system is a state-machine-like loop. Most "AI scientists" just prompt a model once; Catalyst maintains a "Theory State" that evolves. Hypothesis Generation: The agent analyzes the data distribution and proposes a potential rule or mathematical relationship. Verification Step: It writes a Python script to test if the proposed theory holds across the entire dataset. Refinement: If the test fails e.g., the code throws an error or the result contradicts the theory , the error trace is fed back into the prompt as a "counter-example," forcing the model to pivot. Real-World Performance and Constraints From a practical standpoint, this is a powerful tool for prompt engineering researchers or data scientists who are tired of manually iterating through hypotheses. However, it has specific limitations: Compute Cost: Because it iterates Hypothesis - Test - Refine , a single discovery task can trigger 10-20 LLM calls. If you're using GPT-4o or Claude 3.5, the token spend adds up quickly. Dependency on Tooling: The agent is only as good as the libraries it can use. If your theory requires a niche physics library that isn't installed in the environment, the agent will hallucinate a solution or get stuck in a loop. If you want to customize the agent's behavior, you can modify the system prompts in the prompts/ directory. I found that adding a constraint to "prioritize Occam's Razor" in the theory generation prompt significantly reduced the number of overly complex, overfitted hypotheses the agent was producing.For a full walkthrough of the implementation, you can check the specific documentation here: https://github.com/imbue-ai/catalyst This is a solid foundation for anyone trying to move from simple RAG /en/tags/rag/ workflows to a full-scale AI workflow for research. It shifts the AI from a "chatbot that knows things" to an "agent that finds things." Next Codex Slides: Turning Repos into Decks from Scratch → /en/threads/2890/