For those of us into LLM agents, the real value here isn't just the "scientist" label—it's the architecture. It's designed to handle the "discovery loop": observing a phenomenon, proposing a formal theory, testing that theory via code, and updating the theory based on the results.
Getting Started with Catalyst #
To get this running locally, you'll need a Python environment (3.10+ recommended) and an API key for your chosen LLM (Claude 3.5 Sonnet is typically the best performer for this kind of reasoning).
- Clone the repository and install dependencies:
git clone https://github.com/imbue-ai/catalyst.git
cd catalyst
pip install -r requirements.txt
- Configure your environment variables. Create a
.env
file in the root directory:
ANTHROPIC_API_KEY=your_api_key_here
OPENAI_API_KEY=your_api_key_here
- Run a basic discovery task. You can point the agent at a dataset and a specific question you're trying to solve:
python main.py --dataset path/to/your/data.csv --query "Find the relationship between X and Y"
Technical Deep Dive: The Discovery Loop #
The core of the system is a state-machine-like loop. Most "AI scientists" just prompt a model once; Catalyst maintains a "Theory State" that evolves.
Hypothesis Generation: The agent analyzes the data distribution and proposes a potential rule or mathematical relationship.Verification Step: It writes a Python script to test if the proposed theory holds across the entire dataset.Refinement: If the test fails (e.g., the code throws an error or the result contradicts the theory), the error trace is fed back into the prompt as a "counter-example," forcing the model to pivot.
Real-World Performance and Constraints #
From a practical standpoint, this is a powerful tool for prompt engineering researchers or data scientists who are tired of manually iterating through hypotheses. However, it has specific limitations:
Compute Cost: Because it iterates (Hypothesis -> Test -> Refine), a single discovery task can trigger 10-20 LLM calls. If you're using GPT-4o or Claude 3.5, the token spend adds up quickly.Dependency on Tooling: The agent is only as good as the libraries it can use. If your theory requires a niche physics library that isn't installed in the environment, the agent will hallucinate a solution or get stuck in a loop.
If you want to customize the agent's behavior, you can modify the system prompts in the
prompts/
directory. I found that adding a constraint to "prioritize Occam's Razor" in the theory generation prompt significantly reduced the number of overly complex, overfitted hypotheses the agent was producing.For a full walkthrough of the implementation, you can check the specific documentation here:
https://github.com/imbue-ai/catalyst
This is a solid foundation for anyone trying to move from simple RAG workflows to a full-scale AI workflow for research. It shifts the AI from a "chatbot that knows things" to an "agent that finds things."
Next Codex Slides: Turning Repos into Decks from Scratch →