One of my biggest motivations for doing data work in Haskell has been the promise that types can enable better program verification and synthesis. In fact, it was one of my stretch goals for writing dataframe in the first place. Additionally, I had seen a video some years ago that notebooks are a good platform for program synthesis. My first swing at the problem was a SKILL.md that instructed an LLM on how to use Sabela notebooks. Large/frontier models didn’t struggle with writing Haskell but their contexts were typically more bloated by internet searches and churn from trying to fix simple compiler errors. Python was “in the weights” so was generated faster and with less tokens overall.
With small, local modes (<20b params) models the problem worsens. These models hallucinate Haskell modules, struggle with rule following, have smaller context windows. They typically fall back to writing Haskell from “the weights” and will steer clear of using dataframe or a newer library because a simple web search would bloat context.
The problem has two potential solutions:
In the spirit of synthesis I went with the second approach.
Siza, another Ndebele word, is a harness (and some associatd mcp tools) that drives a Sabela notebook. The goal of the harness is to address the problems above. I’ll follow up with a longer blog post on the sorts of interventions that made this possible but broadly speaking it’s a lot of type directed searching and a little bit of context management.
You can see a verbose transcript of how it performed with minimal to no guidance on an out of distribution task: Can you load the wine dataset into a dataframe and show some summary statistics about it?
Haskell is an amazing language for these sorts of tasks and the core of making it more useful is a pretty interesting engineering problem in my view. And small models, like testing anything in low resource environments, really teases those engineering problems out.
Again, will do a bigger blog post later.