In this demo, we automatically deduce a FlashAttention kernel with ~25 lines of SQL, which just expresses vanilla attention. This is possible due to three portable SQL extensions that we’re building: - xarray-sql: a library that efficiently represents petabyte scale array datasets in the cloud as tables (relations). - ddx: a database extension that adds automatic differentiation in the style of JAX to standard SQL, assuming the above array data model. - einfold: a modular query planner for fast tensor contractions in vanilla SQL, again, assuming the above array data model.
My hope is that the relational model and algebra provide fresh perspectives on how we might do accelerator programming. Here, our thesis is that separating the logical layers in a tensor program from the physical layers is both more correct and more scalable.
If the Bitter Lesson is a central dogma of machine learning, then why not formalize it with relational algebra?
If this interests you and you’d like to tinker with us at the intersection of database theory and ML systems, join us on Discord at [https://xql.systems](https://xql.systems) (link at the bottom). Cheers.
Comments URL: [https://news.ycombinator.com/item?id=50309479](https://news.ycombinator.com/item?id=50309479)
Points: 5