Extending Polars with Rust Expression Plugins

Fenic, a semantic DataFrame library for AI and LLM pipelines built on Polars, developed nine Rust expression plugins to extend Polars' native capabilities for text operations like chunking, prompt templating, jq queries, fuzzy matching, and markdown parsing. The plugins, written via pyo3-polars, replace slow Python UDFs with native expressions that run in-engine over Arrow, preserve declared types, and compose with built-in operations in a single expression tree. Fenic's CEO stated that Python UDFs cost too much on speed, composition, and type safety for these workloads.

fenic is a semantic DataFrame library. A PySpark-style API for building AI and LLM pipelines over messy, unstructured data. Its local engine is Polars. This post is about one specific problem we hit while building it, and the part of Polars that solved it: the expression plugin system. We ended up writing nine Rust plugins that extend Polars' expression engine. What follows is both why we went that route and how they're built, with the real code. tl;dr. The operations an AI pipeline needs over text chunking, prompt templating, jq, fuzzy matching, markdown and transcript parsing, richer type casts aren't in Polars. Doing them as Python UDFs is slow, and it breaks composition. Writing them as Polars expression plugins in Rust, via pyo3-polars https://github.com/pola-rs/pyo3-polars , turns them into native expressions. They run in-engine over Arrow, they keep their declared types, and they compose with built-in ops in a single expression tree. If you're weighing a UDF against a plugin, this is the case for the plugin. Why we built this What is fenic fenic is a DataFrame library for building AI and LLM pipelines, with an API modeled on PySpark. If you come from Polars, the important thing to know is that Polars is the core execution engine for fenic. Every DataFrame operation you write becomes a Polars expression, or a plan over pl.DataFrame s. The semantic operators, the LLM map/extract/classify, the embeddings, the similarity joins, all sit on top of that same machinery. So in practice, what fenic can do is bounded by what we can express in Polars. That's a deliberate bet. Polars gives us a fast, columnar, Arrow-native engine with a real expression language and a lazy optimizer. We had no interest in rebuilding any of it. We wanted to add to it. What we wanted to do The workloads fenic targets are pipelines over unstructured text. Documents, chat logs, transcripts, scraped JSON, markdown. Concretely, that means row-level operations Polars has no native equivalent for: Chunk a document into overlapping windows sized by token count, for embedding or retrieval. Render a prompt template per row. Real Jinja, with a struct of columns as the variables, ahead of an LLM call. Query JSON columns with jq , and parse markdown into a structured AST. Fuzzy-match strings six edit-distance metrics for dedup and joins. Parse transcripts SRT/WebVTT into typed, timestamped cue records. Cast values into fenic's richer logical types embeddings, markdown, typed structs that Polars' physical dtypes don't model directly. Each of these has to run over a whole column, produce a typed result, and slot into the middle of a larger DataFrame pipeline. Not sit off to the side. Where the obvious approaches fell short The reflexive way to add a custom operation to Polars is a Python UDF. map elements for per-row work, map batches for whole-Series work. For genuinely opaque, IO-bound work, like an LLM API call, that's still the right tool, and fenic uses map batches for exactly that. But for the text operations above, UDFs cost too much on three axes. Speed. map elements runs Python per row, under the GIL, with a Python-object round-trip for every value. For tokenizing or fuzzy-matching millions of rows, that's the bottleneck. Not the work itself. Composition. This is the one that actually hurt. Polars is fast because it plans and executes an expression tree as a whole. The moment one step is an opaque Python callback, the engine can't see through it. It becomes an optimization barrier, forces a materialization, and breaks the single-pass pipeline. Chain three UDFs and you've got three trips out of the engine and back. Types. A UDF's output type is something you assert loosely and hope holds. We needed operations that produce real, declared dtypes, like List<String , Struct , fixed-size embedding arrays, so the rest of the plan can type-check against them. The alternatives were worse. Forking Polars to add native kernels means owning a fork forever. Doing the text work outside the DataFrame, preprocess then load, throws away the laziness and composition that are the whole reason to use Polars in the first place. We wanted these operations to be first-class citizens of the expression engine. Not neighbors of it. Why plugins Polars has a purpose-built answer for exactly this: expression plugins. You write a kernel in Rust, register it, and it becomes a normal pl.Expr . To the engine, it's indistinguishable from a built-in. That checks every box the UDF route missed. It runs in-engine, in Rust, over Arrow buffers. Vectorized, parallelizable, eligible for streaming, with no Python and no per-row object round-trip. It returns a pl.Expr , so it composes. Plugins chain with each other and with native ops in one expression tree the engine plans as a whole. It declares its output dtype, so the plan stays typed through the custom step. And Rust gives us a mature ecosystem for the hot loops jaq , minijinja , rapidfuzz , tiktoken , a markdown parser without reimplementing any of it. The rule we settled on isn't "rewrite everything in Rust." Native Polars stays the fast path. A plugin only fills a gap Polars can't express: a tokenizer, a jq engine, a regex whose capture-group index is itself a column. Even the fuzzy matcher keeps just its six primitive kernels in Rust and composes the higher-order ratios from ordinary Polars expressions. pyo3-polars generates the FFI, the Arrow marshaling, and the keyword-argument bridge, so the cost of writing one is low. We wrote nine. What it bought us The end state is worth stating before the mechanics. Every one of fenic's text operations is now a native Polars expression. They run inside the engine over shared Arrow memory, they keep their types, and here's the part that matters most. They compose with native Polars ops in a single expression, with zero Python round-trips. Parsing a markdown document, filtering its AST with jq, indexing into the result, and casting it into a typed struct is one expression. Polars plans and executes it in a single pass, custom Rust and built-in list ops side by side. We'll come back to that exact example once the pieces are on the table. The rest of this post is how it's built, from the Python registration down to the Arrow memory, using nothing but the real code. How it's built: an implementation walkthrough The mental model: two thin layers around a contract A Polars expression plugin is two small pieces of code with a well-defined contract between them: Python side. Register a function so it looks like native Polars: expr.my namespace.my op ... . Rust side. A function that takes & Series , returns PolarsResult<Series , and declares its output dtype. Everything between them is generated for you by pyo3-polars . Moving Series across the FFI boundary via Arrow, marshaling keyword arguments, wiring the symbol lookup. You never touch the FFI by hand. Here's the entire Python surface for fenic's json.jq operator: Two Polars APIs are doing the work: @pl.api.register expr namespace "json" bolts a .json accessor onto every pl.Expr in the process. After import, pl.col "payload" .json.jq ".name" is a legal expression anywhere Polars expressions are legal. register plugin function ... returns a normal pl.Expr that, when the engine evaluates it, dlopens the compiled library at plugin path , looks up the symbol named function name , hands it the input Series , and reads back the result. That's the whole idea. .json is not special-cased anywhere in Polars. It's a user-registered namespace, and fenic registers nine of them json , jinja , markdown , chunking , tokenization , fuzz , regexp , dtypes , transcript . The plugin looks exactly like a built-in because, to the expression engine, there's no meaningful difference. New to plugins? The Polars plugin docs https://docs.pola.rs/user-guide/plugins/expr plugins/ and the pyo3-polars https://github.com/pola-rs/pyo3-polars repo are the canonical references. Walking through one plugin, end to end The Python jq method above points at a Rust symbol called jq expr . Here it is, in full, on the other side of the boundary: Everything you need to understand the contract is in that snippet: Signature. The polars expr ... attribute macro from pyo3 polars::derive turns an ordinary Rust function into an exported, C-ABI symbol that Polars can dlopen and call. Your function just receives inputs: & Series and returns PolarsResult<Series . No unsafe , no manual pointer juggling. The macro generates the Arrow marshaling. Typed inputs. inputs 0 .str ? downcasts the first Series to a StringChunked . If the column isn't a string, you get a clean PolarsError instead of a segfault. Do work once, not per row. The jq filter is compiled a single time build jq query and reused across every row. This is a recurring pattern. Hoist compilation, allocation sizing, and setup out of the row loop. Build the output with a builder. ListStringChunkedBuilder , pre-sized with a capacity estimate, produces the output column. The null semantics are deliberate. append null fires for a null input row or an empty jq result. A failed jq query aborts the batch with a ComputeError . And malformed JSON hits an unreachable , because the input column is typed JsonType and upstream validation guarantees every non-null row is valid JSON. That guarantee is a nice side effect of building on a typed engine. The kernel gets to assume its inputs. Return a Series . builder.finish .into series . The jaq crate a pure-Rust jq does the actual filtering. Polars provides the columnar plumbing. The plugin is the seam between them. Output types, and why they matter Look again at the attribute: polars expr output type func=jq output . That jq output function returns DataType::List String and never touches the data. This is the single most important thing to get right about plugins, and the easiest to get wrong. Polars resolves the schema of an expression before it executes it. A plugin is opaque Rust. The engine can't infer what comes out by looking at your loop. So you must declare the output dtype up front, and the declaration has to run without the data. fenic uses all three declaration styles the macro supports, picked by how much the output type depends on. 1. Static. The type never changes. All six fuzzy-match kernels always return a Float64 : tokenization.count tokens is likewise a fixed polars expr output type=UInt32 . 2. From input fields. The type is a function of the inputs. jq and text chunking both always produce List<String , declared by a small function that receives the input Field s and here ignores them . 3. From the arguments. The type depends on kwargs. This is where it gets interesting. fenic's dtypes.cast kernel implements casting into fenic's own logical type system, so its output dtype is literally an argument, passed as a serialized JSON type descriptor. The output-type function deserializes that JSON to compute the Polars dtype at plan time: The lesson: output type func with kwargs lets a plugin's result type be computed from its configuration, decoded at schema-resolution time. That's what makes it possible to bolt an entirely foreign type system fenic's logical types, including things like fixed-size embedding arrays onto Polars' physical dtypes while keeping lazy schema resolution intact. Your plugin declares, up front and data-free, exactly what shape it will produce. One honest caveat, since we're holding this up as exemplary. The real code .expect s on a malformed type descriptor, so a bad JSON string panics the schema-resolution path rather than surfacing a clean PolarsError . Hardening it would map that to a ComputeError . Get this wrong, declare List<String and then build a Float64 , and you don't get a compile error. You get a runtime failure when the produced column doesn't match the promised schema. The declaration is a contract you're on the hook for. Getting parallelism and broadcasting right Every fenic plugin passes is elementwise=True . That flag is a promise to the engine: this function treats each row independently. Row i of the output depends only on row i of the input. No ordering, no windows, no cross-row state. That promise is worth a lot. It makes the function eligible to run in Polars' streaming engine, and it lets the optimizer reason about it: predicate, projection, and slice pushdown, plus a group by fast path where the function runs over the flattened series instead of once per group. It is also load-bearing. If you set is elementwise=True on a function that actually looks across rows, Polars may hand it a partial batch a streaming morsel , or run it over flattened groups inside a group by . Either of which silently breaks cross-row logic. The flag isn't a hint. It's an invariant you're asserting. "Elementwise" still has to cope with broadcasting. The common case is where one argument is a scalar literal length 1 and another is a full column. fenic handles this two ways depending on the kernel. For the fuzzy matchers, which take two string columns, it leans on a Polars helper that does broadcasting and null handling for you: Two things worth stealing here. First, broadcast binary elementwise from polars::chunked array::ops::arity handles length-1 broadcasting and threads Option<&str through so nulls are explicit. The closure decides what a null means here, null in, null out . Second, note the code comment. We deliberately avoided the values variant that skips the null check, because for UTF-8 string work you do not want the kernel touching the bytes at a null slot. That's a decision you only make after it has already bitten you once. The regex kernel takes the manual route because it needs a per-call compile cache anyway: The broadcasting is a one-liner if pattern is literal { 0 } else { i } . The HashMap<String, Regex compile cache means a literal pattern is compiled once, and every subsequent row is a cache hit, via the Entry API: Because this cache lives for the duration of one call on one chunk, even a per-row-varying pattern column only ever compiles each distinct pattern once. The Arrow boundary, and zero copy Two of fenic's plugins, the Jinja renderer and the markdown/JSON path, need to turn arbitrary Arrow values strings, ints, structs, lists, nested combinations into a foreign representation. A minijinja::Value for template rendering, or a serde json::Value for JSON emission. This is the point where a naive plugin reaches for Series iterators, or worse, materializes Python objects. fenic reads the Arrow arrays directly. It's a deliberate rewrite of an earlier AnyValue -based path that required unnecessary allocations. Reading the Arrow buffer is zero-copy. The only allocation is the one output value the converter actually has to build. The whole thing hangs off one trait and one generic function. The trait abstracts "how do I build a value of type V from an Arrow scalar": It's implemented once for minijinja::Value and once for serde json::Value . Then a single generic converter walks any Arrow array and produces either. The two public methods just pin the concrete type: A few things worth pulling out: It works on It downcasts to the concrete Arrow array &dyn Array from polars arrow , not on Series . Utf8ViewArray , StructArray , ListArray<i32 , … and reads the value at row idx in place. No per-value boxing into a Polars AnyValue , no Python objects.This is the detail that bites people. Modern Polars represents strings as Utf8View is handled explicitly, alongside Utf8 and LargeUtf8 . Utf8View by default. But data zero-copy-imported from external Arrow pyarrow, some IPC/Parquet paths can still arrive as Utf8 or LargeUtf8 , which is exactly why the converter matches all three. A plugin that handles only one layout will fall through and error on perfectly ordinary string columns. If you read Arrow arrays directly, you have to know every physical layout Polars might hand you. Nesting is recursion. convert struct calls convert on each field. convert list calls convert on each element. A List<Struct<... just falls out of the recursion. One function handles arbitrarily nested Arrow. One implementation, two outputs. Adding a third target say, a MessagePack value is one more impl ArrowToValue . The traversal never changes. This is the difference between a plugin that works on strings and one that treats the Arrow memory as a first-class input. Beyond scalars: returning List<Struct Plugins aren't limited to scalar columns. fenic's transcript parser takes raw SRT/WebVTT text per row and emits a list of typed cue records. That's a List<Struct column with a fixed unified schema, regardless of input format. Its output-type function declares the shape of a single cue as a Struct and validates the format name while it's there : And the rows are assembled with AnyValue , building each struct explicitly and collecting them into a list: The kernel builds each cue as an AnyValue::StructOwned , its fields lining up with the cue schema from the output-type function, and collects a row's cues into an AnyValue::List . So the column comes out as List<Struct . A parse failure on one row becomes a null for that row, rather than a hard error for the whole batch. Structured output plus honest null semantics is what lets a downstream .unnest or .explode treat the result like any other native nested column. Configuration across the boundary The bridge for keyword arguments is just serde. On the Python side you build a plain dict marshaling enums to their string values : On the Rust side you declare a derive Deserialize struct with matching field names, and it arrives populated: Polars serializes the kwargs dict, and the derive macro deserializes it into your struct via serde before your function runs. The only discipline it demands is that the two sides agree on names and shapes. There's no schema enforcing it, so the Python dict and the Rust struct are a contract you keep by hand. Option<T fields map cleanly to Python None . Building and shipping The whole thing is one Rust crate compiled to a single dynamic library, built by maturin . The relevant Cargo.toml : cdylib produces the loadable library. rlib keeps it usable from Rust integration tests and benchmarks. pyo3-polars with the derive feature is what gives you the polars expr macro. Note the version coupling. pyo3-polars pins exact polars , polars-arrow , and pyo3 versions, and your plugin links against Arrow types that must be ABI-compatible with the Polars that loads it. So the whole stack has to move as one. On the Python packaging side, maturin is told to place the built module inside the fenic package: So the compiled artifact lands at fenic/ polars plugins.abi3.so . That's why every plugin file computes PLUGIN PATH = Path file .parents 3 . From .../fenic/ backends/local/polars plugins/json.py , three parents up is .../fenic , the package directory that holds the .so . register plugin function plugin path=... points there, and Polars finds the library by name. And because that Rust stack pins in lockstep, the Python dependency is pinned tight too: polars =1.34.0,<1.36.0 . This is the plugin author's tax. Bump polars on its own and you get two incompatible copies of the polars crates in the dependency graph, and a build that fails with type mismatches in every polars expr function. The crates move together, deliberately. A jump to a new Polars can even mean real code changes, not just a lockfile edit. The benefits for fenic Earlier I said the payoff was composition. Now that the pieces are on the table, here it is. Everything composes. Because every plugin returns a plain pl.Expr , plugins compose with each other and with native Polars operations in a single expression. This is fenic's implementation of "extract code blocks from a markdown document." Four operations, three of them custom Rust plugins, one of them a built-in: That .list.get 0 sitting between .json.jq ... and .dtypes.cast ... is the whole point. To the engine there is no boundary between the plugins and the native op. It's one expression tree, planned once, executed in one pass over the data, entirely inside Polars. No intermediate Python lists, no per-step Series -to-Python-to- Series marshaling, no leaving the engine and coming back. It runs the other direction too, with native expressions feeding a plugin. fenic builds LLM prompts by packing columns into a native pl.struct and handing it to the Jinja plugin: The plugin receives a struct column, and the ArrowScalarConverter from earlier is what lets the Rust side read that struct's fields as Jinja variables. Native op produces the struct, plugin consumes it, one expression. Once your custom kernels are pl.Expr , the entire Polars expression language, every native string, list, struct, temporal, and arithmetic op, becomes glue you can put between them. One build, two surfaces. A quieter benefit of writing the logic in Rust. You can expose slices of it as plain Python functions too, not only as expressions. fenic validates jq filters and regex patterns at plan time, when a query is being built, before any data flows, using the exact same Rust that executes them: py validate jq query calls the same build jq query the jq expr kernel uses, and raises a Python ValueError on a bad filter. So an invalid query is caught the moment the user writes it, with the identical parser that will run it later, rather than blowing up mid-execution. This pattern started with regex. Python's re was accepting patterns that then failed under Polars' Rust regex engine at runtime, and validating with the same engine ended the divergence. The Rust crate is both a plugin library and an ordinary pyo3 extension module. You get both surfaces from one build. Add it up, and the plugin system is what let fenic put an entire AI-oriented operation set tokenizing, templating, parsing, matching, casting inside Polars rather than beside it. The semantic layer sits on top. The fast, typed, composable substrate underneath is Polars, extended rather than wrapped. Lessons for plugin authors Distilled from building the nine: The output-type declaration is a hard contract, unchecked at compile time. Declare exactly what you build. output type func with kwargs is the escape hatch when the type depends on configuration, enough to graft a whole foreign type system on, as dtypes.cast does. Match every physical Arrow layout, not just the logical dtype. Utf8View as well as Utf8 and LargeUtf8 , or you'll error on perfectly ordinary string columns. And keep the Python kwargs dict and the Rust Deserialize struct in lockstep. serde is the whole bridge, and nothing but you enforces it.And one you can break silently. While you're at it, hoist setup compile, allocate, size builders out of the row loop. is elementwise=True is an invariant you assert, not a hint. Split the macro shim from the logic. Keep polars expr functions tiny and delegate to plain functions e.g. Jinja's render ... that are unit-testable with cargo test and no Python runtime. Export the same Rust as plain pyfunctions for plan-time validation, so the code that checks a query is the code that runs it. When to reach for a plugin, and where to look next A plugin is the right tool when you need row-level work that Polars doesn't ship natively, you want it to run inside the engine parallel, streamable, composable , and you can express the output as a declared dtype. That covers a lot. Parsing, tokenizing, templating, fuzzy matching, format conversion, custom casts. When the work is genuinely not elementwise, or it's an opaque IO-bound call hitting a model API, say , the honest answer is still map batches or map elements , the Python escape hatches, and keeping the schema explicit there instead. But for the case where Polars can't do something natively and you want it in-engine and composable, nine plugins deep, the system holds up. It doesn't feel like bolting something onto Polars. It feels like extending it. If you want to see all nine in context, fenic is open source. The Rust kernels live in rust/src/ https://github.com/typedef-ai/fenic/tree/main/rust/src and the Python namespaces in . If you're writing your own, the https://github.com/typedef-ai/fenic/tree/main/src/fenic/ backends/local/polars plugins src/fenic/ backends/local/polars plugins/ Polars plugin docs https://docs.pola.rs/user-guide/plugins/expr plugins/ and are the place to start. And we'd genuinely like to hear how it goes. https://github.com/pola-rs/pyo3-polars pyo3-polars And if this post was worth your time, consider giving fenic a star on GitHub https://github.com/typedef-ai/fenic . It's the simplest way to help other people building on Polars find it, and it tells us these deep dives are worth writing.