A Scalable ML Framework with Monadic Design

Originally published on wisl.dev.

In the world of machine learning, going from research to production is often a painful, time-consuming process. As a developer and ML practitioner, I've personally felt this friction: juggling multiple libraries, inconsistent data formats, fragile pipelines, and the perpetual anxiety of things breaking in production.

To solve this, I built a highly scalable, Python-based machine learning framework that streamlines the entire ML lifecycle, from exploration to deployment, using monadic design principles to bring structure, composability, and reliability to the process.

Here's how it worked and what I learned.

A typical ML project might involve:

Each tool is great on its own, but stitching them together into a consistent, maintainable workflow? Not so much.

Worse, when it's time to deploy, you often end up rewriting large chunks of code, manually fixing bugs due to unexpected inputs, or patching over pipeline inconsistencies with brittle logic.

I set out to build a research-to-production machine learning framework with three goals in mind:

To achieve this, I drew inspiration from functional programming: specifically, monads.

In functional programming, a monad is a design pattern that wraps values with context (like logging, errors, or side effects) and allows transformations to be chained without losing that context.

In this ML framework, I designed a custom monadic pattern that revolves around two key entities:

1. DataPod: The State Carrier

The DataPod object acts as the context holder: it contains:

main, support_df, etc.) As the pipeline evolves, the DataPod flows from one transformer to the next, getting updated with new data or attributes while keeping the full research state intact.

2. Transformer: The Behavior Capsule

Each Transformer is a composable, stateful function that:

DataPod (e.g., scaling, encoding, feature engineering) This separation of data (in DataPod) and behavior (in Transformer) allows clean chaining of transformations, while also making the pipeline reproducible and deployable.

The flow looks like this: a DataPod (data + state) passes through a series of transformers, each of which learns something during fit and leaves its footprint behind. After the chain completes, the accumulated footprints list is what gets replayed in production.

Let's walk through how the monadic pattern works in this ML framework using simplified Python code.

This is the core monadic context. DataPod holds all the data and shared state passed from one transformer to the next.

class DataPod:
    def __init__(self, dfs):
        self.dfs = dfs  # Dictionary of dataframes (main, support, etc.)
        self.metadata = {}  # Optional: Store any global metadata or pipeline state
        self.footprints = []  # Track the sequence of transformers used

    def fit_transform(self, transformer):
        transformer = transformer.fit_transform(self)
        self.footprints.append(transformer)  # Record the transformer
        return self

Each transformer is a self-contained unit that holds any trained variables (e.g., mean, model) and knows how to transform a DataPod.

class TransformerA:
    def fit_transform(self, dp: DataPod):
        self.mean_val = dp.dfs["main"]["feature1"].mean()

        dp.metadata["mean_val"] = self.mean_val

        return self  # Important: Return self to store in footprints

    def transform(self, dp: DataPod):
        dp.dfs["main"]["feature1_scaled"] = dp.dfs["main"]["feature1"] / self.mean_val
        return dp

You create a DataPod with your raw data and apply transformations in a chainable, declarative way:

import pandas as pd

main_df = pd.DataFrame({"feature1": [100, 200, 300], "target": [1, 0, 1]})
support_df = pd.DataFrame({...})  # Optional

dfs = {"main": main_df, "support_df": support_df}
dp = DataPod(dfs=dfs)

dp = (
    dp.fit_transform(TransformerA())
    .fit_transform(TransformerB())
    .fit_transform(TransformerC())
)

print(dp.dfs["main"].head())
print(dp.metadata)

One of the powerful aspects of this design is the ability to easily compose and reuse sequences of transformations during the research phase. The Serializer class is essentially a convenient way to chain multiple transformers together into a single reusable pipeline, enabling you to apply all transformations in order without repeating code.

Here's the Serializer class that applies a list of transformers sequentially:

class Serializer:
    def __init__(self, transformers):
        self.transformers = transformers

    def transform(self, dp: DataPod):
        for transformer in self.transformers:
            dp = transformer.transform(dp)
        return dp

pipeline = Serializer(
    transformers=[
        TransformerA(),
        TransformerB(),
        TransformerC(),
    ]
)

dp = dp.fit_transform(pipeline)

One of the key benefits of this monadic design is that each Transformer stores its trained parameters internally (e.g., learned model weights, scaling factors). This means the entire pipeline can be reproduced exactly for deployment, ensuring consistency between research and production environments.

One detail worth calling out: the footprints list is the single artifact you ship. It holds the fitted transformers in application order, so research and production run byte-identical logic with no export/import step in between.

After training, your DataPod keeps a record of all applied transformers in dp.footprints. This list acts as a serialized artifact capturing the entire pipeline's state.

To deploy the pipeline on new production data, you simply:

Here's how it looks in code:

pipeline = dp.footprints  # List of trained Transformer instances

dp_prod = DataPod(dfs=data_prod)

dp_prod = dp_prod.transform(pipeline)

The framework grew to support a wide range of ML tasks out of the box:

It also included built-in error handling to catch and adapt to common production-time issues, like incompatible data types, schema mismatches, or missing fields, without halting execution.

Across our internal projects, the framework cut research-to-deployment time roughly in half and eliminated the "works in the notebook, breaks in production" class of incidents entirely.

What started as a developer's frustration turned into a powerful internal ML framework, unifying machine learning best practices with composable software design.

Using monads might seem abstract at first, but they offer real, pragmatic value in ML engineering: allowing you to build predictable, traceable, and extensible pipelines that scale from experiment to production without rework.

If you're tired of rebuilding pipelines for every use case or firefighting deployment issues, this architecture may be the shift you need.