Robust Serialization with JSON and Custom Types A tutorial demonstrates building a Python JSON serialization pipeline that handles custom types such as datetime, set, Path and Enum by subclassing json.JSONEncoder and overriding its default() method, then reconstructing those objects with a custom decoder and a factory function that applies defaults and schema migrations during deserialization. The article frames the problem around machine learning data plumbing, noting that json.dumps() on a sample experiment configuration raises "TypeError: Object of type ModelType is not JSON serializable" when it encounters an Enum, set, Path or datetime value. In this article, you will learn how to build a robust JSON serialization pipeline in Python that handles custom data types, graceful error recovery, and schema migrations. Topics we will cover include: - How to define a custom JSON encoder to handle Python types like datetime , set , Path , and Enum . - How to write a custom decoder function that reconstructs those original Python objects from a serialized JSON string, including graceful handling of malformed data. - How to manage schema changes over time using a factory function that applies defaults and migrations during deserialization. Introduction and Motivation Anyone fairly experienced in constructing machine learning pipelines might have noticed at some stage that only a small percentage of their time is dedicated to writing the model architecture. What do they spend the rest of that time on? Data plumbing : the process of cleaning, preprocessing, and transforming data to make it look perfectly digestible for downstream models. This article tackles a hidden payoff often discovered along the machine learning engineering journey: the constant need to move structured datasets between files, processes, microservices, and APIs. Throughout these processes, you need a solution to serialize and deserialize data in a reliable and robust fashion. We will explore some practical strategies for serializing custom types in a way that enables parsing them back securely, as well as managing data schema changes and dealing with malformed data — all without necessarily crashing the whole pipeline Before going deep into the three-step solution, let’s look at the root problem: JSON limitations when encountering certain data types like datetime , set , Path , or Enum . Suppose this is the configuration for a machine learning experiment: python from datetime import datetime from enum import Enum from pathlib import Path import json class ModelType Enum : RANDOM FOREST = "rf" NEURAL NET = "nn" experiment config = { "experiment id": 1042, "model type": ModelType.NEURAL NET, "features": {"age", "income", "location"}, A set "data path": Path "/data/v2/train.csv" , "timestamp": datetime.now , "nested meta": { "retries": 3 } } 12345678910111213141516171819 from datetime import datetimefrom enum import Enumfrom pathlib import Pathimport json class ModelType Enum : RANDOM FOREST = "rf" NEURAL NET = "nn" experiment config = { "experiment id": 1042, "model type": ModelType.NEURAL NET, "features": {"age", "income", "location"}, A set "data path": Path "/data/v2/train.csv" , "timestamp": datetime.now , "nested meta": { "retries": 3 }} The moment you try to serialize this experiment configuration using json.dumps experiment config , you will immediately come across an error like this: TypeError: Object of type ModelType is not JSON serializable 1 TypeError: Object of type ModelType is not JSON serializable These kinds of errors are what we are about to fix by defining a robust pipeline for encoding and decoding JSON objects with custom types. 1. Defining a Custom Encoder First, we will define a custom encoder class that inherits from json.JSONEncoder . This is crucial to be able to “customize” the existing json.dumps with the extra logic we will implement in our class to handle custom data types that JSON cannot handle by default. Besides subclassing json.JSONEncoder , we also need to override the default method. This is where, one by one, we will define the custom approach to handle each specific data type we might need to work with, using conditionals in the form if isinstance obj, DATA TYPE : , replacing DATA TYPE with a specific data type or class name, as shown below: python class RobustEncoder json.JSONEncoder : def default self, obj : Handling Datetimes if isinstance obj, datetime : return {" type ": "datetime", "value": obj.isoformat } Handling Sets convert to list for JSON if isinstance obj, set : return {" type ": "set", "value": list obj } Handling Paths convert to string if isinstance obj, Path : return {" type ": "path", "value": str obj } Handling Enums extract the underlying value if isinstance obj, Enum : return obj.value Letting the base class handle everything else or raise TypeError return super .default obj Serializing our config object json string = json.dumps experiment config, cls=RobustEncoder, indent=2 print json string 123456789101112131415161718192021222324 class RobustEncoder json.JSONEncoder : def default self, obj : Handling Datetimes if isinstance obj, datetime : return {" type ": "datetime", "value": obj.isoformat } Handling Sets convert to list for JSON if isinstance obj, set : return {" type ": "set", "value": list obj } Handling Paths convert to string if isinstance obj, Path : return {" type ": "path", "value": str obj } Handling Enums extract the underlying value if isinstance obj, Enum : return obj.value Letting the base class handle everything else or raise TypeError return super .default obj Serializing our config objectjson string = json.dumps experiment config, cls=RobustEncoder, indent=2 print json string One more important detail: the extra trick for robust serialization in the above code is attaching the type marker to objects that might lose their original typing when they are turned into JSON primitives. This occurs, for instance, when converting Python sets into lists. Notice how we use cls=RobustEncoder in this new call to json.dumps to use our custom encoder. Result: { "experiment id": 1042, "model type": "nn", "features": { " type ": "set", "value": "income", "location", "age" }, "data path": { " type ": "path", "value": "/data/v2/train.csv" }, "timestamp": { " type ": "datetime", "value": "2026-09-28T14:26:10.328310" }, "nested meta": { "retries": 3 } } 1234567891011121314151617181920212223 { "experiment id": 1042, "model type": "nn", "features": { " type ": "set", "value": "income", "location", "age" }, "data path": { " type ": "path", "value": "/data/v2/train.csv" }, "timestamp": { " type ": "datetime", "value": "2026-09-28T14:26:10.328310" }, "nested meta": { "retries": 3 }} No TypeError anymore 2. Defining a Custom Decoder For the inverse process — obtaining Python objects back from a serialized JSON string — the key is using a different method and argument: json.loads alongside an object hook parameter. We don’t need to define a class through inheritance this time, but we do create a custom function named robust decoder which, when passed to json.loads , will run on all nested dictionaries in the JSON string during parsing. The code includes the logic to reconstruct original Python objects, such as Path , as needed: python import logging def robust decoder dct : Checking if this dictionary represents one of our custom types if " type " in dct and "value" in dct: type name = dct " type " val = dct "value" try: if type name == "datetime": return datetime.fromisoformat val elif type name == "set": return set val elif type name == "path": return Path val except ValueError, TypeError as e: Handling Malformed Data: Logging the error without causing a crash. Return a safe fallback or a specialized error dictionary. logging.warning f"Malformed {type name} data detected: {e}. Falling back to raw value." return val return dct Deserialize back to Python objects parsed config = json.loads json string, object hook=robust decoder print type parsed config "timestamp" print type parsed config "features" 123456789101112131415161718192021222324252627 import logging def robust decoder dct : Checking if this dictionary represents one of our custom types if " type " in dct and "value" in dct: type name = dct " type " val = dct "value" try: if type name == "datetime": return datetime.fromisoformat val elif type name == "set": return set val elif type name == "path": return Path val except ValueError, TypeError as e: Handling Malformed Data: Logging the error without causing a crash. Return a safe fallback or a specialized error dictionary. logging.warning f"Malformed {type name} data detected: {e}. Falling back to raw value." return val return dct Deserialize back to Python objectsparsed config = json.loads json string, object hook=robust decoder print type parsed config "timestamp" print type parsed config "features" The function also includes exception handling in case malformed data is encountered. Result output of the two print calls :