{"slug": "robust-serialization-with-json-and-custom-types", "title": "Robust Serialization with JSON and Custom Types", "summary": "A tutorial demonstrates building a Python JSON serialization pipeline that handles custom types such as datetime, set, Path and Enum by subclassing json.JSONEncoder and overriding its default() method, then reconstructing those objects with a custom decoder and a factory function that applies defaults and schema migrations during deserialization. The article frames the problem around machine learning data plumbing, noting that json.dumps() on a sample experiment configuration raises \"TypeError: Object of type ModelType is not JSON serializable\" when it encounters an Enum, set, Path or datetime value.", "body_md": "In this article, you will learn how to build a robust JSON serialization pipeline in Python that handles custom data types, graceful error recovery, and schema migrations.\n\nTopics we will cover include:\n\n- How to define a custom JSON encoder to handle Python types like `datetime` ,`set` ,`Path` , and`Enum` .\n- How to write a custom decoder function that reconstructs those original Python objects from a serialized JSON string, including graceful handling of malformed data.\n- How to manage schema changes over time using a factory function that applies defaults and migrations during deserialization.\n\n## Introduction and Motivation\n\nAnyone fairly experienced in constructing **machine learning pipelines** might have noticed at some stage that only a small percentage of their time is dedicated to writing the model architecture. What do they spend the rest of that time on? **Data plumbing**: the process of cleaning, preprocessing, and transforming data to make it look perfectly digestible for downstream models.\n\nThis article tackles a hidden payoff often discovered along the machine learning engineering journey: the constant need to move structured datasets between files, processes, microservices, and APIs. Throughout these processes, you need a solution to **serialize and deserialize data** in a reliable and robust fashion.\n\nWe will explore some practical strategies for serializing custom types in a way that enables parsing them back securely, as well as managing data schema changes and dealing with malformed data — all without necessarily crashing the whole pipeline!\n\nBefore going deep into the three-step solution, let’s look at the root problem: JSON limitations when encountering certain data types like `datetime`, `set`, `Path`, or `Enum`. Suppose this is the configuration for a machine learning experiment:\n\n``` python\nfrom datetime import datetime\nfrom enum import Enum\nfrom pathlib import Path\nimport json\n\nclass ModelType(Enum):\n    RANDOM_FOREST = \"rf\"\n    NEURAL_NET = \"nn\"\n\nexperiment_config = {\n    \"experiment_id\": 1042,\n    \"model_type\": ModelType.NEURAL_NET,\n    \"features\": {\"age\", \"income\", \"location\"}, # A set!\n    \"data_path\": Path(\"/data/v2/train.csv\"),\n    \"timestamp\": datetime.now(),\n    \"nested_meta\": {\n        \"retries\": 3\n    }\n}\n\n12345678910111213141516171819\n\nfrom datetime import datetimefrom enum import Enumfrom pathlib import Pathimport json class ModelType(Enum):    RANDOM_FOREST = \"rf\"    NEURAL_NET = \"nn\" experiment_config = {    \"experiment_id\": 1042,    \"model_type\": ModelType.NEURAL_NET,    \"features\": {\"age\", \"income\", \"location\"}, # A set!    \"data_path\": Path(\"/data/v2/train.csv\"),    \"timestamp\": datetime.now(),    \"nested_meta\": {        \"retries\": 3    }}\n```\n\nThe moment you try to serialize this experiment configuration using `json.dumps(experiment_config)`, you will immediately come across an error like this:\n\n```\nTypeError: Object of type ModelType is not JSON serializable\n\n1\n\nTypeError: Object of type ModelType is not JSON serializable\n```\n\nThese kinds of errors are what we are about to fix by defining a robust pipeline for encoding and decoding JSON objects with custom types.\n\n## 1. Defining a Custom Encoder\n\nFirst, we will define a custom encoder class that inherits from `json.JSONEncoder`. This is crucial to be able to “customize” the existing `json.dumps()` with the extra logic we will implement in our class to handle custom data types that JSON cannot handle by default.\n\nBesides subclassing `json.JSONEncoder`, we also need to override the `default()` method. This is where, one by one, we will define the custom approach to handle each specific data type we might need to work with, using conditionals in the form `if isinstance(obj, DATA_TYPE):`, replacing `DATA_TYPE` with a specific data type or class name, as shown below:\n\n``` python\nclass RobustEncoder(json.JSONEncoder):\n    def default(self, obj):\n        # Handling Datetimes\n        if isinstance(obj, datetime):\n            return {\"__type__\": \"datetime\", \"value\": obj.isoformat()}\n        \n        # Handling Sets (convert to list for JSON)\n        if isinstance(obj, set):\n            return {\"__type__\": \"set\", \"value\": list(obj)}\n        \n        # Handling Paths (convert to string)\n        if isinstance(obj, Path):\n            return {\"__type__\": \"path\", \"value\": str(obj)}\n            \n        # Handling Enums (extract the underlying value)\n        if isinstance(obj, Enum):\n            return obj.value\n            \n        # Letting the base class handle everything else (or raise TypeError)\n        return super().default(obj)\n\n# Serializing our config object\njson_string = json.dumps(experiment_config, cls=RobustEncoder, indent=2)\nprint(json_string)\n\n123456789101112131415161718192021222324\n\nclass RobustEncoder(json.JSONEncoder):    def default(self, obj):        # Handling Datetimes        if isinstance(obj, datetime):            return {\"__type__\": \"datetime\", \"value\": obj.isoformat()}                # Handling Sets (convert to list for JSON)        if isinstance(obj, set):            return {\"__type__\": \"set\", \"value\": list(obj)}                # Handling Paths (convert to string)        if isinstance(obj, Path):            return {\"__type__\": \"path\", \"value\": str(obj)}                    # Handling Enums (extract the underlying value)        if isinstance(obj, Enum):            return obj.value                    # Letting the base class handle everything else (or raise TypeError)        return super().default(obj) # Serializing our config objectjson_string = json.dumps(experiment_config, cls=RobustEncoder, indent=2)print(json_string)\n```\n\nOne more important detail: the extra trick for robust serialization in the above code is attaching the `__type__` marker to objects that might lose their original typing when they are turned into JSON primitives. This occurs, for instance, when converting Python sets into lists.\n\nNotice how we use `cls=RobustEncoder` in this new call to `json.dumps()` to use our custom encoder.\n\nResult:\n\n```\n{\n  \"experiment_id\": 1042,\n  \"model_type\": \"nn\",\n  \"features\": {\n    \"__type__\": \"set\",\n    \"value\": [\n      \"income\",\n      \"location\",\n      \"age\"\n    ]\n  },\n  \"data_path\": {\n    \"__type__\": \"path\",\n    \"value\": \"/data/v2/train.csv\"\n  },\n  \"timestamp\": {\n    \"__type__\": \"datetime\",\n    \"value\": \"2026-09-28T14:26:10.328310\"\n  },\n  \"nested_meta\": {\n    \"retries\": 3\n  }\n}\n\n1234567891011121314151617181920212223\n\n{  \"experiment_id\": 1042,  \"model_type\": \"nn\",  \"features\": {    \"__type__\": \"set\",    \"value\": [      \"income\",      \"location\",      \"age\"    ]  },  \"data_path\": {    \"__type__\": \"path\",    \"value\": \"/data/v2/train.csv\"  },  \"timestamp\": {    \"__type__\": \"datetime\",    \"value\": \"2026-09-28T14:26:10.328310\"  },  \"nested_meta\": {    \"retries\": 3  }}\n```\n\nNo `TypeError` anymore!\n\n## 2. Defining a Custom Decoder\n\nFor the inverse process — obtaining Python objects back from a serialized JSON string — the key is using a different method and argument: `json.loads()` alongside an `object_hook` parameter.\n\nWe don’t need to define a class through inheritance this time, but we do create a custom function named `robust_decoder` which, when passed to `json.loads()`, will run on all nested dictionaries in the JSON string during parsing. The code includes the logic to reconstruct original Python objects, such as `Path`, as needed:\n\n``` python\nimport logging\n\ndef robust_decoder(dct):\n    # Checking if this dictionary represents one of our custom types\n    if \"__type__\" in dct and \"value\" in dct:\n        type_name = dct[\"__type__\"]\n        val = dct[\"value\"]\n        \n        try:\n            if type_name == \"datetime\":\n                return datetime.fromisoformat(val)\n            elif type_name == \"set\":\n                return set(val)\n            elif type_name == \"path\":\n                return Path(val)\n        except (ValueError, TypeError) as e:\n            # Handling Malformed Data: Logging the error without causing a crash.\n            # Return a safe fallback or a specialized error dictionary.\n            logging.warning(f\"Malformed {type_name} data detected: {e}. Falling back to raw value.\")\n            return val \n\n    return dct\n\n# Deserialize back to Python objects\nparsed_config = json.loads(json_string, object_hook=robust_decoder)\nprint(type(parsed_config[\"timestamp\"])) \nprint(type(parsed_config[\"features\"])) \n\n123456789101112131415161718192021222324252627\n\nimport logging def robust_decoder(dct):    # Checking if this dictionary represents one of our custom types    if \"__type__\" in dct and \"value\" in dct:        type_name = dct[\"__type__\"]        val = dct[\"value\"]                try:            if type_name == \"datetime\":                return datetime.fromisoformat(val)            elif type_name == \"set\":                return set(val)            elif type_name == \"path\":                return Path(val)        except (ValueError, TypeError) as e:            # Handling Malformed Data: Logging the error without causing a crash.            # Return a safe fallback or a specialized error dictionary.            logging.warning(f\"Malformed {type_name} data detected: {e}. Falling back to raw value.\")            return val      return dct # Deserialize back to Python objectsparsed_config = json.loads(json_string, object_hook=robust_decoder)print(type(parsed_config[\"timestamp\"])) print(type(parsed_config[\"features\"]))\n```\n\nThe function also includes exception handling in case malformed data is encountered.\n\nResult (output of the two print calls):\n\n```\n<class 'datetime.datetime'>\n<class 'set'>\n\n12\n\n<class 'datetime.datetime'><class 'set'>\n```\n\n## 3. Dealing with Schema Changes\n\nOne extra common challenge in machine learning projects is that model metadata schemas (model specifications or configuration, in simpler terms) change constantly. It is not rare to add new hyperparameters, deprecate old ones, or rename a metric. Loading old JSON config objects into newer versions of your code therefore often requires wrapping the deserialized dictionary in an additional validation layer.\n\nThe following factory function can help achieve this, aided by the robust decoder function we implemented earlier:\n\n``` php\ndef load_experiment_config(json_data: str) -> dict:\n    # 1. Deserializing with our robust hook\n    raw_dict = json.loads(json_data, object_hook=robust_decoder)\n    \n    # 2. Applying Schema Migrations / Defaults\n    # Our older configuration didn't have 'learning_rate', so we provide a safe default.\n    validated_config = {\n        \"experiment_id\": raw_dict.get(\"experiment_id\", \"unknown\"),\n        \"model_type\": raw_dict.get(\"model_type\", \"rf\"), \n        \"features\": raw_dict.get(\"features\", set()),\n        \"data_path\": raw_dict.get(\"data_path\", Path(\"./default_data.csv\")),\n        \"timestamp\": raw_dict.get(\"timestamp\", datetime.now()),\n        \"learning_rate\": raw_dict.get(\"learning_rate\", 0.001), # New schema hyperparameter\n        \"nested_meta\": raw_dict.get(\"nested_meta\", {})\n    }\n    \n    # 3. Discarding deprecated fields (e.g., if 'old_metric' was in raw_dict, optionally drop it here)\n    return validated_config\n\n123456789101112131415161718\n\ndef load_experiment_config(json_data: str) -> dict:    # 1. Deserializing with our robust hook    raw_dict = json.loads(json_data, object_hook=robust_decoder)        # 2. Applying Schema Migrations / Defaults    # Our older configuration didn't have 'learning_rate', so we provide a safe default.    validated_config = {        \"experiment_id\": raw_dict.get(\"experiment_id\", \"unknown\"),        \"model_type\": raw_dict.get(\"model_type\", \"rf\"),         \"features\": raw_dict.get(\"features\", set()),        \"data_path\": raw_dict.get(\"data_path\", Path(\"./default_data.csv\")),        \"timestamp\": raw_dict.get(\"timestamp\", datetime.now()),        \"learning_rate\": raw_dict.get(\"learning_rate\", 0.001), # New schema hyperparameter        \"nested_meta\": raw_dict.get(\"nested_meta\", {})    }        # 3. Discarding deprecated fields (e.g., if 'old_metric' was in raw_dict, optionally drop it here)    return validated_config\n```\n\nWe explicitly added the defaulting mechanism for a `\"learning_rate\"` hyperparameter that didn’t exist in the original schema, but you can follow a similar approach for any other new hyperparameter that should be present in your updated model specifications.\n\nLet’s look at how this enhanced strategy works by passing the original JSON string through the full pipeline:\n\n```\nprint(\"Raw JSON being loaded:\")\nprint(json_string)\nprint(\"\\n\" + \"=\"*40 + \"\\n\")\n\n# Processing the raw, original JSON object through our robust loading and validation pipeline\nfinal_config = load_experiment_config(json_string)\n\nprint(\"--- Validated Python Configuration ---\")\nfor key, value in final_config.items():\n    # Printing the key, the reconstructed value, and its underlying Python type\n    print(f\"{key}: {value} (Type: {type(value).__name__})\")\n\n1234567891011\n\nprint(\"Raw JSON being loaded:\")print(json_string)print(\"\\n\" + \"=\"*40 + \"\\n\") # Processing the raw, original JSON object through our robust loading and validation pipelinefinal_config = load_experiment_config(json_string) print(\"--- Validated Python Configuration ---\")for key, value in final_config.items():    # Printing the key, the reconstructed value, and its underlying Python type    print(f\"{key}: {value} (Type: {type(value).__name__})\")\n```\n\nResults:\n\n```\nRaw JSON being loaded:\n{\n  \"experiment_id\": 1042,\n  \"model_type\": \"nn\",\n  \"features\": {\n    \"__type__\": \"set\",\n    \"value\": [\n      \"income\",\n      \"location\",\n      \"age\"\n    ]\n  },\n  \"data_path\": {\n    \"__type__\": \"path\",\n    \"value\": \"/data/v2/train.csv\"\n  },\n  \"timestamp\": {\n    \"__type__\": \"datetime\",\n    \"value\": \"2026-09-28T14:26:10.328310\"\n  },\n  \"nested_meta\": {\n    \"retries\": 3\n  }\n}\n\n========================================\n\n--- Validated Python Configuration ---\nexperiment_id: 1042 (Type: int)\nmodel_type: nn (Type: str)\nfeatures: {'income', 'location', 'age'} (Type: set)\ndata_path: /data/v2/train.csv (Type: PosixPath)\ntimestamp: 2026-09-28 14:26:10.328310 (Type: datetime)\nlearning_rate: 0.001 (Type: float)\nnested_meta: {'retries': 3} (Type: dict)\n\n1234567891011121314151617181920212223242526272829303132333435\n\nRaw JSON being loaded:{  \"experiment_id\": 1042,  \"model_type\": \"nn\",  \"features\": {    \"__type__\": \"set\",    \"value\": [      \"income\",      \"location\",      \"age\"    ]  },  \"data_path\": {    \"__type__\": \"path\",    \"value\": \"/data/v2/train.csv\"  },  \"timestamp\": {    \"__type__\": \"datetime\",    \"value\": \"2026-09-28T14:26:10.328310\"  },  \"nested_meta\": {    \"retries\": 3  }} ======================================== --- Validated Python Configuration ---experiment_id: 1042 (Type: int)model_type: nn (Type: str)features: {'income', 'location', 'age'} (Type: set)data_path: /data/v2/train.csv (Type: PosixPath)timestamp: 2026-09-28 14:26:10.328310 (Type: datetime)learning_rate: 0.001 (Type: float)nested_meta: {'retries': 3} (Type: dict)\n```\n\n## Wrapping Up\n\nThrough three practical steps, we have learned how to take full control of Python’s tools for serializing and deserializing JSON objects. These steps can be combined to form a robust serialization pipeline that handles custom data types which would otherwise raise errors. Not only can we handle nested JSON objects through the classes and functions we defined, but they are also designed to fail gracefully when malformed data is encountered. These techniques can meaningfully sharpen your data plumbing skills for machine learning engineering.", "url": "https://wpnews.pro/news/robust-serialization-with-json-and-custom-types", "canonical_source": "https://machinelearningmastery.com/robust-serialization-with-json-and-custom-types/", "published_at": "2026-10-26 11:00:00+00:00", "updated_at": "2026-10-26 13:12:26.246406+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["Python", "json", "ModelType", "RobustEncoder"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/robust-serialization-with-json-and-custom-types", "markdown": "https://wpnews.pro/news/robust-serialization-with-json-and-custom-types.md", "text": "https://wpnews.pro/news/robust-serialization-with-json-and-custom-types.txt", "jsonld": "https://wpnews.pro/news/robust-serialization-with-json-and-custom-types.jsonld"}}