{"slug": "how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python", "title": "How We Built a Robust EPUB Parsing and Rebuilding Pipeline in Python", "summary": "LectuLibre built a robust EPUB parsing and rebuilding pipeline in Python to translate entire books while preserving visual structure. The team used ebooklib for metadata handling and lxml for fast, namespace-aware XHTML manipulation, overcoming issues with broken markup, performance, and namespace chaos. The pipeline extracts translatable text segments, groups them into sentences for LLM translation, and reassembles the book with images, CSS, fonts, and layout intact.", "body_md": "*Dealing with broken markup, embedded fonts, and namespace chaos while building LectuLibre's translation engine*\n\nAt [LectuLibre](https://lectulibre.com), we needed to translate entire EPUB books while preserving their exact visual structure. The core challenge: parse the EPUB, extract all translatable text, send it to an LLM, then reassemble the book with the translated content—images, CSS, fonts, and layout untouched. This turned out to be much harder than it looked. Here’s how we solved it, what broke, and what we learned.\n\nAn EPUB is a ZIP archive containing XHTML, CSS, images, and a few XML control files (like `container.xml`\n\nand the OPF manifest). In theory, it’s a clean format. In practice, real‑world EPUBs are a mess:\n\n`Hello World`\n\n), requiring sentence‑aware translation.We needed a pipeline that could handle 90%+ of books without manual intervention, run fast enough for an interactive web service, and survive the most broken inputs we’d inevitably receive.\n\nWe reached for [ ebooklib](https://github.com/aerkalov/ebooklib)—a dedicated Python library for reading and writing EPUB files. It gives you a nice object model: an\n\n`EpubBook`\n\nwith items (documents, images, stylesheets), a spine, table of contents, and metadata. The code to open a book and grab all XHTML files looks deceptively simple:\n\n``` python\nimport ebooklib\nfrom ebooklib import epub\n\nbook = epub.read_epub('the-old-man-and-the-sea.epub')\nfor item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):\n content = item.get_content().decode('utf-8')\n # translate content ...\n item.set_content(translated_content.encode('utf-8'))\n\nepub.write_epub('translated.epub', book)\n```\n\nThis works for many clean EPUBs. But when we stress‑tested it with 100 public‑domain books, we quickly hit walls:\n\n`ebooklib`\n\nuses `xml.dom.minidom`\n\ninternally; reading a 20 MB book with many XHTML files took over 6 seconds, and writing it back took even longer. Memory usage would spike to 1 GB+ because the entire DOM was held in memory.``\n\n). `ebooklib`\n\n’s XML serialization would sometimes drop these namespaces, producing output that failed validation.Clearly, we needed something lower‑level for the actual content manipulation.\n\n`ebooklib`\n\nfor Metadata, `lxml`\n\nfor XHTML\nWe settled on a hybrid architecture:\n\n`ebooklib`\n\nto read and write the EPUB `lxml.etree`\n\n(which is fast, namespaces‑aware, and can recover from broken markup). We walk the tree, extract translatable text segments, translate them, and then inject the translations back into the tree.Here’s the core extraction logic:\n\n``` python\nfrom lxml import etree\n\ndef extract_translatable_blocks(html_bytes):\n parser = etree.HTMLParser(recover=True, encoding='utf-8')\n tree = etree.HTML(html_bytes, parser)\n # We only care about text that appears in the body.\n body = tree.find('.//body')\n if body is None:\n return []\n segments = []\n for element in body.iter():\n # Skip script, style, and void elements\n if element.tag in ('script', 'style', 'br', 'hr', 'img'):\n continue\n text = (element.text or '').strip()\n if text:\n segments.append((element, 'text', text))\n tail = (element.tail or '').strip()\n if tail:\n segments.append((element.getparent(), 'tail', tail))\n return segments\n```\n\nNotice we track both `element.text`\n\nand `element.tail`\n\n—this is critical because in HTML like `

Hello World

`\n\n, the word “World” is actually the `tail`\n\nof the ``\n\nelement.\n\nBefore translation, we group adjacent text segments into sentences. Our sentencizer (a lightweight regex‑based splitter) joins text across inline tags, so we send a single unit `\"Hello World\"`\n\nto the AI instead of two separate fragments. After translation, we split the result back across the original boundaries, taking care to preserve leading/trailing whitespace.\n\nOnce the tree is modified, we serialize it back with namespace preservation:\n\n``` python\ndef serialize_html(root_node):\n # lxml's etree.tostring handles namespaces correctly if you pass the tree with nsmap\n html_str = etree.tostring(\n root_node,\n method='html',\n encoding='unicode',\n xml_declaration=False,\n pretty_print=True\n )\n # Wrap back into a full XHTML document if needed\n return f'\\n\\n{html_str}'\n```\n\nWe then call `item.set_content(serialized_html.encode('utf-8'))`\n\non the `ebooklib`\n\nitem and write the book back out.\n\n`ebooklib`\n\nhandled images and fonts transparently as `ITEM_IMAGE`\n\nand `ITEM_OTHER`\n\n. We simply skip translation for non‑XHTML items. However, we discovered that some books rely on font‑face declarations in CSS that must remain valid after rebuild. We don’t modify CSS (translating `content: \"Chapter 1\"`\n\nwould be suicidal), but we do parse each CSS to check for font‑face `src`\n\nURLs and ensure they are preserved as relative paths in the rebuilt EPUB.\n\n`epubcheck`\n\nWe run every rebuilt EPUB through [epubcheck](https://github.com/w3c/epubcheck) (the official Java validator) as a final sanity check. Initially, 30% of our output files failed—mostly because `ebooklib`\n\nwould omit the `mimetype`\n\nfile entry at the beginning of the ZIP, or because we inadvertently stripped `xml:lang`\n\nattributes. We patched our write routine to always inject the mimetype file first, and we now preserve all XML namespaces and attributes during the lxml manipulation.\n\nProcessing a 500‑page novel end‑to‑end (parse, translate, rebuild) takes roughly:\n\nWe parallelise XHTML file processing with `asyncio`\n\n(`asyncio.to_thread`\n\nfor lxml work) because each chapter is independent. This brings wall‑clock time down to about 10 seconds for a typical book—acceptable for a real‑time web service.\n\nMemory usage stays stable at ~150–200 MB by avoiding loading huge DOMs simultaneously.\n\n`ebooklib`\n\nsaved us weeks of work on the ZIP container and manifest. But debugging why a book wouldn’t open on iBooks meant reading the EPUB 3 spec and checking the OPF line by line.``\n\ntags). `lxml`\n\n’s `recover=True`\n\nwas a lifesaver.`lxml`\n\n’s `nsmap`\n\nwhen parsing; never assume default namespace prefixes.`content`\n\n, `
`\n\nformatted blocks, and math should be left alone. We filter out elements based on a configurable allow‑list.We’re still not 100% satisfied with our pipeline. `ebooklib`\n\nis slow on large files due to its DOM‑based approach; rewriting the ZIP and OPF ourselves with `zipfile`\n\n+ `lxml`\n\ncould be faster, but it’s a lot of code. Are there other Python EPUB libraries that offer more granular control without the overhead? Would it make sense to fork `ebooklib`\n\nand swap out the XML backends for `lxml`\n\n? We’d love to hear your war stories—especially if you’ve built a similar translation or conversion pipeline.\n\n*This article was written by the LectuLibre engineering team. We’re building an AI‑powered book translation service—if you wrestle with EPUBs too, let’s talk!*", "url": "https://wpnews.pro/news/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python", "canonical_source": "https://dev.to/jacob_gong/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python-29f9", "published_at": "2026-06-24 03:02:01+00:00", "updated_at": "2026-06-24 03:13:40.277304+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "natural-language-processing", "ai-products"], "entities": ["LectuLibre", "ebooklib", "lxml", "Python"], "alternates": {"html": "https://wpnews.pro/news/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python", "markdown": "https://wpnews.pro/news/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python.md", "text": "https://wpnews.pro/news/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python.txt", "jsonld": "https://wpnews.pro/news/how-we-built-a-robust-epub-parsing-and-rebuilding-pipeline-in-python.jsonld"}}