Dealing with broken markup, embedded fonts, and namespace chaos while building LectuLibre's translation engine
At LectuLibre, we needed to translate entire EPUB books while preserving their exact visual structure. The core challenge: parse the EPUB, extract all translatable text, send it to an LLM, then reassemble the book with the translated content—images, CSS, fonts, and layout untouched. This turned out to be much harder than it looked. Here’s how we solved it, what broke, and what we learned.
An EPUB is a ZIP archive containing XHTML, CSS, images, and a few XML control files (like container.xml
and the OPF manifest). In theory, it’s a clean format. In practice, real‑world EPUBs are a mess:
<b>Hello</b> <i>World</i>
), requiring sentence‑aware translation.We needed a pipeline that could handle 90%+ of books without manual intervention, run fast enough for an interactive web service, and survive the most broken inputs we’d inevitably receive.
We reached for ebooklib—a dedicated Python library for reading and writing EPUB files. It gives you a nice object model: an
EpubBook
with items (documents, images, stylesheets), a spine, table of contents, and metadata. The code to open a book and grab all XHTML files looks deceptively simple:
import ebooklib
from ebooklib import epub
book = epub.read_epub('the-old-man-and-the-sea.epub')
for item in book.get_items_of_type(ebooklib.ITEM_DOCUMENT):
content = item.get_content().decode('utf-8')
item.set_content(translated_content.encode('utf-8'))
epub.write_epub('translated.epub', book)
This works for many clean EPUBs. But when we stress‑tested it with 100 public‑domain books, we quickly hit walls:
ebooklib
uses xml.dom.minidom
internally; reading a 20 MB book with many XHTML files took over 6 seconds, and writing it back took even longer. Memory usage would spike to 1 GB+ because the entire DOM was held in memory.<html xmlns="http://www.w3.org/1999/xhtml">
). ebooklib
’s XML serialization would sometimes drop these namespaces, producing output that failed validation.Clearly, we needed something lower‑level for the actual content manipulation.
ebooklib
for Metadata, lxml
for XHTML We settled on a hybrid architecture:
ebooklib
to read and write the EPUB lxml.etree
(which is fast, namespaces‑aware, and can recover from broken markup). We walk the tree, extract translatable text segments, translate them, and then inject the translations back into the tree.Here’s the core extraction logic:
from lxml import etree
def extract_translatable_blocks(html_bytes):
parser = etree.HTMLParser(recover=True, encoding='utf-8')
tree = etree.HTML(html_bytes, parser)
body = tree.find('.//body')
if body is None:
return []
segments = []
for element in body.iter():
if element.tag in ('script', 'style', 'br', 'hr', 'img'):
continue
text = (element.text or '').strip()
if text:
segments.append((element, 'text', text))
tail = (element.tail or '').strip()
if tail:
segments.append((element.getparent(), 'tail', tail))
return segments
Notice we track both element.text
and element.tail
—this is critical because in HTML like <p><b>Hello</b> <i>World</i></p>
, the word “World” is actually the tail
of the <b>
element.
Before translation, we group adjacent text segments into sentences. Our sentencizer (a lightweight regex‑based splitter) joins text across inline tags, so we send a single unit "Hello World"
to the AI instead of two separate fragments. After translation, we split the result back across the original boundaries, taking care to preserve leading/trailing whitespace.
Once the tree is modified, we serialize it back with namespace preservation:
def serialize_html(root_node):
html_str = etree.tostring(
root_node,
method='html',
encoding='unicode',
xml_declaration=False,
pretty_print=True
)
return f'<?xml version="1.0" encoding="utf-8"?>\n<!DOCTYPE html>\n{html_str}'
We then call item.set_content(serialized_html.encode('utf-8'))
on the ebooklib
item and write the book back out.
ebooklib
handled images and fonts transparently as ITEM_IMAGE
and ITEM_OTHER
. We simply skip translation for non‑XHTML items. However, we discovered that some books rely on font‑face declarations in CSS that must remain valid after rebuild. We don’t modify CSS (translating content: "Chapter 1"
would be suicidal), but we do parse each CSS to check for font‑face src
URLs and ensure they are preserved as relative paths in the rebuilt EPUB.
epubcheck
We run every rebuilt EPUB through epubcheck (the official Java validator) as a final sanity check. Initially, 30% of our output files failed—mostly because ebooklib
would omit the mimetype
file entry at the beginning of the ZIP, or because we inadvertently stripped xml:lang
attributes. We patched our write routine to always inject the mimetype file first, and we now preserve all XML namespaces and attributes during the lxml manipulation.
Processing a 500‑page novel end‑to‑end (parse, translate, rebuild) takes roughly:
We parallelise XHTML file processing with asyncio
(asyncio.to_thread
for lxml work) because each chapter is independent. This brings wall‑clock time down to about 10 seconds for a typical book—acceptable for a real‑time web service.
Memory usage stays stable at ~150–200 MB by avoiding huge DOMs simultaneously.
ebooklib
saved us weeks of work on the ZIP container and manifest. But debugging why a book wouldn’t open on iBooks meant reading the EPUB 3 spec and checking the OPF line by line.<body>
tags). lxml
’s recover=True
was a lifesaver.lxml
’s nsmap
when parsing; never assume default namespace prefixes.content
, <pre>
formatted blocks, and math should be left alone. We filter out elements based on a configurable allow‑list.We’re still not 100% satisfied with our pipeline. ebooklib
is slow on large files due to its DOM‑based approach; rewriting the ZIP and OPF ourselves with zipfile
lxml
could be faster, but it’s a lot of code. Are there other Python EPUB libraries that offer more granular control without the overhead? Would it make sense to fork ebooklib
and swap out the XML backends for lxml
? We’d love to hear your war stories—especially if you’ve built a similar translation or conversion pipeline.
This article was written by the LectuLibre engineering team. We’re building an AI‑powered book translation service—if you wrestle with EPUBs too, let’s talk!