Building a High-Throughput Article-to-Markdown API for LLM Ingestion with FastAPI and Playwright

A developer designed a self-hosted article-to-Markdown extraction microservice using FastAPI, Trafilatura, Readability-lxml, and an asynchronous Playwright fallback to convert raw web pages into clean Markdown for LLM ingestion. The tiered extraction waterfall lets roughly 85% of standard web content pass through the lightweight Python tier without launching Chromium, keeping memory usage stable and infrastructure costs near zero. The service caches results in Redis and returns Markdown with parsed page metadata.

Feeding raw HTML into LLM context windows is one of the most expensive and inefficient mistakes in modern AI engineering. A standard modern news or blog page easily spans 1.5MB to 4MB of raw DOM payload. When passed straight into an LLM or vector database, 90% of those tokens are spent on tracking scripts, serialized JSON-LD blobs, cookie banners, navigation menus, and inline CSS styles. This not only causes severe context bloat and escalates inference bills, but it also degrades retrieval-augmented generation RAG semantic search precision by polluting your vector space with boilerplate noise. Here is how to design and build an enterprise-grade, self-hosted extraction microservice using FastAPI, Trafilatura, Readability, and an asynchronous Playwright fallback for SPA rendering. Most teams start with simple libraries like BeautifulSoup or newspaper3k . These quickly break down: <div id="root" </div shells to standard HTTP clients. To balance latency, compute cost, and reliability, the optimal architecture uses a tiered extraction waterfall : trafilatura . Latency: ~150-300ms. readability-lxml algorithm. Incoming Request: URL │ ▼ ┌───────────────────┐ │ Redis Cache Check │ ── Hit ──► Return Markdown & Metadata └───────────────────┘ │ Miss ▼ ┌───────────────────┐ │ HTTPX Async Fetch │ └───────────────────┘ │ Static HTML │ ▼ ┌───────────────────┐ │ Trafilatura │ ── Success: Length Threshold ──► Parse Meta & Return └───────────────────┘ │ Failed / Empty ▼ ┌───────────────────┐ │ Readability-lxml │ ── Success: Length Threshold ──► Parse Meta & Return └───────────────────┘ │ Failed / SPA Detected ▼ ┌───────────────────┐ │ Playwright Worker │ ── Render DOM ──► Re-extract via Trafilatura └───────────────────┘ │ ▼ Store in Cache & Return Markdown This setup allows 85% of standard web content to pass through the lightweight Python tier without launching Chromium, keeping memory usage stable and infrastructure costs near zero. Below is the complete implementation of the dual-engine pipeline using FastAPI, Pydantic v2, and async execution. python from pydantic import BaseModel, HttpUrl, Field from typing import Optional, Dict, Any from datetime import datetime class ExtractionRequest BaseModel : url: HttpUrl force playwright: bool = Field default=False, description="Bypass fast path and force browser rendering" max chars: Optional int = Field default=None, description="Truncate body content for token limits" class PageMetadata BaseModel : title: Optional str = None author: Optional str = None published date: Optional str = None site name: Optional str = None reading time minutes: int = 0 opengraph: Dict str, Any = {} class ExtractionResponse BaseModel : url: str markdown: str metadata: PageMetadata engine used: str execution time ms: float python import time import httpx import trafilatura from readability import Document from bs4 import BeautifulSoup from markdownify import markdownify as md from playwright.async api import async playwright USER AGENT = "Mozilla/5.0 Windows NT 10.0; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/122.0.0.0 Safari/537.36" async def fetch html fast url: str - str: async with httpx.AsyncClient timeout=10.0, follow redirects=True as client: response = await client.get url, headers={"User-Agent": USER AGENT} response.raise for status return response.text async def fetch html playwright url: str - str: async with async playwright as p: browser = await p.chromium.launch headless=True, args= "--no-sandbox", "--disable-dev-shm-usage" context = await browser.new context user agent=USER AGENT page = await context.new page await page.goto url, wait until="networkidle", timeout=20000 content = await page.content await browser.close return content def extract metadata html: str - PageMetadata: soup = BeautifulSoup html, "lxml" og data = {} for tag in soup.find all "meta" : prop = tag.get "property", tag.get "name", "" if prop.startswith "og:" or prop.startswith "twitter:" : og data prop = tag.get "content", "" title = og data.get "og:title" or soup.title.string if soup.title else None author = og data.get "article:author" or og data.get "twitter:creator" pub date = og data.get "article:published time" return PageMetadata title=title, author=author, published date=pub date, site name=og data.get "og:site name" , opengraph=og data def extract content html: str, url: str - tuple str, str : Primary Engine: Trafilatura extracted = trafilatura.extract html, url=url, output format="markdown", include links=True, include images=False, favor recall=False if extracted and len extracted.strip 200: return extracted, "trafilatura" Fallback Engine: Readability + Markdownify doc = Document html summary html = doc.summary markdown output = md summary html, heading style="ATX" .strip if len markdown output 100: return markdown output, "readability" return "", "none" python from fastapi import FastAPI, HTTPException, status app = FastAPI title="Article-to-Markdown Extraction API", version="1.0.0" @app.post "/api/v1/extract", response model=ExtractionResponse async def extract article payload: ExtractionRequest : start time = time.perf counter url str = str payload.url engine used = "trafilatura" try: if payload.force playwright: html = await fetch html playwright url str markdown, engine used = extract content html, url str engine used = f"playwright+{engine used}" else: Tier 1 & 2: Fast HTTP path try: html = await fetch html fast url str markdown, engine used = extract content html, url str except Exception: markdown = "" Tier 3: SPA / Fallback if empty if not markdown or len markdown.strip < 150: html = await fetch html playwright url str markdown, engine used = extract content html, url str engine used = f"playwright fallback+{engine used}" if not markdown: raise HTTPException status code=status.HTTP 422 UNPROCESSABLE ENTITY, detail="Failed to extract meaningful content from the target URL." metadata = extract metadata html words = len markdown.split metadata.reading time minutes = max 1, round words / 200 if payload.max chars and len markdown payload.max chars: markdown = markdown :payload.max chars + "\n\n Content Truncated " exec duration = time.perf counter - start time 1000 return ExtractionResponse url=url str, markdown=markdown, metadata=metadata, engine used=engine used, execution time ms=round exec duration, 2 except HTTPException: raise except Exception as e: raise HTTPException status code=status.HTTP 500 INTERNAL SERVER ERROR, detail=str e When exposing this service in production pipelines e.g., connecting n8n webhooks or feeding continuous Celery queues , consider these three safeguards: Launching a new Chromium instance via async playwright on every request will instantly lead to Linux OOM crashes. In a production build: BrowserContext sessions per request. --disable-dev-shm-usage and --no-sandbox flags inside your Docker container. Articles rarely mutate hourly. Adding an upstream Redis cache using a SHA-256 hash of the normalized canonical URL drastically drops response times to sub-10ms for repeated requests: php import hashlib def generate cache key url: str - str: normalized = url.split "?" 0 .rstrip "/" .lower return f"extract:{hashlib.sha256 normalized.encode .hexdigest }" FROM python:3.11-slim ENV PYTHONUNBUFFERED=1 \ PIP NO CACHE DIR=1 \ PLAYWRIGHT BROWSERS PATH=/ms-playwright WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends \ libxml2-dev libxslt-dev gcc libc-dev curl \ && rm -rf /var/lib/apt/lists/ COPY requirements.txt . RUN pip install -r requirements.txt RUN playwright install --with-deps chromium COPY . . EXPOSE 8000 CMD "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2" You can manually wire this FastAPI microservice into your infrastructure following the architectural design and code above. If you want the complete, production-ready implementation out of the box—including: You can grab the complete source repository directly: EARLYBIRD