{"slug": "scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating", "title": "Scaling Multilingual Search: Our FastAPI + PostgreSQL Architecture for Translating Books", "summary": "The LectuLibre team built a multilingual book translation platform on FastAPI and PostgreSQL, replacing slow LIKE-based text searches with PostgreSQL full-text search using per-language generated tsvector columns and GIN indexes. The team reported that common-word searches that previously took 300–500 ms across tens of thousands of books were addressed by language-specific text analysis, avoiding the operational overhead of Elasticsearch or Meilisearch on a single VPS.", "body_md": "*How we built a multilingual book platform with FastAPI and PostgreSQL's full-text search to handle translations in multiple languages.*\n\nWhen we started building LectuLibre, an AI-powered book translation service, we knew we'd have to handle content in many languages. Users upload EPUB or PDF books, our backend translates them using LLMs like Claude and DeepSeek, and readers can browse and search the translated versions. That meant our database needed to store and search text in dozens of languages efficiently. Here's how we built that with FastAPI and PostgreSQL, and what we learned along the way.\n\nEarly on, we stored translated text as plain strings in a PostgreSQL `TEXT` column. Searching was as simple as `WHERE content LIKE '%query%'`. That worked for a demo with a few hundred books. But as our library grew to tens of thousands of books across Spanish, French, German, Japanese, and more, `LIKE` queries became painfully slow. A simple search for a common word could take 300–500 ms. Not acceptable for an API that needs to feel snappy.\n\nWe also had a deeper issue: language-specific search behavior. Searching for \"café\" in Spanish should match \"cafe\" without the accent, and stemming rules differ per language. A one-size-fits-all approach wouldn't cut it.\n\nWe considered external search engines like Elasticsearch or Meilisearch. They're powerful but add operational complexity—another service to deploy, monitor, and keep in sync. We were a small team running on a single VPS. We needed something simpler.\n\nPostgreSQL's built-in full-text search (FTS) turned out to be the right fit. It supports language-specific text analysis, GIN indexes for speed, and it lives right next to our data. No sync jobs, no extra services.\n\nOur stack is Python/FastAPI on the backend, PostgreSQL for data, and SQLAlchemy 2.0 for ORM with asyncpg as the async driver. We use Alembic for migrations and Pydantic for validation. The API is fully async, which pairs nicely with asyncpg and FastAPI's async endpoints.\n\nFor multilingual search, we decided to:\n\n`translations` table, one row per language.`to_tsquery()` with the appropriate language configuration based on the user's query language.\nLet's dive into the details.\n\nHere's a simplified version of our SQLAlchemy models:\n\n``` python\nfrom sqlalchemy import Column, Integer, String, Text, ForeignKey, Index\nfrom sqlalchemy.dialects.postgresql import TSVECTOR\nfrom sqlalchemy.orm import declarative_base, relationship\n\nBase = declarative_base()\n\nclass Book(Base):\n    __tablename__ = \"books\"\n    id = Column(Integer, primary_key=True)\n    title = Column(String, nullable=False)\n    original_language = Column(String(10), nullable=False)\n    # ... other metadata\n    translations = relationship(\"Translation\", back_populates=\"book\")\n\nclass Translation(Base):\n    __tablename__ = \"translations\"\n    id = Column(Integer, primary_key=True)\n    book_id = Column(Integer, ForeignKey(\"books.id\"), nullable=False)\n    language = Column(String(10), nullable=False)  # 'en', 'es', 'fr', etc.\n    content = Column(Text, nullable=False)  # translated full text\n    # Generated tsvector columns for search\n    content_search_en = Column(TSVECTOR)\n    content_search_es = Column(TSVECTOR)\n    content_search_fr = Column(TSVECTOR)\n    # ... one per supported language\n    book = relationship(\"Book\", back_populates=\"translations\")\n```\n\nWe generate the tsvector columns automatically using a database trigger or a generated column. We chose **generated columns** (PostgreSQL 12+) because they're maintained by the database and don't require application logic.\n\nHere's how we create a generated column in a migration:\n\n``` python\nfrom alembic import op\nimport sqlalchemy as sa\nfrom sqlalchemy.dialects.postgresql import TSVECTOR\n\ndef upgrade():\n    op.add_column(\n        \"translations\",\n        sa.Column(\n            \"content_search_en\",\n            TSVECTOR,\n            sa.Computed(\n                \"to_tsvector('english', content)\",\n                persisted=True\n            )\n        )\n    )\n    op.create_index(\n        \"ix_translations_content_search_en\",\n        \"translations\",\n        [\"content_search_en\"],\n        postgresql_using=\"gin\"\n    )\n```\n\nWe do this for each language we support. The `persisted=True` makes it stored on disk, so updates to `content` automatically update the tsvector column. The GIN index then makes search lightning fast.\n\nOur search endpoint accepts a `q` parameter and an optional `lang` parameter. If `lang` is provided, we search only that language's tsvector column. If not, we search across all columns using a combined query.\n\nHere's a simplified FastAPI endpoint:\n\n``` python\nfrom fastapi import FastAPI, Query\nfrom sqlalchemy import select, or_\nfrom sqlalchemy.dialects.postgresql import to_tsquery\nfrom sqlalchemy.ext.asyncio import AsyncSession\n\napp = FastAPI()\n\n@app.get(\"/search\")\nasync def search_books(\n    q: str = Query(..., min_length=2),\n    lang: str = Query(None, description=\"Language code e.g. 'en', 'es'\"),\n    db: AsyncSession = Depends(get_db)\n):\n    query = select(Translation).join(Book)\n\n    if lang:\n        # Map language to its tsvector column and FTS configuration\n        tsvector_col = getattr(Translation, f\"content_search_{lang}\")\n        query = query.where(\n            tsvector_col.op(\"@@\")(to_tsquery(lang, q))\n        )\n    else:\n        # Search across all languages\n        conditions = []\n        for lang_code in SUPPORTED_LANGUAGES:\n            tsvector_col = getattr(Translation, f\"content_search_{lang_code}\")\n            conditions.append(\n                tsvector_col.op(\"@@\")(to_tsquery(lang_code, q))\n            )\n        query = query.where(or_(*conditions))\n\n    result = await db.execute(query.limit(50))\n    translations = result.scalars().all()\n    return translations\n```\n\nNote: `to_tsquery` is a SQLAlchemy function from `sqlalchemy.dialects.postgresql`. It accepts the language configuration name and the query string.\n\nFor accent-insensitive search, we use the `unaccent` extension. We create the tsvector columns using `to_tsvector(lang, unaccent(content))` and query with `to_tsquery(lang, unaccent(q))`. That way \"cafe\" matches \"café\".\n\nBefore optimization, a `LIKE` search on 100k rows took around 300–500 ms. After moving to FTS with GIN indexes, the same search returns in **10–20 ms**—a 20x improvement. Even with multiple language columns combined via `OR`, we stay under 50 ms for typical queries.\n\nWe currently support 12 languages, each with its own generated column and index. The storage overhead for tsvector columns is roughly 30–40% of the original text size, which is acceptable.\n\nWe chose PostgreSQL FTS because it eliminated a separate search service. For our scale (tens of thousands of books, maybe a few million rows eventually), it's more than sufficient. But we know its limits:\n\n`pg_trgm` can help with misspellings, but it's not as sophisticated as Elasticsearch's analyzers.`ts_rank` works, but tuning relevance for multilingual content is tricky.\nHaving a separate tsvector column per language works but feels clunky. Adding a new language requires a new column and index, which means a migration and possible table lock. For now, it's fine. An alternative would be to store a `tsvector` built with the `simple` configuration and do language-specific processing at query time, but that loses stemming. We decided the explicit columns are worth it.\n\nWe store the full translated text in the `content` column. For large books (e.g., 500k+ words), this bloats the database. We experimented with storing only the first N characters for search and keeping the full text in object storage, but then searching beyond the excerpt fails. For now, we keep everything in PostgreSQL. The VPS has enough disk.\n\nEnsure your database is UTF-8. We also set `lc_collate` to `en_US.UTF-8` for consistent sorting across languages. For accent-insensitive search, `unaccent` is a must.\n\nWe initially used `psycopg2` (sync) with FastAPI, but switched to `asyncpg` and SQLAlchemy's async support. It improved throughput under load. However, be careful with long-running transactions and connection pooling. We use `asyncpg`'s pool with `max_size=20` for our VPS.\n\nLectuLibre now handles search across 12 languages with sub-50ms response times for the API. Users can search in their own language and find books translated into that language, with proper stemming and accent-insensitive matching. The architecture is simple enough to maintain with a small team, and we avoided the operational overhead of Elasticsearch.\n\nOne challenge we haven't fully solved: cross-lingual search. If a user searches in English but wants results in Spanish, our current system can't match semantically. We're exploring options like multilingual embeddings or an external search index. How are others handling cross-lingual search in PostgreSQL? We'd love to hear your approaches.\n\n**Takeaway**: For a multilingual app at moderate scale, PostgreSQL's full-text search with generated tsvector columns is a pragmatic, low-maintenance choice. Pair it with FastAPI's async capabilities and you have a robust, fast search without extra infrastructure.", "url": "https://wpnews.pro/news/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating", "canonical_source": "https://dev.to/jacob_gong/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating-books-ijj", "published_at": "2026-10-24 03:01:07+00:00", "updated_at": "2026-10-24 03:10:52.610666+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure", "large-language-models"], "entities": ["LectuLibre", "FastAPI", "PostgreSQL", "SQLAlchemy", "asyncpg", "Alembic", "Claude", "DeepSeek"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating", "markdown": "https://wpnews.pro/news/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating.md", "text": "https://wpnews.pro/news/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating.txt", "jsonld": "https://wpnews.pro/news/scaling-multilingual-search-our-fastapi-postgresql-architecture-for-translating.jsonld"}}