Scoring Documents Against a Content Model Without an LLM

A developer building a Google Docs to Contentful import pipeline replaced the naive approach of sending entire documents to an LLM with a deterministic pre-filtering stage that scores document scopes against content types using Jaccard similarity. The method narrows forty content types to a shortlist of candidates before any LLM call, reducing token costs and improving accuracy.

The naive way to map a Google Doc into a CMS is to send the whole document to an LLM and ask it to figure out which sections go where. That works at small scale, with a small content model. It stops working when your content model has forty content types, each with a dozen fields, and each document has a different structure. You end up burning a large context window on mostly-irrelevant content types, paying for tokens that don't help, and getting worse results because the model is reasoning about the full space of possibilities instead of a curated shortlist. The pipeline I was working on a Google Docs → Contentful import addressed this with a deterministic pre-filtering stage that runs before any LLM call. Its job is not to do the mapping. Its job is to narrow the problem: segment the document into contiguous regions, score each region against every content type, and hand the LLM a small set of scope, candidate content types pairs instead of a full Cartesian product. This post is about how that scoring works and why the design choices look the way they do. The document is treated as a sequence of blocks. Any block that is a tab opener, an H1, or an H2 starts a new scope. Every block between two openers belongs to the scope opened by the first one. The result is a flat array of DocumentScope objects: export type ScopeReasonCode = 'tab-label' | 'heading-match' | 'body-term-match' export type ContentTypeMatch = { contentTypeId: string score: number reasons: ScopeReasonCode } export type DocumentScope = { openerText: string | null openerKind: 'tab' | 'heading' | 'document-root' blocks: NormalizedBlock matches: ContentTypeMatch topMatch: ContentTypeMatch | null needsReview: boolean } openerText is the raw text of the tab label or heading that opened the scope. blocks is the contiguous slice of the document. matches is every content type scored against this scope in descending score order. topMatch is a convenience pointer to the first match - null if there were no matches above the noise floor. needsReview is a flag, not a gate. The function signature is straightforward: export function segmentDocumentScopes doc: NormalizedDocument, slice: ContentModelOntologySlice, : DocumentScope The ContentModelOntologySlice is a pre-built graph of content type nodes, field nodes, and the intentTerms sets derived from each; more on those below. The score for a scope, content type pair is Jaccard similarity over tokenized text: function jaccardSimilarity a: Set<string , b: Set<string : number { if a.size === 0 && b.size === 0 return 0 let intersection = 0 for const term of a { if b.has term intersection++ } const union = a.size + b.size - intersection return union === 0 ? 0 : intersection / union } function tokenize text: string : Set<string { return new Set text .toLowerCase .replace / ^a-z0-9\s /g, ' ' .split /\s+/ .filter t = t.length 1 , } |A ∩ B| / |A ∪ B| . That is the entire base score. No TF-IDF weights, no vector embeddings, no pre-computed corpus statistics. The score for a scope with { "product", "info", "title" } against a content type with intent terms { "product", "title", "description" } is 2 / 4 = 0.5 . The reason for Jaccard is not that it produces better scores than cosine similarity over TF-IDF vectors. It almost certainly does not. The reason is that this function runs synchronously on every request, before any LLM call. Embeddings require a model call. Cosine similarity over TF-IDF requires pre-computing corpus statistics from the full set of documents, which is not available at request time. Jaccard over a simple tokenized set is O |A| + |B| , deterministic, reproducible, and has no external dependencies. It does not need to be perfect; it needs to be good enough to narrow forty content types to three, fast enough that it adds no measurable latency, and auditable enough that when it gets something wrong, you can read the code and understand exactly why. A tab label that reads "Product Info" is a categorically stronger signal than a body paragraph that mentions "product" once. The base Jaccard score treats every term equally regardless of where it appears. The bonus layer addresses this: function computeScore scope: RawScope, ctNode: ContentTypeNode, : { score: number; reasons: ScopeReasonCode } { const intentTerms = ctNode.intentTerms const bodyTerms = tokenize scope.blocks.map b = b.text .join ' ' const baseScore = jaccardSimilarity bodyTerms, intentTerms const reasons: ScopeReasonCode = let bonus = 0 if scope.openerKind === 'tab' && scope.openerText { const tabTerms = tokenize scope.openerText const tabOverlap = jaccardSimilarity tabTerms, intentTerms if tabOverlap 0 { bonus += 0.2 tabOverlap reasons.push 'tab-label' } } if scope.openerKind === 'heading' && scope.openerText { const headingTerms = tokenize scope.openerText const headingOverlap = jaccardSimilarity headingTerms, intentTerms if headingOverlap 0 { bonus += 0.1 headingOverlap reasons.push 'heading-match' } } if baseScore 0 { reasons.push 'body-term-match' } return { score: baseScore + bonus, reasons } } The bonus is a multiplier on the overlap in the structural position, not a flat constant. A tab label that shares half its terms with the content type's intent terms contributes 0.2 × 0.5 = 0.1 to the score. A tab label that fully matches contributes 0.2 . The multiplier means the bonus scales with how strongly the structural position matched, not just whether it matched at all. A document that opens a tab called "Product Info" against a content type called "Product" with intent terms derived from that name gets a materially higher score than a document that mentions "product" once in a body paragraph with no structural indicator. The three reason codes: tab-label , heading-match , body-term-match ; exist for observability, not routing. When the LLM receives a scope with its top matches, the reason codes tell it and any downstream monitoring what evidence the scorer used. A match driven purely by body-term-match with a low score deserves more skepticism from the LLM than one driven by tab-label with a high score. Making the evidence explicit, not just the verdict, gives the downstream consumer something to work with. The ContentModelOntologySlice is built deterministically from the raw content type definitions before any document is processed: export function buildContentModelOntologySlice contentTypes: ContentType , : ContentModelOntologySlice { const ctNodes: ContentTypeNode = contentTypes.map ct = { const terms = new Set<string // CT-level signals for const t of tokenize ct.sys.id terms.add t for const t of tokenize ct.name terms.add t if ct.description for const t of tokenize ct.description terms.add t // Field-level signals for const field of ct.fields { for const t of tokenize field.id terms.add t for const t of tokenize field.name terms.add t if field.type === 'Link' && field.validations { for const v of field.validations { if v.linkContentType { for const linkedCtId of v.linkContentType { for const t of tokenize linkedCtId terms.add t } } } } // helpText, validation summaries, etc. } return { contentTypeId: ct.sys.id, intentTerms: terms, fields: } } return { ctNodes, fieldNodes: , edges: } } The key decision here is tokenizing field IDs, content type IDs, and validation rules, not just display names and descriptions. A content type named ctProductVariant with a field named linkedEntries that has a linkContentType: "BlogPost" validation produces intent terms including ctproductvariant , linkedentries , and blogpost . A content type with a sparse or technical name pgBlogEntry still has vocabulary the scorer can match against; it isn't limited to whatever its display name happens to be. This matters because many production content models have programmatic IDs that carry semantic information. A content type called "Page: Blog Entry" in the display name but pgBlogEntry in the API name will produce terms for both. The validation relationships linkContentType , allowedContentTypes bring in referenced content type IDs, which means a scope that references "blog posts" in passing can score against a container content type that links to BlogPost , even if the container's own name doesn't mention it. After scoring, three conditions set needsReview: true on a scope: function computeNeedsReview scope: RawScope, matches: ContentTypeMatch : boolean { // No content blocks — nothing to score against if scope.blocks.length === 0 return true const top = matches 0 const second = matches 1 // Weak top score — no content type matched confidently if top || top.score < 0.1 return true // Ambiguous — top two are nearly tied if second && top.score - second.score < 0.05 return true return false } None of these conditions throw an error or halt the pipeline. The pipeline continues with all scopes, flagged or not. The needsReview flag is a signal passed downstream to the LLM and to monitoring , not a gate. The reason for this design is that hard-failing on low scores would block legitimate edge cases: a document with no headings produces a single large scope that scores diffusely across many content types; a content type with sparse intent terms produces low scores even against obviously relevant documents. These are not pipeline errors; they are cases where the deterministic scorer ran out of signal. The right behavior is to continue and let the LLM handle those scopes with the full context of knowing the scorer wasn't confident, not to abort the import and force a manual retry. The top-2 gap check deserves its own explanation. A score of 0.15 vs. 0.14 is not a decision, it is noise in a simple set-overlap function. Rather than let a near-tie pass through as a confident first-place assignment, the gap check flags it explicitly. The LLM sees two near-equal candidates and needsReview: true , which is accurate: the scorer does not have enough signal to pick a winner, and the LLM should treat both candidates seriously rather than assuming the top-ranked one is correct. The specific implementation here, Jaccard similarity, structural bonuses, needsReview flags, is tied to this pipeline. The underlying decisions are not: needsReview as a first-class output is better than crashing or silently passing through low-confidence decisions.The LLM in this pipeline gets a scoped problem instead of a full document-by-content-model matrix. That is the only job of this stage. Getting it right deterministically means the expensive, slow, creative part of the pipeline; the LLM spends its context on decisions it actually needs to make.