{"slug": "scoring-documents-against-a-content-model-without-an-llm", "title": "Scoring Documents Against a Content Model Without an LLM", "summary": "A developer building a Google Docs to Contentful import pipeline replaced the naive approach of sending entire documents to an LLM with a deterministic pre-filtering stage that scores document scopes against content types using Jaccard similarity. The method narrows forty content types to a shortlist of candidates before any LLM call, reducing token costs and improving accuracy.", "body_md": "The naive way to map a Google Doc into a CMS is to send the whole document to an LLM and ask it to figure out which sections go where. That works at small scale, with a small content model. It stops working when your content model has forty content types, each with a dozen fields, and each document has a different structure. You end up burning a large context window on mostly-irrelevant content types, paying for tokens that don't help, and getting worse results because the model is reasoning about the full space of possibilities instead of a curated shortlist.\n\nThe pipeline I was working on (a Google Docs → Contentful import) addressed this with a deterministic pre-filtering stage that runs before any LLM call. Its job is not to do the mapping. Its job is to narrow the problem: segment the document into contiguous regions, score each region against every content type, and hand the LLM a small set of (scope, candidate content types) pairs instead of a full Cartesian product. This post is about how that scoring works and why the design choices look the way they do.\n\nThe document is treated as a sequence of blocks. Any block that is a tab opener, an H1, or an H2 starts a new scope. Every block between two openers belongs to the scope opened by the first one. The result is a flat array of `DocumentScope`\n\nobjects:\n\n```\nexport type ScopeReasonCode = 'tab-label' | 'heading-match' | 'body-term-match'\n\nexport type ContentTypeMatch = {\n  contentTypeId: string\n  score: number\n  reasons: ScopeReasonCode[]\n}\n\nexport type DocumentScope = {\n  openerText: string | null\n  openerKind: 'tab' | 'heading' | 'document-root'\n  blocks: NormalizedBlock[]\n  matches: ContentTypeMatch[]\n  topMatch: ContentTypeMatch | null\n  needsReview: boolean\n}\n```\n\n`openerText`\n\nis the raw text of the tab label or heading that opened the scope. `blocks`\n\nis the contiguous slice of the document. `matches`\n\nis every content type scored against this scope in descending score order. `topMatch`\n\nis a convenience pointer to the first match - `null`\n\nif there were no matches above the noise floor. `needsReview`\n\nis a flag, not a gate.\n\nThe function signature is straightforward:\n\n```\nexport function segmentDocumentScopes(\n  doc: NormalizedDocument,\n  slice: ContentModelOntologySlice,\n): DocumentScope[]\n```\n\nThe `ContentModelOntologySlice`\n\nis a pre-built graph of content type nodes, field nodes, and the `intentTerms`\n\nsets derived from each; more on those below.\n\nThe score for a (scope, content type) pair is Jaccard similarity over tokenized text:\n\n```\nfunction jaccardSimilarity(a: Set<string>, b: Set<string>): number {\n  if (a.size === 0 && b.size === 0) return 0\n  let intersection = 0\n  for (const term of a) {\n    if (b.has(term)) intersection++\n  }\n  const union = a.size + b.size - intersection\n  return union === 0 ? 0 : intersection / union\n}\n\nfunction tokenize(text: string): Set<string> {\n  return new Set(\n    text\n      .toLowerCase()\n      .replace(/[^a-z0-9\\s]/g, ' ')\n      .split(/\\s+/)\n      .filter((t) => t.length > 1),\n  )\n}\n```\n\n`|A ∩ B| / |A ∪ B|`\n\n. That is the entire base score. No TF-IDF weights, no vector embeddings, no pre-computed corpus statistics. The score for a scope with `{ \"product\", \"info\", \"title\" }`\n\nagainst a content type with intent terms `{ \"product\", \"title\", \"description\" }`\n\nis `2 / 4 = 0.5`\n\n.\n\nThe reason for Jaccard is not that it produces better scores than cosine similarity over TF-IDF vectors. It almost certainly does not. The reason is that this function runs synchronously on every request, before any LLM call. Embeddings require a model call. Cosine similarity over TF-IDF requires pre-computing corpus statistics from the full set of documents, which is not available at request time. Jaccard over a simple tokenized set is O(|A| + |B|), deterministic, reproducible, and has no external dependencies. It does not need to be perfect; it needs to be good enough to narrow forty content types to three, fast enough that it adds no measurable latency, and auditable enough that when it gets something wrong, you can read the code and understand exactly why.\n\nA tab label that reads \"Product Info\" is a categorically stronger signal than a body paragraph that mentions \"product\" once. The base Jaccard score treats every term equally regardless of where it appears. The bonus layer addresses this:\n\n```\nfunction computeScore(\n  scope: RawScope,\n  ctNode: ContentTypeNode,\n): { score: number; reasons: ScopeReasonCode[] } {\n  const intentTerms = ctNode.intentTerms\n  const bodyTerms = tokenize(scope.blocks.map((b) => b.text).join(' '))\n  const baseScore = jaccardSimilarity(bodyTerms, intentTerms)\n  const reasons: ScopeReasonCode[] = []\n\n  let bonus = 0\n\n  if (scope.openerKind === 'tab' && scope.openerText) {\n    const tabTerms = tokenize(scope.openerText)\n    const tabOverlap = jaccardSimilarity(tabTerms, intentTerms)\n    if (tabOverlap > 0) {\n      bonus += 0.2 * tabOverlap\n      reasons.push('tab-label')\n    }\n  }\n\n  if (scope.openerKind === 'heading' && scope.openerText) {\n    const headingTerms = tokenize(scope.openerText)\n    const headingOverlap = jaccardSimilarity(headingTerms, intentTerms)\n    if (headingOverlap > 0) {\n      bonus += 0.1 * headingOverlap\n      reasons.push('heading-match')\n    }\n  }\n\n  if (baseScore > 0) {\n    reasons.push('body-term-match')\n  }\n\n  return { score: baseScore + bonus, reasons }\n}\n```\n\nThe bonus is a multiplier on the overlap in the structural position, not a flat constant. A tab label that shares half its terms with the content type's intent terms contributes `0.2 × 0.5 = 0.1`\n\nto the score. A tab label that fully matches contributes `0.2`\n\n. The multiplier means the bonus scales with how strongly the structural position matched, not just whether it matched at all. A document that opens a tab called \"Product Info\" against a content type called \"Product\" with intent terms derived from that name gets a materially higher score than a document that mentions \"product\" once in a body paragraph with no structural indicator.\n\nThe three reason codes: `tab-label`\n\n, `heading-match`\n\n, `body-term-match`\n\n; exist for observability, not routing. When the LLM receives a scope with its top matches, the reason codes tell it (and any downstream monitoring) what evidence the scorer used. A match driven purely by `body-term-match`\n\nwith a low score deserves more skepticism from the LLM than one driven by `tab-label`\n\nwith a high score. Making the evidence explicit, not just the verdict, gives the downstream consumer something to work with.\n\nThe `ContentModelOntologySlice`\n\nis built deterministically from the raw content type definitions before any document is processed:\n\n```\nexport function buildContentModelOntologySlice(\n  contentTypes: ContentType[],\n): ContentModelOntologySlice {\n  const ctNodes: ContentTypeNode[] = contentTypes.map((ct) => {\n    const terms = new Set<string>()\n\n    // CT-level signals\n    for (const t of tokenize(ct.sys.id)) terms.add(t)\n    for (const t of tokenize(ct.name)) terms.add(t)\n    if (ct.description) for (const t of tokenize(ct.description)) terms.add(t)\n\n    // Field-level signals\n    for (const field of ct.fields) {\n      for (const t of tokenize(field.id)) terms.add(t)\n      for (const t of tokenize(field.name)) terms.add(t)\n      if (field.type === 'Link' && field.validations) {\n        for (const v of field.validations) {\n          if (v.linkContentType) {\n            for (const linkedCtId of v.linkContentType) {\n              for (const t of tokenize(linkedCtId)) terms.add(t)\n            }\n          }\n        }\n      }\n      // helpText, validation summaries, etc.\n    }\n\n    return { contentTypeId: ct.sys.id, intentTerms: terms, fields: [] }\n  })\n\n  return { ctNodes, fieldNodes: [], edges: [] }\n}\n```\n\nThe key decision here is tokenizing field IDs, content type IDs, and validation rules, not just display names and descriptions. A content type named `ctProductVariant`\n\nwith a field named `linkedEntries`\n\nthat has a `linkContentType: [\"BlogPost\"]`\n\nvalidation produces intent terms including `ctproductvariant`\n\n, `linkedentries`\n\n, and `blogpost`\n\n. A content type with a sparse or technical name (`pgBlogEntry`\n\n) still has vocabulary the scorer can match against; it isn't limited to whatever its display name happens to be.\n\nThis matters because many production content models have programmatic IDs that carry semantic information. A content type called \"Page: Blog Entry\" in the display name but `pgBlogEntry`\n\nin the API name will produce terms for both. The validation relationships (`linkContentType`\n\n, `allowedContentTypes`\n\n) bring in referenced content type IDs, which means a scope that references \"blog posts\" in passing can score against a container content type that links to `BlogPost`\n\n, even if the container's own name doesn't mention it.\n\nAfter scoring, three conditions set `needsReview: true`\n\non a scope:\n\n```\nfunction computeNeedsReview(scope: RawScope, matches: ContentTypeMatch[]): boolean {\n  // No content blocks — nothing to score against\n  if (scope.blocks.length === 0) return true\n\n  const top = matches[0]\n  const second = matches[1]\n\n  // Weak top score — no content type matched confidently\n  if (!top || top.score < 0.1) return true\n\n  // Ambiguous — top two are nearly tied\n  if (second && top.score - second.score < 0.05) return true\n\n  return false\n}\n```\n\nNone of these conditions throw an error or halt the pipeline. The pipeline continues with all scopes, flagged or not. The `needsReview`\n\nflag is a signal passed downstream to the LLM (and to monitoring), not a gate.\n\nThe reason for this design is that hard-failing on low scores would block legitimate edge cases: a document with no headings produces a single large scope that scores diffusely across many content types; a content type with sparse intent terms produces low scores even against obviously relevant documents. These are not pipeline errors; they are cases where the deterministic scorer ran out of signal. The right behavior is to continue and let the LLM handle those scopes with the full context of knowing the scorer wasn't confident, not to abort the import and force a manual retry.\n\nThe top-2 gap check deserves its own explanation. A score of 0.15 vs. 0.14 is not a decision, it is noise in a simple set-overlap function. Rather than let a near-tie pass through as a confident first-place assignment, the gap check flags it explicitly. The LLM sees two near-equal candidates and `needsReview: true`\n\n, which is accurate: the scorer does not have enough signal to pick a winner, and the LLM should treat both candidates seriously rather than assuming the top-ranked one is correct.\n\nThe specific implementation here, Jaccard similarity, structural bonuses, `needsReview`\n\nflags, is tied to this pipeline. The underlying decisions are not:\n\n`needsReview`\n\nas a first-class output is better than crashing or silently passing through low-confidence decisions.The LLM in this pipeline gets a scoped problem instead of a full document-by-content-model matrix. That is the only job of this stage. Getting it right deterministically means the expensive, slow, creative part of the pipeline; the LLM spends its context on decisions it actually needs to make.", "url": "https://wpnews.pro/news/scoring-documents-against-a-content-model-without-an-llm", "canonical_source": "https://dev.to/david_shibley/scoring-documents-against-a-content-model-without-an-llm-8lg", "published_at": "2026-07-28 20:12:45+00:00", "updated_at": "2026-07-28 21:03:11.320419+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Google Docs", "Contentful"], "alternates": {"html": "https://wpnews.pro/news/scoring-documents-against-a-content-model-without-an-llm", "markdown": "https://wpnews.pro/news/scoring-documents-against-a-content-model-without-an-llm.md", "text": "https://wpnews.pro/news/scoring-documents-against-a-content-model-without-an-llm.txt", "jsonld": "https://wpnews.pro/news/scoring-documents-against-a-content-model-without-an-llm.jsonld"}}