{"slug": "youtube-captions-in-postgres-preserve-the-revision-your-answer-cited", "title": "YouTube captions in Postgres: preserve the revision your answer cited", "summary": "A developer outlined a Postgres schema for preserving YouTube transcript evidence behind timestamped citations, separating recordings, immutable transcript artifacts, and collection observations into distinct tables. The approach keys artifacts on video ID, returned language, and an evidence hash that covers both text and segment boundaries, rather than upserting transcript text on video ID conflict. The developer also recommended pinning ingestion inputs with a strict Zod validator and disabling Whisper fallback to keep the evidence contract dependable.", "body_md": "Your answer still cites the video. The transcript behind that citation has changed, and your upsert has helpfully deleted the version you embedded.\n\nYou discover this while investigating an answer someone challenged. The recording is available. The evidence isn't.\n\nA video ID identifies a recording, not a transcript revision. The Actor processes duplicate video IDs once, but language selection and collection paths determine which text comes back. Deduplicating collection doesn't preserve evidence.\n\nFor timestamped citations, preserve text and segment boundaries together. A text-only hash misses timing changes, even when those changes move the passage your answer points to.\n\nUse Node with ES modules, `apify-client`, `zod`, and `pg`. Supply `APIFY_TOKEN` and your usual Postgres connection environment variables.\n\nThis Example input requests text and segments while disabling AI fallback:\n\n``` python\nimport { ApifyClient } from 'apify-client';\nimport { z } from 'zod';\nimport pg from 'pg';\nimport { createHash } from 'node:crypto';\n\nconst client = new ApifyClient({ token: process.env.APIFY_TOKEN });\nconst pool = new pg.Pool();\n\nconst run_input = z.object({\n  mode: z.literal('videos'),\n  videoUrls: z.array(z.string().url()).min(1),\n  language: z.literal('en'),\n  outputFormat: z.literal('full'),\n  useWhisper: z.literal(false),\n  includeWordTimings: z.literal(false),\n}).strict().parse({\n  mode: 'videos',\n  videoUrls: ['https://www.youtube.com/watch?v=jNQXAC9IVRw'],\n  language: 'en',\n  outputFormat: 'full',\n  useWhisper: false,\n  includeWordTimings: false,\n});\n```\n\nThe input validator deliberately accepts less than the source supports. Your ingestion job needs a dependable evidence contract, so allowing someone to switch it to subtitle output would create flexibility in precisely the wrong place.\n\nDisabling fallback sacrifices coverage. If you enable it later, use `maxWhisperMinutes` to bound eligible video length and review the spending policy separately.\n\nThat setting limits transcription eligibility. It isn't a monetary ceiling, and database deduplication doesn't refund collection work.\n\nThe dangerous statement is an upsert that replaces transcript text whenever `video_id` conflicts. It makes a recording look like a mutable document, although your citation needs the exact document it originally used.\n\nKeep these identities apart:\n\n``` php\nrecording: video_id\n   |\n   +-- returned language + evidence hash --> immutable artifact\n   |                                            |\n   +-- run ID + dataset position --> observation-+\n                                                |\n                                       embedding / citation\n```\n\n| Record | Key | Purpose | \n|---|---|---|\n| Video | `video_id` | Identify the recording | \n| Artifact | Video, returned language, evidence hash | Preserve citation evidence | \n| Observation | Run ID, dataset position | Preserve collection history | \n\nRequested language describes your intention. Returned language describes the artifact.\n\nLikewise, `transcript_source` describes how collection obtained the text. A transport change shouldn't manufacture a revision when the evidence remains identical, though you should retain that provenance for diagnosis.\n\nApply this schema before running the ingestion module:\n\n```\nCREATE TABLE ingest_runs (\n  run_id text PRIMARY KEY,\n  run_status text,\n  ingestion_complete boolean NOT NULL DEFAULT false\n);\nCREATE TABLE videos (video_id text PRIMARY KEY);\nCREATE TABLE transcript_artifacts (\n  video_id text NOT NULL REFERENCES videos,\n  transcript_language text NOT NULL,\n  evidence_hash text NOT NULL,\n  full_text text NOT NULL,\n  segments jsonb NOT NULL,\n  PRIMARY KEY (video_id, transcript_language, evidence_hash)\n);\nCREATE TABLE collection_observations (\n  run_id text NOT NULL REFERENCES ingest_runs,\n  item_index bigint NOT NULL,\n  video_id text REFERENCES videos,\n  transcript_language text,\n  evidence_hash text,\n  flags jsonb NOT NULL,\n  validation_errors jsonb NOT NULL,\n  raw jsonb NOT NULL,\n  PRIMARY KEY (run_id, item_index),\n  FOREIGN KEY (video_id, transcript_language, evidence_hash)\n    REFERENCES transcript_artifacts\n);\n```\n\nTyped columns carry identity and the fields needed to retrieve evidence. Raw JSONB retains each observation, including requested language, source, translation status, and collection time.\n\nThat split is intentional. Turning every metadata field into a column increases migration work without improving citation recovery; keeping everything inside JSONB makes the joins protecting that recovery harder to inspect.\n\nPromote provenance fields into indexed columns when your operational queries justify it. Until then, the original observation remains available without becoming part of artifact identity.\n\nA strict output schema makes harmless metadata additions fail ingestion. Unchecked passthrough does the opposite: it accepts change while leaving you unaware that anything moved.\n\nUse passthrough with explicit drift flags. Known evidence fields must still have the expected types, and a malformed record must retain its raw body instead of receiving helpful coercion.\n\nThese schemas distinguish collection outcomes from transcripts your indexer can accept:\n\n``` js\nconst Segment = z.object({\n  start: z.number().finite().nonnegative(),\n  dur: z.number().finite().nonnegative(),\n  end: z.number().finite().nonnegative(),\n  text: z.string(),\n}).passthrough().refine(s => s.end >= s.start);\n\nconst Envelope = z.object({\n  video_id: z.string().min(1).optional(),\n  requested_language: z.string().optional(),\n  transcript_language: z.string().min(1).optional(),\n  was_translated: z.boolean().optional(),\n  is_auto_generated: z.boolean().optional(),\n  transcript_source: z.enum([\n    'library', 'android_player', 'playwright_dom', 'whisper',\n  ]).optional(),\n  full_text: z.string().optional(),\n  segments: z.array(Segment).optional(),\n  success: z.boolean(),\n  transcript_available: z.boolean(),\n  scrapedAt: z.string().datetime({ offset: true }),\n  errorCode: z.string().optional(),\n  retryable: z.boolean().optional(),\n}).passthrough();\n\nconst Evidence = Envelope.extend({\n  video_id: z.string().min(1),\n  transcript_language: z.string().min(1),\n  success: z.literal(true),\n  transcript_available: z.literal(true),\n  full_text: z.string().refine(text => text.trim().length > 0),\n  segments: z.array(Segment).min(1),\n});\n\nconst knownFields = new Set(`\n  video_id title channel_name channel_id canonicalUrl sourceUrl\n  published_date publishedAt duration_seconds views like_count category\n  keywords description thumbnail channelHandle channelUrl channelThumbnail\n  channelVerified subscriberCountText subscriberCount commentCountText\n  commentCount isUnlisted is_family_safe is_live_content is_upcoming\n  accessStatus wasLive liveStatus scheduledStartTime chapters heatmap\n  captionTracks translationLanguages requested_language transcript_language\n  transcript_language_name language_matched is_auto_generated was_translated\n  transcript_source available_languages segments full_text wordTimings\n  srt vtt ttml segment_count transcript_available success recordType\n  error errorCode retryable language_probability inputUrl scrapedAt\n`.trim().split(/\\s+/));\nconst segmentFields = new Set(['start', 'dur', 'end', 'text']);\n```\n\nThis validates the fields the ingestion path depends on. It preserves other metadata without claiming that every nested structure has been checked.\n\nThe distinction matters during an incident. “Stored successfully” means you retained the observation; “accepted evidence” means its text and timing passed the narrower contract your citations depend on.\n\nMissing optional metadata doesn't prevent acceptance. Missing transcript evidence does.\n\nCanonicalization needs an explicit policy. Here, it preserves exact text and ordered segment values, with a version marker identifying the application's hashing convention.\n\nDon't trim whitespace or round timestamps unless your consumer applies those same transformations before storing and citing evidence. Otherwise, the hash promises equivalence that your users don't actually receive.\n\nThis function prepares an observation and attaches evidence only after validation:\n\n``` js\nfunction prepare(raw: unknown, item_index: number) {\n  const row = {\n    item_index, raw,\n    video_id: null as string | null,\n    transcript_language: null as string | null,\n    evidence_hash: null as string | null,\n    artifact: null as {\n      full_text: string;\n      segments: { start: number; dur: number; end: number; text: string }[];\n    } | null,\n    flags: [] as string[],\n    validation_errors: [] as unknown[],\n  };\n  if (raw && typeof raw === 'object' &&\n      Object.keys(raw).some(key => !knownFields.has(key))) {\n    row.flags.push('unknown fields');\n  }\n  const parsed = Envelope.safeParse(raw);\n  if (!parsed.success) {\n    row.flags.push('invalid shape');\n    row.validation_errors = parsed.error.issues;\n    return row;\n  }\n  const record = parsed.data;\n  row.video_id = record.video_id ?? null;\n  row.transcript_language = record.transcript_language ?? null;\n  if (record.segments?.some(s =>\n    Object.keys(s).some(key => !segmentFields.has(key))\n  )) row.flags.push('unknown segment fields');\n  if (!record.success || !record.transcript_available) {\n    row.flags.push('collection failed');\n    return row;\n  }\n  const checked = Evidence.safeParse(record);\n  if (!checked.success) {\n    row.flags.push('invalid evidence');\n    row.validation_errors = checked.error.issues;\n    return row;\n  }\n  row.artifact = {\n    full_text: checked.data.full_text,\n    segments: checked.data.segments.map(({ start, dur, end, text }) =>\n      ({ start, dur, end, text })),\n  };\n  row.evidence_hash = createHash('sha256').update(JSON.stringify([\n    'timed-transcript-v1',\n    row.artifact.full_text,\n    row.artifact.segments.map(s => [s.start, s.dur, s.end, s.text]),\n  ])).digest('hex');\n  return row;\n}\n```\n\nNotice what stays outside the hash: scrape time, requested language, and transport provenance. Identical evidence collected again produces the same artifact key while gaining another observation.\n\nThe version marker isn't decoration. Changing normalization rules requires a migration decision, because the resulting hashes belong to a different identity scheme.\n\nThis contract fits segment-based citations. If you later cite individual word timings or require source-specific approvals, extend the contract before treating those properties as interchangeable.\n\nEach page commits atomically. The database inserts video identities, inserts unseen artifacts, and records observations without updating existing transcript content.\n\nThe transaction also takes an application-wide writer lock. That's conservative, but it prevents concurrent imports from overlooking each other's revisions when deciding whether new evidence differs from already stored material.\n\nThis writer uses batch statements rather than issuing an insert for every record:\n\n```\nasync function persist(runId: string, rows: ReturnType<typeof prepare>[]) {\n  const db = await pool.connect();\n  const payload = JSON.stringify(rows);\n  try {\n    await db.query('BEGIN');\n    await db.query('SELECT pg_advisory_xact_lock(9173)');\n    await db.query(`\n      INSERT INTO videos\n      SELECT DISTINCT video_id\n      FROM jsonb_to_recordset($1::jsonb) AS x(video_id text)\n      WHERE video_id IS NOT NULL ON CONFLICT DO NOTHING\n    `, [payload]);\n    await db.query(`\n      WITH incoming AS (\n        SELECT * FROM jsonb_to_recordset($2::jsonb) AS x(\n          item_index bigint, video_id text, transcript_language text,\n          evidence_hash text, artifact jsonb, flags jsonb,\n          validation_errors jsonb, raw jsonb\n        )\n      ), added AS (\n        INSERT INTO transcript_artifacts\n        SELECT DISTINCT ON (video_id, transcript_language, evidence_hash)\n          video_id, transcript_language, evidence_hash,\n          artifact->>'full_text', artifact->'segments'\n        FROM incoming WHERE evidence_hash IS NOT NULL\n        ON CONFLICT DO NOTHING\n        RETURNING video_id, transcript_language, evidence_hash\n      )\n      INSERT INTO collection_observations\n      SELECT $1, x.item_index, x.video_id, x.transcript_language,\n        x.evidence_hash,\n        x.flags || CASE WHEN\n          EXISTS (SELECT 1 FROM added n\n            WHERE (n.video_id, n.transcript_language, n.evidence_hash) =\n                  (x.video_id, x.transcript_language, x.evidence_hash))\n          AND EXISTS (SELECT 1 FROM transcript_artifacts old\n            WHERE old.video_id = x.video_id\n              AND old.transcript_language = x.transcript_language\n              AND old.evidence_hash <> x.evidence_hash)\n        THEN jsonb_build_array('changed evidence') ELSE '[]'::jsonb END,\n        x.validation_errors, COALESCE(x.raw, 'null'::jsonb)\n      FROM incoming x\n      ON CONFLICT (run_id, item_index) DO NOTHING\n    `, [runId, payload]);\n    await db.query('COMMIT');\n  } catch (error) {\n    await db.query('ROLLBACK');\n    throw error;\n  } finally {\n    db.release();\n  }\n}\n```\n\nThe revision comparison sees artifacts present before that statement. A newly inserted artifact receives a change flag when its language already has different evidence; replaying an existing artifact doesn't manufacture another revision alert.\n\nThe flag strings belong to your application. They describe ingestion decisions, not fields returned by the source.\n\nThe global lock limits write throughput. At larger scale, assign ownership by video or use consistently ordered per-video locks, preserving the requirement that revision detection and artifact insertion agree about prior state.\n\nStore the artifact's composite key with every embedding and citation. Looking up the newest transcript at answer time recreates the original failure with better table names.\n\nA terminal run gives dataset positions a stable role in observation identity. Don't use this positional key while results are still arriving, and keep the dataset intact while recovery remains possible.\n\nThe driver accepts `APIFY_RUN_ID` for replaying an existing collection. Otherwise, it starts a collection, waits for termination, and drains every dataset page:\n\n``` js\ntry {\n  const started = process.env.APIFY_RUN_ID\n    ? { id: process.env.APIFY_RUN_ID }\n    : await client.actor(\n        'nimbryxo-studio/youtube-caption-and-whisper-ai-transcript'\n      ).start(run_input);\n\n  await pool.query(`\n    INSERT INTO ingest_runs (run_id) VALUES ($1)\n    ON CONFLICT (run_id) DO UPDATE SET ingestion_complete = false\n  `, [started.id]);\n\n  const run = await client.run(started.id).waitForFinish();\n  if (!run || !['SUCCEEDED', 'FAILED', 'TIMED-OUT', 'ABORTED']\n    .includes(run.status)) {\n    throw new Error(`Collection still pending: ${started.id}`);\n  }\n  await pool.query(`\n    UPDATE ingest_runs SET run_status = $2 WHERE run_id = $1\n  `, [started.id, run.status]);\n\n  if (!run.defaultDatasetId) throw new Error('Dataset unavailable');\n  let offset = 0;\n  while (true) {\n    const { items } = await client.dataset(run.defaultDatasetId)\n      .listItems({ offset, limit: 100 });\n    if (items.length === 0) break;\n    await persist(started.id,\n      items.map((item, index) => prepare(item, offset + index)));\n    offset += items.length;\n  }\n  await pool.query(`\n    UPDATE ingest_runs SET ingestion_complete = true WHERE run_id = $1\n  `, [started.id]);\n} finally {\n  await pool.end();\n}\n```\n\nA page failure rolls back that page. Previously committed pages remain available, and replaying the same dataset encounters their existing observation keys.\n\nCompleting ingestion means you consumed the dataset, not that every video produced indexable evidence. Terminal collection failures can still leave rows worth preserving, so the run status and ingestion status remain separate.\n\nThere's an unavoidable external boundary here: collection can start before database registration fails. Recover that run ID before retrying, because a transaction in your database can't make an external start request exactly once.\n\nPersist or capture the returned ID in your job runner as well. Another collection is another operational attempt and can incur additional charges, even when every resulting artifact already exists.\n\nThe worked source is [YouTube Caption & Whisper AI Transcript Extractor](https://apify.com/nimbryxo-studio/youtube-caption-and-whisper-ai-transcript). Its language behavior makes the identity problem concrete.\n\nIt selects an existing language track, including a regional variant, before attempting translation. Selecting a track isn't translation.\n\nTranslation can be rate-limited. When that path fails, the Actor can return the original transcript with `language_matched: false`, which means an artifact keyed by requested language could label your evidence incorrectly.\n\nThis recipe stores acceptable evidence under its returned language. Your indexer still needs a separate admission policy for the collection's intended language; preserving an artifact doesn't require serving it in the wrong search index.\n\nAccess failures need different treatment. Restricted, unavailable, live, or upcoming videos can return metadata with `success: false`, and persistent connection blocks can also prevent transcription.\n\nThose observations describe collection outcomes. They don't invalidate a transcript you accepted earlier.\n\nAlert on invalid shapes, incomplete ingestion, and non-successful run status. Review unknown fields as schema drift, and route collection failures using `errorCode` and `retryable` rather than repeatedly retrying every unavailable video.\n\nFor changed evidence, join the flagged artifact's video and language to your embedding references before paging someone. A revision affecting an active citation deserves attention; unindexed material usually needs processing rather than sleep deprivation.\n\nThe recipe flags changes against previously stored evidence, including artifacts that haven't been indexed. That wider detection boundary is useful, provided your alert routing narrows it to the dependencies your users actually rely on.\n\nA citation should still identify the evidence your answer used after the next scrape disagrees.\n\n*Drafted with AI assistance and reviewed for accuracy.*", "url": "https://wpnews.pro/news/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited", "canonical_source": "https://dev.to/nimbryxo-studio/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited-42eg", "published_at": "2026-11-18 10:09:28+00:00", "updated_at": "2026-11-18 10:13:41.598246+00:00", "lang": "en", "topics": ["ai-search", "ai-agents", "structured-data"], "entities": ["YouTube", "Postgres", "Apify", "Node.js", "Zod"], "also_reported_by": [], "takeaway": "Key transcript artifacts on video ID, returned language and a hash of text plus segment boundaries instead of upserting on video ID, so a changed transcript never overwrites the revision a citation used.", "recommendation": {"advice": "Store transcript text and segment boundaries together under an evidence hash keyed by video ID and returned language, and never upsert transcript text on video ID conflict alone.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited", "markdown": "https://wpnews.pro/news/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited.md", "text": "https://wpnews.pro/news/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited.txt", "jsonld": "https://wpnews.pro/news/youtube-captions-in-postgres-preserve-the-revision-your-answer-cited.jsonld"}}