{"slug": "building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and", "title": "Building a Real-Time AI Interview Analysis Pipeline with LiveKit, Deepgram, and Grok", "summary": "TrueVoice HQ has built a real-time AI interview analysis pipeline using LiveKit, Deepgram, and Grok that scores live video interviews chunk by chunk, providing interviewers with live authenticity signals rather than a delayed report. The system processes transcript chunks every 30-60 seconds, scoring four dimensions and flagging patterns like AI-generated opener phrases, while handling reasoning model output with a custom JSON extraction utility.", "body_md": "Most AI apps process data after the fact. You upload a file, wait, get a result. This post is about doing it the other way: scoring a live video interview as the candidate speaks, chunk by chunk, so the interviewer sees signals in real time rather than a report 20 minutes later.\n\nThis is the architecture behind TrueVoice HQ ([https://truevoicehq.com](https://truevoicehq.com)), an AI interview analysis platform. Here's how the real-time pipeline works.\n\nThe Pipeline\n\nCandidate speaks\n\n↓\n\nLiveKit (WebRTC audio/video stream)\n\n↓\n\nDeepgram (real-time STT → transcript chunks)\n\n↓\n\nSupabase Edge Function: analyze-chunk (Grok scores each chunk live)\n\n↓\n\nSupabase Realtime (scores pushed to interviewer dashboard)\n\n↓\n\nSupabase Edge Function: generate-final-report (Grok synthesizes full report on end)\n\nFive services. One direction. No polling.\n\nWhy Chunk-Based Analysis?\n\nThe naive approach: record the whole interview, transcribe it, send the full transcript to an LLM, get scores back. This works fine for async review. It's useless for live interviews.\n\nChunk-based analysis means the interviewer sees a live authenticity score updating every 30-60 seconds. By the time the interview ends, the final report is synthesizing data that's already been processed, not starting from scratch.\n\nEach chunk scores four dimensions (0-25 each, 100 total):\n\nThe system also flags specific patterns: AI-generated opener phrases (\"Certainly\", \"Absolutely\", \"That's a great question\"), suspiciously precise statistics without citations, and company-specific phrases each organization has flagged from past interviews.\n\nThe Edge Function: analyze-chunk\n\nDeployed on Supabase Edge Functions (Deno runtime). Called every time a new transcript chunk arrives from Deepgram.\n\n// Uses grok-3-fast -- non-reasoning model for lower latency on chunk analysis\n\nconst grokResponse = await fetch(\"[https://api.x.ai/v1/chat/completions](https://api.x.ai/v1/chat/completions)\", {\n\nmethod: \"POST\",\n\nheaders: { Authorization: `Bearer ${xaiKey}`\n\n, \"Content-Type\": \"application/json\" },\n\nbody: JSON.stringify({\n\nmodel: \"grok-3-fast\",\n\nmessages: [\n\n{ role: \"system\", content: ANALYSIS_PROMPT },\n\n{\n\nrole: \"user\",\n\ncontent: [\n\n`Elapsed time: ${elapsed_seconds}s`\n\n,\n\n`Response timing data: ${JSON.stringify(response_delays)}`\n\n,\n\n`Transcript chunk: \"${chunk_text}\"`\n\n,\n\n].join(\"\\n\"),\n\n},\n\n],\n\ntemperature: 0.3,\n\n}),\n\n});\n\nThe response timing data (response_delays) is the interesting part. It's measured on the client side by tracking the gap between when the interviewer finishes a question and when the candidate starts speaking. That timestamp gets passed with each chunk so the model has real delay data, not just inference from speech patterns.\n\nHandling Reasoning Model Output in Production\n\nOne problem you hit immediately when using LLMs in a structured data pipeline: the model sometimes wraps its JSON in markdown fences, adds explanation text, or (for reasoning models) prepends a ... block before the actual output.\n\nThis utility handles all three cases:\n\nfunction extractJson(raw: string): string | null {\n\n// Strip reasoning model thinking blocks\n\nconst stripped = raw.replace(/[\\s\\S]*?<\\/think>/gi, \"\").trim();\n\n``` js\n// Find all {...} blocks, try from last to first\nconst matches = [...stripped.matchAll(/\\{[\\s\\S]*?\\}/g)];\nfor (let i = matches.length - 1; i >= 0; i--) {\n  try {\n    JSON.parse(matches[i][0]);\n    return matches[i][0];\n  } catch {\n    continue;\n  }\n}\n\n// Fall back to greedy match\nconst greedy = stripped.match(/\\{[\\s\\S]*\\}/);\nreturn greedy ? greedy[0] : null;\n```\n\n}\n\nWhy take the last match rather than the first? When a model produces explanation text before JSON, the last {...} block is the actual structured output. The earlier ones are fragments from the explanation. This pattern has held up across Grok, Claude, and GPT-4o.\n\nIf parsing fails completely, fall back to neutral mid-range scores (15/25 each) rather than zeros. Zeros would make the interviewer dashboard look alarming. Neutral scores let the session continue without a false signal.\n\nLiveKit JWT Generation in Deno (No Library Required)\n\nLiveKit access tokens are signed JWTs. In Node.js you'd use the livekit-server-sdk. In Deno edge functions, npm libraries aren't always available cleanly, but the Web Crypto API is. Here's a\n\npure-Deno JWT generator:\n\nasync function generateLiveKitToken(\n\napiKey: string,\n\napiSecret: string,\n\nroomName: string,\n\nparticipantIdentity: string,\n\nparticipantName: string,\n\nisHost: boolean = false\n\n): Promise {\n\nconst now = Math.floor(Date.now() / 1000);\n\nconst payload = {\n\niss: apiKey,\n\nsub: participantIdentity,\n\nexp: now + 86400,\n\nnbf: now,\n\nname: participantName,\n\nvideo: {\n\nroom: roomName,\n\nroomJoin: true,\n\ncanPublish: true,\n\ncanSubscribe: true,\n\ncanPublishData: true,\n\n...(isHost && { roomAdmin: true, roomCreate: true, roomRecord: true }),\n\n},\n\n};\n\n``` js\nconst encoder = new TextEncoder();\nconst headerB64 = toBase64Url(encoder.encode(JSON.stringify({ alg: \"HS256\", typ: \"JWT\" })));\nconst payloadB64 = toBase64Url(encoder.encode(JSON.stringify(payload)));\nconst signingInput = `${headerB64}.${payloadB64}`;\n\nconst key = await crypto.subtle.importKey(\n  \"raw\",\n  encoder.encode(apiSecret),\n  { name: \"HMAC\", hash: \"SHA-256\" },\n  false,\n  [\"sign\"]\n);\n\nconst signature = await crypto.subtle.sign(\"HMAC\", key, encoder.encode(signingInput));\nreturn `${signingInput}.${toBase64Url(new Uint8Array(signature))}`;\n```\n\n}\n\nrole: \"user\",\n\ncontent: [\n\n`Elapsed time: ${elapsed_seconds}s`\n\n,\n\n`Response timing data: ${JSON.stringify(response_delays)}`\n\n,\n\n`Transcript chunk: \"${chunk_text}\"`\n\n,\n\n].join(\"\\n\"),\n\n},\n\n],\n\ntemperature: 0.3,\n\n}),\n\n});\n\nThe response timing data (response_delays) is the interesting part. It's measured on the client side by tracking the gap between when the interviewer finishes a question and when the candidate starts speaking. That timestamp gets passed with each chunk so the model has real delay data, not just inference from speech patterns.\n\nHandling Reasoning Model Output in Production\n\nOne problem you hit immediately when using LLMs in a structured data pipeline: the model sometimes wraps its JSON in markdown fences, adds explanation text, or (for reasoning models) prepends a ... block before the actual output.\n\nThis utility handles all three cases:\n\nfunction extractJson(raw: string): string | null {\n\n// Strip reasoning model thinking blocks\n\nconst stripped = raw.replace(/[\\s\\S]*?<\\/think>/gi, \"\").trim();\n\n``` js\n// Find all {...} blocks, try from last to first\nconst matches = [...stripped.matchAll(/\\{[\\s\\S]*?\\}/g)];\nfor (let i = matches.length - 1; i >= 0; i--) {\n  try {\n    JSON.parse(matches[i][0]);\n    return matches[i][0];\n  } catch {\n    continue;\n  }\n}\n\n// Fall back to greedy match\nconst greedy = stripped.match(/\\{[\\s\\S]*\\}/);\nreturn greedy ? greedy[0] : null;\n```\n\n}\n\nWhy take the last match rather than the first? When a model produces explanation text before JSON, the last {...} block is the actual structured output. The earlier ones are fragments from the explanation. This pattern has held up across Grok, Claude, and GPT-4o.\n\nIf parsing fails completely, fall back to neutral mid-range scores (15/25 each) rather than zeros. Zeros would make the interviewer dashboard look alarming. Neutral scores let the session continue without a false signal.\n\nLiveKit JWT Generation in Deno (No Library Required)\n\nLiveKit access tokens are signed JWTs. In Node.js you'd use the livekit-server-sdk. In Deno edge functions, npm libraries aren't always available cleanly, but the Web Crypto API is. Here's a\n\npure-Deno JWT generator:\n\nasync function generateLiveKitToken(\n\napiKey: string,\n\napiSecret: string,\n\nroomName: string,\n\nparticipantIdentity: string,\n\nparticipantName: string,\n\nisHost: boolean = false\n\n): Promise {\n\nconst now = Math.floor(Date.now() / 1000);\n\nconst payload = {\n\niss: apiKey,\n\nsub: participantIdentity,\n\nexp: now + 86400,\n\nnbf: now,\n\nname: participantName,\n\nvideo: {\n\nroom: roomName,\n\nroomJoin: true,\n\ncanPublish: true,\n\ncanSubscribe: true,\n\ncanPublishData: true,\n\n...(isHost && { roomAdmin: true, roomCreate: true, roomRecord: true }),\n\n},\n\n};\n\n``` js\nconst encoder = new TextEncoder();\nconst headerB64 = toBase64Url(encoder.encode(JSON.stringify({ alg: \"HS256\", typ: \"JWT\" })));\nconst payloadB64 = toBase64Url(encoder.encode(JSON.stringify(payload)));\nconst signingInput = `${headerB64}.${payloadB64}`;\n\nconst key = await crypto.subtle.importKey(\n  \"raw\",\n  encoder.encode(apiSecret),\n  { name: \"HMAC\", hash: \"SHA-256\" },\n  false,\n  [\"sign\"]\n);\n\nconst signature = await crypto.subtle.sign(\"HMAC\", key, encoder.encode(signingInput));\nreturn `${signingInput}.${toBase64Url(new Uint8Array(signature))}`;\n```\n\n}\n\nfunction toBase64Url(data: Uint8Array): string {\n\nreturn btoa(String.fromCharCode(...data))\n\n.replace(/=/g, \"\")\n\n.replace(/+/g, \"-\")\n\n.replace(/\\//g, \"_\");\n\n}\n\ncrypto.subtle is available globally in Deno, btoa handles the base64 encoding, and the URL-safe replacement (+ to -, / to _, strip =) is required by the JWT spec.\n\nFinal Report: Synthesis, Not Re-Analysis\n\nWhen the interview ends, generate-final-report doesn't re-process the transcript from scratch. It synthesizes from the chunk data already stored:\n\nconst [chunksRes, interviewRes, flagsRes] = await Promise.all([\n\nsupabase.from(\"transcript_chunks\").select(\"*\").eq(\"interview_id\", interview_id).order(\"chunk_index\"),\nsupabase.from(\"interviews\").select(\"*\").eq(\"interview_id\", interview_id).single(),\n\nThe model gets the chunk-level score history, the flag log with timestamps, and the full transcript. This produces a more accurate final report than sending the raw transcript alone. The chunk scores act as a ground truth the model can't contradict.\n\nTwo post-report tasks run fire-and-forget (non-blocking):\n\nif (interview.resume_text) {\n\nfetch(`${supabaseUrl}/functions/v1/analyze-resume`\n\n, { ... }).catch(() => {});\n\n}\n\nif (interview.candidate_email) {\n\nfetch(`${supabaseUrl}/functions/v1/send-interview-email`\n\n, { ... }).catch(() => {});\n\n}\n\nThe .catch(() => {}) is intentional. These are nice-to-have enrichments. If they fail, the report is already saved. Don't let optional steps block the critical path.\n\nWhat I'd Do Differently\n\nMove the Deepgram connection server-side. The current architecture has the client receiving Deepgram transcripts and forwarding chunks to the edge function. Moving it server-side would eliminate one round trip and reduce the chance of a client network hiccup dropping a chunk.\n\nChunk overlap. Each chunk is currently independent. A candidate who trails off at the end of one chunk and picks up mid-thought in the next looks like two separate utterances. Adding the last 2-3 sentences of the previous chunk as context would smooth out scoring at boundaries.\n\nSupabase Realtime subscriptions instead of polling. Scores get written to the database and the interviewer dashboard polls. Switching to Realtime would push score updates instantly.\n\nStack Summary\n\n┌───────────────────────┬───────────────────────────┐\n\n│ Layer │ Service │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Video/audio streaming │ LiveKit Cloud │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Speech-to-text │ Deepgram Nova-2 │\n\n├───────────────────────┼───────────────────────────┤\n\n│ AI analysis │ xAI Grok 3 Fast │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Edge functions │ Supabase (Deno) │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Database + auth │ Supabase (PostgreSQL) │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Frontend │ React + Vite + TypeScript │\n\n├───────────────────────┼───────────────────────────┤\n\n│ Deployment │ Vercel │\n\n└───────────────────────┴───────────────────────────┘\n\nSource: [https://github.com/patrickboxfordpartners/truevoice](https://github.com/patrickboxfordpartners/truevoice) · Live: [https://truevoicehq.com](https://truevoicehq.com)\n\nPatrick Mitchell is the founder of Boxford Partners ([https://boxfordpartners.com](https://boxfordpartners.com)), a product studio that has shipped 8 companies across AI, SaaS, voice, and fintech.", "url": "https://wpnews.pro/news/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and", "canonical_source": "https://dev.to/patrickboxfordpartners/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and-grok-1oj2", "published_at": "2026-07-24 05:16:38+00:00", "updated_at": "2026-07-24 06:02:12.328229+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-tools", "developer-tools"], "entities": ["TrueVoice HQ", "LiveKit", "Deepgram", "Grok", "Supabase", "xAI"], "alternates": {"html": "https://wpnews.pro/news/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and", "markdown": "https://wpnews.pro/news/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and.md", "text": "https://wpnews.pro/news/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and.txt", "jsonld": "https://wpnews.pro/news/building-a-real-time-ai-interview-analysis-pipeline-with-livekit-deepgram-and.jsonld"}}