A multimodal voice companion creates an awkward product tension: users want to ask “What am I looking at?” without granting an AI system indefinite access to their camera.
The easiest implementation—forwarding frames continuously—also creates hidden costs. It increases data transfer and model work, makes visual context harder to reproduce, and leaves users unsure when the companion is actually observing them. It can also produce a subtler correctness bug: the model answers from an old frame while speaking as if it can see the present.
A better default for many companion experiences is one-shot visual context:
This tutorial builds that control layer in TypeScript. Tencent RTC supplies the real-time conversational setting, while Gemini sits behind an application-owned multimodal model port. We will not treat the model as the camera controller, consent authority, speech recognizer, or media transport.
Multimodal capability does not automatically justify continuous vision. Choose the smallest visual scope that supports the task.
| User task | Visual policy | Trade-off |
|---|---|---|
| Identify an object or read a label | One approved frame | Low exposure, but the user may need to recapture |
| Compare two arrangements | Two explicitly labelled frames | More application state and UI work |
| Explain ongoing movement | Time-bounded video may be necessary | Higher privacy, bandwidth, and moderation cost |
| General voice companionship | Camera off by default | The companion cannot answer visual questions until invited |
A still frame is the wrong abstraction for motion. If someone asks whether their exercise form remains correct over ten seconds, do not send one image and let the model imply that it observed the full movement.
The demonstrated capability is narrower: a multimodal model can receive a bounded text-and-image request. The hype-shaped interpretation—that it continuously understands the user’s environment—is a product decision your application should not silently make.
A production conversational pipeline should remain separable:
Microphone
-> RTC/media transport
-> speech recognition
-> application turn coordinator
-> approved visual snapshot
-> Gemini/model adapter
-> output moderation
-> speech synthesis
-> RTC/media transport
-> user
Tencent Conversational AI is documented as a real-time voice interaction scenario that can work with multiple LLM providers. Its overview is the appropriate starting point for the voice architecture:
Tencent RTC also documents LLM configuration, including OpenAI-compatible model connections, agent platforms such as Dify or Coze, and request identifiers used for routing and observability:
Do not infer from voice connectivity that a selected model route accepts images. Validate multimodal support for the exact provider and model configuration you operate. A text-only route should fail as text-only, not quietly discard the image and produce a confident answer.
The visual permission and the voice turn are related, but they are not the same state.
Our snapshot can be:
off -> capturing -> ready -> consumed
off -> capturing -> unavailable
capturing/ready/requesting -> off, when the user withdraws it
A monotonically increasing visualEpoch
invalidates asynchronous work. Every capture and withdrawal advances the epoch. A callback may mutate state only if it still belongs to the current epoch.
The important invariants are:
mkdir one-shot-visual-companion
cd one-shot-visual-companion
npm init -y
npm install --save-dev typescript tsx @types/node
npx tsc --init
mkdir src
Add these scripts to package.json
:
{
"scripts": {
"check": "tsc --noEmit",
"test": "tsx --test src/*.test.ts"
}
}
Use a strict TypeScript configuration:
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noUncheckedIndexedAccess": true
}
}
Create src/visual-turn.ts
:
import { randomUUID } from 'node:crypto';
export type Snapshot = {
id: string;
capturedAt: number;
mimeType: 'image/jpeg' | 'image/png';
bytes: Uint8Array;
};
type VisualState =
| { kind: 'off' }
| { kind: 'capturing'; epoch: number }
| { kind: 'ready'; epoch: number; snapshot: Snapshot }
| { kind: 'unavailable'; epoch: number; reason: string };
export interface CameraPort {
captureOneFrame(): Promise<Snapshot>;
}
export type ModelRequest = {
requestId: string;
text: string;
image?: {
mimeType: Snapshot['mimeType'];
bytes: Uint8Array;
};
signal: AbortSignal;
};
export interface MultimodalModelPort {
generate(request: ModelRequest): Promise<string>;
}
export interface OutputGate {
approve(text: string): Promise<boolean>;
}
export interface CompanionUi {
status(message: string): void;
speakApprovedText(text: string): void;
}
type ActiveRequest = {
turnId: string;
visualEpoch: number;
usedImage: boolean;
abort: AbortController;
};
export class VisualTurnCoordinator {
private visualEpoch = 0;
private visual: VisualState = { kind: 'off' };
private active?: ActiveRequest;
constructor(
private readonly camera: CameraPort,
private readonly model: MultimodalModelPort,
private readonly outputGate: OutputGate,
private readonly ui: CompanionUi,
private readonly now: () => number = Date.now,
private readonly maxSnapshotAgeMs = 15_000
) {}
async shareOneFrame(): Promise<void> {
this.abortImageRequest('A newer visual context was requested.');
const epoch = ++this.visualEpoch;
this.visual = { kind: 'capturing', epoch };
this.ui.status('Capturing one frame…');
try {
const snapshot = await this.camera.captureOneFrame();
// The user may have withdrawn permission while capture was pending.
if (epoch !== this.visualEpoch) return;
this.visual = { kind: 'ready', epoch, snapshot };
this.ui.status('One frame is ready for your next question.');
} catch (error) {
if (epoch !== this.visualEpoch) return;
const reason = error instanceof Error ? error.message : 'Capture failed';
this.visual = { kind: 'unavailable', epoch, reason };
this.ui.status('I could not capture the frame. Voice mode is still available.');
}
}
stopSharing(): void {
++this.visualEpoch;
this.visual = { kind: 'off' };
this.abortImageRequest('Visual sharing was withdrawn.');
this.ui.status('Visual context is off.');
}
async onFinalTranscript(text: string): Promise<void> {
const trimmed = text.trim();
if (!trimmed) return;
const selected = this.takeFreshSnapshot();
if (refersToVisibleContext(trimmed) && !selected) {
this.ui.status('I do not have a current frame. Share one frame, or describe it aloud.');
return;
}
const turnId = randomUUID();
const abort = new AbortController();
const visualEpoch = this.visualEpoch;
this.active = {
turnId,
visualEpoch,
usedImage: selected !== undefined,
abort
};
this.ui.status(selected ? 'Thinking about the approved frame…' : 'Thinking…');
try {
const answer = await this.model.generate({
requestId: turnId,
text: trimmed,
image: selected
? { mimeType: selected.mimeType, bytes: selected.bytes }
: undefined,
signal: abort.signal
});
if (!this.isCurrent(turnId, visualEpoch, selected !== undefined)) return;
if (!(await this.outputGate.approve(answer))) {
if (this.active?.turnId === turnId) this.active = undefined;
this.ui.status('That response could not be played. Please rephrase the question.');
return;
}
if (!this.isCurrent(turnId, visualEpoch, selected !== undefined)) return;
this.active = undefined;
this.ui.speakApprovedText(answer);
} catch (error) {
if (this.active?.turnId !== turnId) return;
this.active = undefined;
if (abort.signal.aborted) {
this.ui.status('That visual turn was cancelled.');
} else {
this.ui.status('The model could not answer. You can retry without sharing another frame.');
}
}
}
debugVisualKind(): VisualState['kind'] {
return this.visual.kind;
}
private takeFreshSnapshot(): Snapshot | undefined {
if (this.visual.kind !== 'ready') return undefined;
const { snapshot } = this.visual;
this.visual = { kind: 'off' }; // Consume it before starting asynchronous work.
if (this.now() - snapshot.capturedAt > this.maxSnapshotAgeMs) {
this.ui.status('That frame expired. Share a new one if the question is visual.');
return undefined;
}
return snapshot;
}
private abortImageRequest(message: string): void {
if (!this.active?.usedImage) return;
this.active.abort.abort(message);
this.active = undefined;
}
private isCurrent(
turnId: string,
visualEpoch: number,
usedImage: boolean
): boolean {
if (this.active?.turnId !== turnId) return false;
if (usedImage && visualEpoch !== this.visualEpoch) return false;
return true;
}
}
function refersToVisibleContext(text: string): boolean {
return /\b(this|that|these|those|here|camera|in front of me)\b/i.test(text);
}
The regular expression is intentionally not an AI safety classifier. It only catches obvious phrases that would otherwise invite a visual guess. False positives should result in a clarification, not an unsafe action.
Notice that image bytes leave the coordinator only inside MultimodalModelPort.generate
. This gives the application one auditable transfer boundary. It does not prove that an upstream provider deletes the image; retention, regional processing, and provider logging still need to be communicated and configured separately.
Avoid spreading provider-specific request objects through camera, transcript, and UI code. Implement a small adapter around the Gemini client or gateway selected by your backend:
export interface GeminiGateway {
generate(input: {
requestId: string;
prompt: string;
imageBase64?: string;
imageMimeType?: string;
signal: AbortSignal;
}): Promise<{ text: string }>;
}
export class GeminiAdapter implements MultimodalModelPort {
constructor(private readonly gateway: GeminiGateway) {}
async generate(request: ModelRequest): Promise<string> {
const result = await this.gateway.generate({
requestId: request.requestId,
prompt: request.text,
imageBase64: request.image
? Buffer.from(request.image.bytes).toString('base64')
: undefined,
imageMimeType: request.image?.mimeType,
signal: request.signal
});
if (!result.text.trim()) {
throw new Error('Gemini returned no usable text');
}
return result.text;
}
}
GeminiGateway
is an application-owned interface, not a claim about a provider’s wire format. Its backend implementation should use the currently supported Gemini client or your controlled model gateway. Keep the Tencent RTC LLM configuration, provider credentials, model identifier, and routing policy outside browser code.
Carry requestId
through the gateway where supported. That lets operational logs connect a voice turn to its model request without using the transcript itself as the identifier.
The coordinator consumes normalized events rather than SDK callbacks directly:
shareFrameButton.onclick = () => coordinator.shareOneFrame();
stopVisualButton.onclick = () => coordinator.stopSharing();
speechRecognition.onFinalText = (text: string) => {
void coordinator.onFinalTranscript(text);
};
Your integration layer remains responsible for:
CameraPort
;Do not capture a remote participant merely because their video track is technically available. The person asking the question and the person shown in a frame may have different consent rights.
Tencent RTC’s social entertainment scenarios include AI virtual companions and character dialogue, which is useful context for where this interaction can fit:
Create src/visual-turn.test.ts
:
import assert from 'node:assert/strict';
import test from 'node:test';
import {
VisualTurnCoordinator,
type ModelRequest,
type Snapshot
} from './visual-turn.js';
function deferred<T>() {
let resolve!: (value: T) => void;
const promise = new Promise<T>(r => { resolve = r; });
return { promise, resolve };
}
const frame: Snapshot = {
id: 'frame-1',
capturedAt: 1_000,
mimeType: 'image/jpeg',
bytes: new Uint8Array([1, 2, 3])
};
test('withdrawal discards a late model answer', async () => {
const pending = deferred<string>();
const spoken: string[] = [];
let request: ModelRequest | undefined;
const coordinator = new VisualTurnCoordinator(
{ captureOneFrame: async () => frame },
{
generate: async input => {
request = input;
return pending.promise;
}
},
{ approve: async () => true },
{
status: () => undefined,
speakApprovedText: text => spoken.push(text)
},
() => 1_100
);
await coordinator.shareOneFrame();
const turn = coordinator.onFinalTranscript('What is this?');
coordinator.stopSharing();
pending.resolve('A late visual answer');
await turn;
assert.equal(request?.signal.aborted, true);
assert.deepEqual(spoken, []);
});
test('a frame can be consumed only once', async () => {
const requests: ModelRequest[] = [];
const statuses: string[] = [];
const coordinator = new VisualTurnCoordinator(
{ captureOneFrame: async () => frame },
{
generate: async input => {
requests.push(input);
return 'Approved answer';
}
},
{ approve: async () => true },
{
status: text => statuses.push(text),
speakApprovedText: () => undefined
},
() => 1_100
);
await coordinator.shareOneFrame();
await coordinator.onFinalTranscript('What is this?');
await coordinator.onFinalTranscript('What about this?');
assert.equal(requests.length, 1);
assert.equal(requests[0]?.image?.mimeType, 'image/jpeg');
assert.equal(
statuses.at(-1),
'I do not have a current frame. Share one frame, or describe it aloud.'
);
});
Run the checks:
npm run check
npm test
These tests deliberately use a model fake that ignores cancellation. That matters because aborting a local request does not guarantee that every upstream service immediately stops computing. The epoch and turn checks still prevent the late result from reaching speech synthesis.
Expected behavior: the epoch has changed, so the late frame is discarded. The UI must not return to “ready.”
Also inspect browser memory and temporary object URLs. Application state can forget an image while a UI or upload helper still retains a copy.
Expected behavior: surface a recoverable visual failure and preserve voice-only use. Do not retry the same bytes repeatedly, and do not remove the image while asking the model to answer as though it saw one.
This is also a deployment check: model names and capabilities belong in configuration, but the application must verify the behavior of the exact route before enabling the visual button.
There are two reasonable policies:
Do not silently send the transcript as text-only if it includes “this” or “here.” That converts a temporary capture delay into a hallucination opportunity.
Expected behavior: cancel and invalidate the old request. The new frame should not be substituted into the old transcript because the user may have changed both the object and their intended question.
Choose this policy explicitly. For a social or companion experience, a conservative default is to withhold model output rather than route unchecked text into speech synthesis. A spoken response is harder to stop after disclosure than text that has not yet been rendered.
RTC reconnection and model completion are different facts. Keep the model result pending only if your session policy still permits the turn. Otherwise invalidate it. A restored media connection must not automatically resurrect visual consent.
Before enabling the feature outside development, verify all of the following:
Multimodal companions are often presented as a model-selection problem. The more consequential decision is who controls observation.
A larger model cannot decide whether an old frame is socially appropriate, whether another person consented to appear, or whether “look at this” should authorize another minute of video. Those are application and human decisions.
One-shot context gives up some fluidity. Users may need to tap twice or recapture an object. In return, the system gains a reproducible input, a bounded consent event, a clear failure mode, and an honest way to say, “I cannot currently see that.” For many voice companion tasks, that is a better starting point than an always-open camera.
Relationship disclosure: I’m connected with Tencent RTC, and I used the official Tencent RTC documentation linked above as the implementation reference for this article.