From a 15-Second Walk to a Digital Memory: Building a Canine Cognitive Twin

A developer built Canine Cognitive Twin, a digital twin system for dogs that captures 15-second videos of walks and uses Gemini 3 Pro for analysis, ElevenLabs for text-to-speech, Snowflake for data processing and anomaly detection, and Solana Devnet for immutable record-keeping. The project is currently in development with synthetic history data for demonstration, and the developer emphasizes transparency about what is implemented versus simulated.

This is a submission for Weekend Challenge: Dog Days Edition Canine Cognitive Twin — a living digital twin for a dog. A phone captures fifteen seconds of a morning walk. Gemini analyzes the video and returns structured observations plus first-person narration. ElevenLabs turns that narration into speech. Snowflake Dynamic Tables build a rolling baseline, Cortex flags potential gait drift, and Solana records each chapter under the owner's own keys. One capture becomes one chapter of a memoir the dog "writes" in its own voice — while also creating a longitudinal record that remains under the owner's control. Not a veterinary device. Not diagnostic. Consult a veterinarian. There is currently no live demo video or deployed demo available. Rather than presenting a simulated walkthrough as a live demo, I want to be transparent about what has been implemented and what is currently synthetic. The project is structured so the complete pipeline can be demonstrated end-to-end once the live demo environment is ready. | Step | Technology | Purpose | |---|---|---| | Capture 15s video + audio | Expo | Records the dog's walk | | Video analysis | Gemini 3 Pro | Extracts behavior tags, gait asymmetry, emotion, and narration | | Text-to-speech | ElevenLabs Flash v2.5 | Converts narration into the dog's voice | | Data processing | Snowflake Dynamic Tables | Builds rolling summaries and baselines | | Anomaly detection | Snowflake Cortex | Detects potential gait drift from historical data | | Immutable record | Solana Devnet | Stores the chapter index, content hash, and anomaly flag | | Dashboard | Streamlit in Snowflake | Displays the dog's memoir and health radar | Because anomaly detection requires historical data, the project includes generated history for demonstration and development purposes. | Data | Status | Reason | |---|---|---| | 90 days of gait/vitals history | Synthetic | A single capture is not enough to establish a meaningful baseline | | Injected gait drift around day 60 | Synthetic | Used to demonstrate anomaly detection | | Voice Design voice | Pre-configured | Created once and reused for the dog's persona | | Sound effects | Pre-rendered | Sniffing, panting, collar jingles, and environmental sounds are prepared ahead of time | | Gemini fallback response | Development fallback | Prevents a temporary API/rate-limit failure from blocking the pipeline | Every dashboard panel that uses generated history is explicitly marked with a SYNTHETIC HISTORY badge. There is no intentional blending of synthetic history with live data without disclosure. The end-to-end data flow is: Expo Mobile App │ │ 15s video + audio ▼ Cloudflare Worker │ ├──► Cloudflare R2 │ └── Media storage │ ├──► SHA-256 │ └──► Gemini 3 Pro │ ├── behavior tags ├── gait asymmetry score ├── emotion └── narration │ ▼ ElevenLabs │ ▼ Phone Playback Gemini Result │ ▼ Snowflake RAW EVENTS │ ▼ Dynamic Tables │ ├──► CLEANED EVENTS │ └──► DAILY SUMMARY │ ▼ Snowflake ML Anomaly Detection │ ▼ GAIT ANOMALIES │ ▼ Streamlit-in-Snowflake ├── Memoir └── Health Radar │ ▼ Oracle Relay │ ▼ Solana Devnet record memory Snowflake Dynamic Tables cannot directly sign Solana transactions or call arbitrary external APIs. Instead, a lightweight oracle-relay polls Snowflake's V ORACLE PAYLOAD , signs the transaction, and submits it to Solana Devnet. This keeps blockchain operations outside the request path, so a slow Solana transaction doesn't block the more time-sensitive Gemini + TTS pipeline. The Anchor program still verifies the oracle public key on-chain. The ingestion worker runs inside the Cloudflare Workers runtime, while the oracle relay runs in Node.js with tsx . Although both use keypair-based authentication, their runtime crypto APIs are different. I intentionally kept the implementations separate rather than introducing an abstraction that would only serve two different runtime environments. The live Gemini API can fail because of rate limits or temporary availability issues. Instead of pretending everything is live, the worker can load a pre-recorded response. The API response contains: { "source": "fallback" } The dashboard can then communicate that state instead of hiding it. | Technology | Used For | Why It Matters | |---|---|---| Google AI / Gemini | Multimodal video analysis and structured JSON responses | Provides the actual observation layer | Snowflake | Dynamic Tables, historical baselines, anomaly detection, Streamlit | Turns individual observations into longitudinal data | ElevenLabs | Voice Design and Flash v2.5 narration | Gives the digital twin its personality and voice | Solana | Dog identity and immutable memory records | Keeps the ownership/proof layer outside a centralized database | These aren't decorative integrations. Each technology has a specific role in the architecture. canine-cognitive-twin/ ├── snowflake/ Dynamic Tables, Cortex anomaly model, Streamlit app ├── gemini-pipeline/ Schema-constrained Gemini analysis ├── elevenlabs-service/ Voice Design, narration, SFX ├── ingestion-worker/ Cloudflare Worker: R2 → Gemini → Snowflake → TTS ├── oracle-relay/ Snowflake → Solana transaction relay ├── solana-program/ Anchor program for dog identity and memories ├── mobile-capture/ Expo capture and upload application └── demo-assets/ Fallback response and demo assets The next milestone is completing and publishing the live end-to-end demo. After that, I would like to build: I'm submitting this project for: The goal wasn't simply to combine four technologies. Each one solves a different part of the problem: observe → understand → remember → prove . There isn't a live demo to link yet, and I don't want to present a pre-recorded or synthetic walkthrough as if it were live. The implementation, architecture, generated-data boundaries, and fallback behavior are documented in the repository so the project can be evaluated honestly. The idea is simple: A phone, fifteen seconds, and a digital memory that belongs to the dog — and ultimately, its owner. Thanks for reading. 🐶