This isn't a quick weekend build. It's a multi-month deployment involving a browser extension for data collection, a backend for storage, and ML pipelines for analysis. The core goal is to capture "implicit signals"—the things I do without thinking, like skipping a track after four seconds or looping a specific chorus—and translate those into a behavioral profile.
The Data Pipeline Architecture #
To make this work, the system has to track the lifecycle of a single song from the moment I click it to the moment it influences a playlist.
- Data Acquisition (Browser Extension)
A content script monitors the YouTube video element. Instead of just tracking "views," it acts as a stenographer, recording timeupdate
, ``
, seeked
, and ended
events. This allows the system to calculate the actual watch percentage and identify specific segments I find appealing.
- Ingestion (Event API)
To avoid over the server, the extension batches these observations into JSON payloads and POSTs them to a FastAPI backend:
{
"video_id": "xyz123",
"title": "Artist - Song Name",
"channel": "OfficialChannel",
"watch_segments": [[0, 41], [132, 222]],
"timestamp": "2023-10-27T21:47:00Z",
"session_id": "sess_98765"
}
- Storage (PostgreSQL)
The raw events are written directly to PostgreSQL. I keep the raw signal intact because any future changes to the ML logic will require re-processing the original data.
- Processing (Offline ML Pipeline)
This is where the "forensic" work happens. A nightly pipeline processes the raw events to:
Classify: Confirm if the video is actually music.Resolve: Parse titles and use external lookups for clean metadata.Analyze: Use audio embeddings and mood tagging to determine the "vibe."Score: Convert watch percentages and replay counts into implicit ratings.
Indexing (Vector Store)
The final step involves pushing audio embeddings into
pgvector
. By storing these as vectors alongside relational metadata, the system can perform similarity searches to find songs that match the specific "fingerprint" of my late-night synthwave sessions or morning lo-fi habits.This setup transforms a messy history page into a structured AI workflow, allowing for a level of personalization that generic algorithms usually miss.
Next Swarmvault: A Lean LLM Agent Workflow →