Stop Recommending Songs. Start Understanding Intent. AI systems and generative audio engineer Jack Meng outlined a five-stage architecture for music recommendation that replaces click-prediction with an LLM-driven pipeline: query understanding, heterogeneous multimodal retrieval, reciprocal rank fusion, reranking, and an agentic explainer. The design combines text, audio (CLAP/VGGish), collaborative, and visual embeddings in vector indexes, merged via weighted reciprocal rank fusion, so vague prompts like "rain, neon, and a heartbeat" are parsed into structured intent such as mood, tempo range, and instruments. Why the next generation of music recommendation systems is multimodal, agentic, and no longer “click-prediction” By Jack Meng · AI Systems & Generative Audio Engineer For a decade, music recommendation looked like this: • Log user plays • Train a two-tower model • Retrieve similar songs • Sort by CTR • Call it “personalization” It worked. But it was shallow. Because users never actually say: “Recommend me three tracks with a 124 BPM groove, minor-key melancholy, 2010s R&B texture, and a little tape saturation.” They say: “Something for late-night driving. Not too sad. A bit nostalgic.” That sentence has no tag, no genre, no clean label. And traditional recommender systems choke on it. The future of music discovery is not a better classifier. It’s a system that can hear, read, reason, and generate. A modern music recommendation system is no longer one model. It’s a pipeline. User Intent │ ▼ ┌──────────────────────────────┐ │ 1. Query Understanding LLM │ ← rewrite vague prompts └──────────────┬───────────────┘ ▼ ┌────────────────────────────────────────────┐ │ 2. Multi-Modal Retrieval │ │ - Collaborative embeddings BPR / Two-Tower │ - Text embeddings lyrics, tags, mood │ - Audio embeddings CLAP / VGGish │ - Visual embeddings album art │ - Lexical search BM25 └──────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────┐ │ 3. Fusion & Candidate Pooling RRF / Router │ └──────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────┐ │ 4. Reranking │ │ - Transformer / LambdaMART │ - Diversity constraints │ - Skip-rate penalty │ - Session-level coherence └──────────────┬─────────────────────────────┘ ▼ ┌────────────────────────────────────────────┐ │ 5. Agentic Explainer LLM │ │ “Because you liked X, but want calmer now” └────────────────────────────────────────────┘ This is not theoretical. This is basically what top RecSys 2026 teams converged on. Users are vague. The LLM’s job is not to generate music—it’s to translate feeling into structure. Example: “I want rain, neon, and a heartbeat” The intent parser outputs: { "mood": "melancholic", "cinematic" , "tempo range": 80, 110 , "instruments": "synth", "pads", "soft drums" , "avoid": "heavy metal", "hyperpop" , "context": "late night, focus, urban" } Now we have something a retrieval system can use. A song is not a row in a database. It’s: • sound • language • image • culture • memory So we encode it multiple ways. Text tower • Lyrics • Tags • Editorial descriptions • User-generated mood words Model: Qwen3-Embedding, BGE-M3 Audio tower • Mel-spectrogram • Chroma • Tempo / key / timbre Model: CLAP, VGGish, fine-tuned HTS-AT Collaborative tower • User–item interaction graph • BPR loss • Artist / session co-occurrence Visual tower • Album cover semantics • Color palette / aesthetic clustering Then we store all of them in vector indexes: FAISS / Milvus ├── text vectors ├── audio vectors ├── collab vectors └── fused vectors Pure vector search lies. Pure collaborative filtering forgets new music. Pure metadata search can’t feel vibe. So we do heterogeneous retrieval: candidates = set candidates |= bm25 search query text, k=200 candidates |= faiss search text vec, index="text", k=200 candidates |= faiss search audio vec, index="audio", k=200 candidates |= collab neighbors user id, k=200 candidates |= artist expansion query Then merge with Weighted Reciprocal Rank Fusion: def rrf lists, k=60 : scores = {} for lst in lists: for rank, item in enumerate lst : scores item = scores.get item, 0 + 1 / k + rank return sorted scores.items , key=lambda x: -x 1 Simple. Robust. Beats most “fancy” end-to-end models in production. Retrieval gets you “plausible”. Reranking gets you “right”. A real reranker looks at: • click probability • skip probability • completion rate • genre balance • artist repetition penalty • time-of-day fit • novelty vs familiarity tradeoff Sketch: class Reranker nn.Module : def forward self, seq features, item features, context : x = torch.cat seq features, item features, context , dim=-1 score = self.mlp x return score P listen 30s Then apply business rules: if same artist in last 3: score = 0.6 if item novelty high and user explorer: score = 1.15 Precision@10 jumps. Fatigue drops. A recommendation without explanation is a black box. Modern UX needs: “Because you listened to Lowlight last week, but tonight your prompt sounds calmer—so I pulled ambient R&B with softer attack.” That’s an LLM reranker + constrained generation, not free-form hallucination. Rules: • Only talk about retrieved items • No fake artist names • No invented lyrics • Always ground explanation in metadata New song. Zero plays. Old systems panic. New system says: “I don’t know who listens yet, but acoustically this sits between Bonobo and Tycho, lyrically it’s abstract, visually it’s dusk-toned.” So we bootstrap with: • audio embedding • lyric embedding • mood taxonomy • creator-provided brief Cold start becomes a content understanding problem, not a statistics problem. If I were building this today: Backend API : Go low latency, concurrency Realtime Gateway : WebSocket + gRPC ML Inference : Python + PyTorch + ONNX Embeddings : Qwen3 / CLAP / BGE-M3 Vector DB : Milvus or FAISS Cache : Redis session vectors Orchestration : LangGraph / custom agent loop Frontend : Vue 3 + Web Audio preview Target latency: • p95 retrieval: < 40ms • rerank: < 25ms • generation/explanation: < 300ms streaming That’s production-grade, not demo-grade. We’re moving from: “What song should I play next?” to: “Build me a 40-minute sonic environment.” That means the system can: • generate missing tracks • remix existing ones • adapt BPM to heart rate Recommendation is no longer search. It’s intent-driven audio synthesis with guardrails. Lessons I’ve learned shipping AI music systems Don’t fine-tune first. Architect first. Most teams waste months on models before defining the retrieval boundary. LLMs are planners, not oracles. Let the LLM call tools. Don’t let it invent songs. Diversity perfection. A slightly imperfect but varied playlist beats 10 near-duplicates. Implicit feedback lies less than you think. Skips, replays, and abandon-time are gold. Generative + retrieval hybrid is the endgame. Pure generation hallucinates. Pure retrieval stagnates. Final thought The next Spotify-killer won’t be a better playlist algorithm. It’ll be a system that understands: • what you say • what you feel • what you heard yesterday • what you didn’t know you wanted tonight Music recommendation is no longer about predicting the next click. It’s about modeling human atmosphere. And that’s a much harder—and much more beautiful—problem. If you build AI music, generative audio, or recommendation infra: I write about real-world architecture for multimodal ML, agentic systems, and production-grade generative audio. No hype. Just pipes, vectors, and signal. Tags: MusicAI RecSys MultimodalML LLM AgenticSystems PyTorch VectorSearch GenerativeAudio