{"slug": "stop-recommending-songs-start-understanding-intent", "title": "Stop Recommending Songs. Start Understanding Intent.", "summary": "AI systems and generative audio engineer Jack Meng outlined a five-stage architecture for music recommendation that replaces click-prediction with an LLM-driven pipeline: query understanding, heterogeneous multimodal retrieval, reciprocal rank fusion, reranking, and an agentic explainer. The design combines text, audio (CLAP/VGGish), collaborative, and visual embeddings in vector indexes, merged via weighted reciprocal rank fusion, so vague prompts like \"rain, neon, and a heartbeat\" are parsed into structured intent such as mood, tempo range, and instruments.", "body_md": "Why the next generation of music recommendation systems is multimodal, agentic, and no longer “click-prediction”\n\nBy Jack Meng · AI Systems & Generative Audio Engineer\n\nFor a decade, music recommendation looked like this:\n\n• Log user plays\n\n• Train a two-tower model\n\n• Retrieve similar songs\n\n• Sort by CTR\n\n• Call it “personalization”\n\nIt worked. But it was shallow.\n\nBecause users never actually say:\n\n“Recommend me three tracks with a 124 BPM groove, minor-key melancholy, 2010s R&B texture, and a little tape saturation.”\n\nThey say:\n\n“Something for late-night driving. Not too sad. A bit nostalgic.”\n\nThat sentence has no tag, no genre, no clean label.\n\nAnd traditional recommender systems choke on it.\n\nThe future of music discovery is not a better classifier.\n\nIt’s a system that can hear, read, reason, and generate.\n\nA modern music recommendation system is no longer one model.\n\nIt’s a pipeline.\n\nUser Intent\n\n   │\n\n   ▼\n\n┌──────────────────────────────┐\n\n│ 1. Query Understanding (LLM) │  ← rewrite vague prompts\n\n└──────────────┬───────────────┘\n\n               ▼\n\n┌────────────────────────────────────────────┐\n\n│ 2. Multi-Modal Retrieval                    │\n\n│  - Collaborative embeddings (BPR / Two-Tower)\n\n│  - Text embeddings (lyrics, tags, mood)\n\n│  - Audio embeddings (CLAP / VGGish)\n\n│  - Visual embeddings (album art)\n\n│  - Lexical search (BM25)\n\n└──────────────┬─────────────────────────────┘\n\n               ▼\n\n┌────────────────────────────────────────────┐\n\n│ 3. Fusion & Candidate Pooling (RRF / Router)│\n\n└──────────────┬─────────────────────────────┘\n\n               ▼\n\n┌────────────────────────────────────────────┐\n\n│ 4. Reranking                                │\n\n│  - Transformer / LambdaMART\n\n│  - Diversity constraints\n\n│  - Skip-rate penalty\n\n│  - Session-level coherence\n\n└──────────────┬─────────────────────────────┘\n\n               ▼\n\n┌────────────────────────────────────────────┐\n\n│ 5. Agentic Explainer (LLM)                  │\n\n│  “Because you liked X, but want calmer now”\n\n└────────────────────────────────────────────┘\n\nThis is not theoretical.\n\nThis is basically what top RecSys 2026 teams converged on. \n\nUsers are vague. The LLM’s job is not to generate music—it’s to translate feeling into structure.\n\nExample:\n\n“I want rain, neon, and a heartbeat”\n\nThe intent parser outputs:\n\n{\n\n  \"mood\": [\"melancholic\", \"cinematic\"],\n\n  \"tempo_range\": [80, 110],\n\n  \"instruments\": [\"synth\", \"pads\", \"soft drums\"],\n\n  \"avoid\": [\"heavy metal\", \"hyperpop\"],\n\n  \"context\": \"late night, focus, urban\"\n\n}\n\nNow we have something a retrieval system can use.\n\nA song is not a row in a database.\n\nIt’s:\n\n• sound\n\n• language\n\n• image\n\n• culture\n\n• memory\n\nSo we encode it multiple ways.\n\nText tower\n\n• Lyrics\n\n• Tags\n\n• Editorial descriptions\n\n• User-generated mood words\n\nModel: Qwen3-Embedding, BGE-M3\n\nAudio tower\n\n• Mel-spectrogram\n\n• Chroma\n\n• Tempo / key / timbre\n\nModel: CLAP, VGGish, fine-tuned HTS-AT\n\nCollaborative tower\n\n• User–item interaction graph\n\n• BPR loss\n\n• Artist / session co-occurrence\n\nVisual tower\n\n• Album cover semantics\n\n• Color palette / aesthetic clustering\n\nThen we store all of them in vector indexes:\n\nFAISS / Milvus\n\n├── text_vectors\n\n├── audio_vectors\n\n├── collab_vectors\n\n└── fused_vectors\n\nPure vector search lies.\n\nPure collaborative filtering forgets new music.\n\nPure metadata search can’t feel vibe.\n\nSo we do heterogeneous retrieval:\n\ncandidates = set()\n\ncandidates |= bm25_search(query_text, k=200)\n\ncandidates |= faiss_search(text_vec, index=\"text\", k=200)\n\ncandidates |= faiss_search(audio_vec, index=\"audio\", k=200)\n\ncandidates |= collab_neighbors(user_id, k=200)\n\ncandidates |= artist_expansion(query)\n\nThen merge with Weighted Reciprocal Rank Fusion:\n\ndef rrf(lists, k=60):\n\n    scores = {}\n\n    for lst in lists:\n\n        for rank, item in enumerate(lst):\n\n            scores[item] = scores.get(item, 0) + 1 / (k + rank)\n\n    return sorted(scores.items(), key=lambda x: -x[1])\n\nSimple. Robust. Beats most “fancy” end-to-end models in production.\n\nRetrieval gets you “plausible”.\n\nReranking gets you “right”.\n\nA real reranker looks at:\n\n• click probability\n\n• skip probability\n\n• completion rate\n\n• genre balance\n\n• artist repetition penalty\n\n• time-of-day fit\n\n• novelty vs familiarity tradeoff\n\nSketch:\n\nclass Reranker(nn.Module):\n\n    def forward(self, seq_features, item_features, context):\n\n        x = torch.cat([seq_features, item_features, context], dim=-1)\n\n        score = self.mlp(x)\n\n        return score  # P(listen > 30s)\n\nThen apply business rules:\n\nif same_artist_in_last_3:\n\n    score *= 0.6\n\nif item_novelty_high and user_explorer:\n\n    score *= 1.15\n\nPrecision@10 jumps. Fatigue drops.\n\nA recommendation without explanation is a black box.\n\nModern UX needs:\n\n“Because you listened to Lowlight last week, but tonight your prompt sounds calmer—so I pulled ambient R&B with softer attack.”\n\nThat’s an LLM reranker + constrained generation, not free-form hallucination.\n\nRules:\n\n• Only talk about retrieved items\n\n• No fake artist names\n\n• No invented lyrics\n\n• Always ground explanation in metadata\n\nNew song. Zero plays.\n\nOld systems panic.\n\nNew system says:\n\n“I don’t know who listens yet, but acoustically this sits between Bonobo and Tycho, lyrically it’s abstract, visually it’s dusk-toned.”\n\nSo we bootstrap with:\n\n• audio embedding\n\n• lyric embedding\n\n• mood taxonomy\n\n• creator-provided brief\n\nCold start becomes a content understanding problem, not a statistics problem.\n\nIf I were building this today:\n\nBackend API        : Go (low latency, concurrency)\n\nRealtime Gateway   : WebSocket + gRPC\n\nML Inference       : Python + PyTorch + ONNX\n\nEmbeddings         : Qwen3 / CLAP / BGE-M3\n\nVector DB          : Milvus or FAISS\n\nCache              : Redis (session vectors)\n\nOrchestration      : LangGraph / custom agent loop\n\nFrontend           : Vue 3 + Web Audio preview\n\nTarget latency:\n\n• p95 retrieval: < 40ms\n\n• rerank: < 25ms\n\n• generation/explanation: < 300ms streaming\n\nThat’s production-grade, not demo-grade.\n\nWe’re moving from:\n\n“What song should I play next?”\n\nto:\n\n“Build me a 40-minute sonic environment.”\n\nThat means the system can:\n\n• generate missing tracks\n\n• remix existing ones\n\n• adapt BPM to heart rate\n\nRecommendation is no longer search.\n\nIt’s intent-driven audio synthesis with guardrails.\n\nLessons I’ve learned shipping AI music systems\n\nDon’t fine-tune first. Architect first.\n\nMost teams waste months on models before defining the retrieval boundary.\n\nLLMs are planners, not oracles.\n\nLet the LLM call tools. Don’t let it invent songs.\n\nDiversity > perfection.\n\nA slightly imperfect but varied playlist beats 10 near-duplicates.\n\nImplicit feedback lies less than you think.\n\nSkips, replays, and abandon-time are gold.\n\nGenerative + retrieval hybrid is the endgame.\n\nPure generation hallucinates. Pure retrieval stagnates.\n\nFinal thought\n\nThe next Spotify-killer won’t be a better playlist algorithm.\n\nIt’ll be a system that understands:\n\n• what you say\n\n• what you feel\n\n• what you heard yesterday\n\n• what you didn’t know you wanted tonight\n\nMusic recommendation is no longer about predicting the next click.\n\nIt’s about modeling human atmosphere.\n\nAnd that’s a much harder—and much more beautiful—problem.\n\nIf you build AI music, generative audio, or recommendation infra:\n\nI write about real-world architecture for multimodal ML, agentic systems, and production-grade generative audio.\n\nNo hype. Just pipes, vectors, and signal.\n\nTags: #MusicAI #RecSys #MultimodalML #LLM #AgenticSystems #PyTorch #VectorSearch #GenerativeAudio", "url": "https://wpnews.pro/news/stop-recommending-songs-start-understanding-intent", "canonical_source": "https://dev.to/jackmeng/stop-recommending-songs-start-understanding-intent-3f8f", "published_at": "2026-11-18 17:32:57+00:00", "updated_at": "2026-11-18 17:42:25.914830+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "natural-language-processing"], "entities": ["Jack Meng", "CLAP", "VGGish", "Qwen3-Embedding", "BGE-M3", "FAISS", "Milvus", "RecSys 2026"], "also_reported_by": [], "takeaway": "Combine BM25, text, audio, collaborative, and artist-expansion retrieval into one candidate pool and merge with weighted reciprocal rank fusion (k=60) rather than relying on a single embedding index.", "recommendation": {"advice": "For vague natural-language queries, parse intent into structured fields with an LLM, retrieve across text, audio, and collaborative indexes, then fuse with reciprocal rank fusion instead of trusting one vector search.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/stop-recommending-songs-start-understanding-intent", "markdown": "https://wpnews.pro/news/stop-recommending-songs-start-understanding-intent.md", "text": "https://wpnews.pro/news/stop-recommending-songs-start-understanding-intent.txt", "jsonld": "https://wpnews.pro/news/stop-recommending-songs-start-understanding-intent.jsonld"}}