cd /news/artificial-intelligence/stop-recommending-songs-start-unders… Β· home β€Ί topics β€Ί artificial-intelligence β€Ί article
[ARTICLE Β· art-171966] src=dev.to β†— pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Stop Recommending Songs. Start Understanding Intent.

AI systems and generative audio engineer Jack Meng outlined a five-stage architecture for music recommendation that replaces click-prediction with an LLM-driven pipeline: query understanding, heterogeneous multimodal retrieval, reciprocal rank fusion, reranking, and an agentic explainer. The design combines text, audio (CLAP/VGGish), collaborative, and visual embeddings in vector indexes, merged via weighted reciprocal rank fusion, so vague prompts like "rain, neon, and a heartbeat" are parsed into structured intent such as mood, tempo range, and instruments.

recommendation

For vague natural-language queries, parse intent into structured fields with an LLM, retrieve across text, audio, and collaborative indexes, then fuse with reciprocal rank fusion instead of trusting one vector search.

more in playbook: Retrieval and RAG
by read5 min views3 publishedNov 18, 2026

Why the next generation of music recommendation systems is multimodal, agentic, and no longer β€œclick-prediction”

By Jack Meng Β· AI Systems & Generative Audio Engineer

For a decade, music recommendation looked like this: β€’ Log user plays

β€’ Train a two-tower model

β€’ Retrieve similar songs

β€’ Sort by CTR

β€’ Call it β€œpersonalization”

It worked. But it was shallow.

Because users never actually say:

β€œRecommend me three tracks with a 124 BPM groove, minor-key melancholy, 2010s R&B texture, and a little tape saturation.”

They say:

β€œSomething for late-night driving. Not too sad. A bit nostalgic.”

That sentence has no tag, no genre, no clean label.

And traditional recommender systems choke on it.

The future of music discovery is not a better classifier.

It’s a system that can hear, read, reason, and generate.

A modern music recommendation system is no longer one model.

It’s a pipeline.

User Intent

β”‚

β–Ό

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”

β”‚ 1. Query Understanding (LLM) β”‚ ← rewrite vague prompts

β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

           β–Ό

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”

β”‚ 2. Multi-Modal Retrieval β”‚

β”‚  - Collaborative embeddings (BPR / Two-Tower)

β”‚  - Text embeddings (lyrics, tags, mood)

β”‚  - Audio embeddings (CLAP / VGGish)

β”‚  - Visual embeddings (album art)

β”‚  - Lexical search (BM25)

β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

           β–Ό

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”

β”‚ 3. Fusion & Candidate Pooling (RRF / Router)β”‚

β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

           β–Ό

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”

β”‚ 4. Reranking β”‚

β”‚ - Transformer / LambdaMART

β”‚ - Diversity constraints

β”‚  - Skip-rate penalty

β”‚  - Session-level coherence

β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

           β–Ό

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”

β”‚ 5. Agentic Explainer (LLM) β”‚

β”‚ β€œBecause you liked X, but want calmer now”

β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

This is not theoretical.

This is basically what top RecSys 2026 teams converged on.

Users are vague. The LLM’s job is not to generate musicβ€”it’s to translate feeling into structure.

Example:

β€œI want rain, neon, and a heartbeat”

The intent parser outputs:

{

  "mood": ["melancholic", "cinematic"],

  "tempo_range": [80, 110],

  "instruments": ["synth", "pads", "soft drums"],

  "avoid": ["heavy metal", "hyperpop"],

  "context": "late night, focus, urban"

}

Now we have something a retrieval system can use.

A song is not a row in a database.

It’s:

β€’ sound

β€’ language

β€’ image

β€’ culture

β€’ memory

So we encode it multiple ways.

Text tower

β€’ Lyrics

β€’ Tags

β€’ Editorial descriptions

β€’ User-generated mood words

Model: Qwen3-Embedding, BGE-M3 Audio tower

β€’ Mel-spectrogram

β€’ Chroma

β€’ Tempo / key / timbre

Model: CLAP, VGGish, fine-tuned HTS-AT Collaborative tower

β€’ User–item interaction graph

β€’ BPR loss

β€’ Artist / session co-occurrence

Visual tower

β€’ Album cover semantics

β€’ Color palette / aesthetic clustering

Then we store all of them in vector indexes:

FAISS / Milvus

β”œβ”€β”€ text_vectors

β”œβ”€β”€ audio_vectors

β”œβ”€β”€ collab_vectors

└── fused_vectors

Pure vector search lies.

Pure collaborative filtering forgets new music.

Pure metadata search can’t feel vibe.

So we do heterogeneous retrieval:

candidates = set()

candidates |= bm25_search(query_text, k=200)

candidates |= faiss_search(text_vec, index="text", k=200)

candidates |= faiss_search(audio_vec, index="audio", k=200)

candidates |= collab_neighbors(user_id, k=200)

candidates |= artist_expansion(query)

Then merge with Weighted Reciprocal Rank Fusion:

def rrf(lists, k=60):

    scores = {}

    for lst in lists:

        for rank, item in enumerate(lst):

            scores[item] = scores.get(item, 0) + 1 / (k + rank)

    return sorted(scores.items(), key=lambda x: -x[1])

Simple. Robust. Beats most β€œfancy” end-to-end models in production.

Retrieval gets you β€œplausible”.

Reranking gets you β€œright”.

A real reranker looks at:

β€’ click probability

β€’ skip probability

β€’ completion rate

β€’ genre balance

β€’ artist repetition penalty

β€’ time-of-day fit β€’ novelty vs familiarity tradeoff

Sketch:

class Reranker(nn.Module):

    def forward(self, seq_features, item_features, context):

        x = torch.cat([seq_features, item_features, context], dim=-1)

        score = self.mlp(x)

        return score  # P(listen > 30s)

Then apply business rules:

if same_artist_in_last_3:

    score *= 0.6

if item_novelty_high and user_explorer:

    score *= 1.15

Precision@10 jumps. Fatigue drops.

A recommendation without explanation is a black box.

Modern UX needs:

β€œBecause you listened to Lowlight last week, but tonight your prompt sounds calmerβ€”so I pulled ambient R&B with softer attack.”

That’s an LLM reranker + constrained generation, not free-form hallucination.

Rules:

β€’ Only talk about retrieved items

β€’ No fake artist names

β€’ No invented lyrics

β€’ Always ground explanation in metadata

New song. Zero plays. Old systems panic.

New system says: β€œI don’t know who listens yet, but acoustically this sits between Bonobo and Tycho, lyrically it’s abstract, visually it’s dusk-toned.”

So we bootstrap with:

β€’ audio embedding

β€’ lyric embedding

β€’ mood taxonomy

β€’ creator-provided brief

Cold start becomes a content understanding problem, not a statistics problem.

If I were building this today:

Backend API        : Go (low latency, concurrency)

Realtime Gateway : WebSocket + gRPC

ML Inference : Python + PyTorch + ONNX

Embeddings : Qwen3 / CLAP / BGE-M3

Vector DB : Milvus or FAISS

Cache : Redis (session vectors) Orchestration : LangGraph / custom agent loop

Frontend : Vue 3 + Web Audio preview

Target latency:

β€’ p95 retrieval: < 40ms

β€’ rerank: < 25ms

β€’ generation/explanation: < 300ms streaming

That’s production-grade, not demo-grade.

We’re moving from:

β€œWhat song should I play next?”

to:

β€œBuild me a 40-minute sonic environment.”

That means the system can:

β€’ generate missing tracks

β€’ remix existing ones

β€’ adapt BPM to heart rate

Recommendation is no longer search.

It’s intent-driven audio synthesis with guardrails.

Lessons I’ve learned shipping AI music systems

Don’t fine-tune first. Architect first.

Most teams waste months on models before defining the retrieval boundary.

LLMs are planners, not oracles.

Let the LLM call tools. Don’t let it invent songs. Diversity > perfection.

A slightly imperfect but varied playlist beats 10 near-duplicates.

Implicit feedback lies less than you think.

Skips, replays, and abandon-time are gold.

Generative + retrieval hybrid is the endgame.

Pure generation hallucinates. Pure retrieval stagnates.

Final thought The next Spotify-killer won’t be a better playlist algorithm.

It’ll be a system that understands:

β€’ what you say

β€’ what you feel

β€’ what you heard yesterday

β€’ what you didn’t know you wanted tonight

Music recommendation is no longer about predicting the next click.

It’s about modeling human atmosphere.

And that’s a much harderβ€”and much more beautifulβ€”problem.

If you build AI music, generative audio, or recommendation infra: I write about real-world architecture for multimodal ML, agentic systems, and production-grade generative audio.

No hype. Just pipes, vectors, and signal.

Tags: #MusicAI #RecSys #MultimodalML #LLM #AgenticSystems #PyTorch #VectorSearch #GenerativeAudio

── more in #artificial-intelligence 4 stories Β· sorted by recency
── more on @jack meng 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/stop-recommending-so…] indexed:0 read:5min 2026-11-18 Β· β€”