This is a simplified guide to an AI model called All-In-One-Music-Structure-Analysis maintained by Cwalo. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
all-in-one-music-structure-analysis
is a unified deep learning model that analyzes music audio to predict tempo (BPM), beat positions, downbeat positions, functional segment boundaries, segment labels (intro, verse, chorus, bridge, outro), and optionally demuxes audio into separate instrument stems. Built by cwalo, this model processes source-separated spectrograms using dilated neighborhood attention layers to capture long-term temporal dependencies and non-dilated attention for local instrumental relationships. The architecture achieves state-of-the-art performance on the Harmonix Set benchmark across all four simultaneous tasks while maintaining relatively low parameter count. The model benefits from multi-task learning where beat, downbeat, and segment prediction tasks improve each other's accuracy. It runs on Replicate with GPU support (T4 recommended) and accepts audio files as input, returning structured predictions and optional demuxed audio stems, visualizations, sonifications, embeddings, and activation maps.
Music production workflow automation: Use this model to automatically analyze song structure and generate detailed breakdowns of tempo, beat grid, and section labels. This is particularly valuable when working with recordings where you need instant, objective measures of BPM stability, downbeat placement, and section timing to align drum programming, synth arrangements, or video edits to the song's natural structure.
Stem separation and remixing pipelines: The demux feature enables automatic separation of drums, bass, vocals, and other instruments from a stereo mix. This supports remix workflows, stem export for sample packs, or feeding isolated instruments into other production tools. The simultaneous structure analysis helps align separated stems to the original timing grid.
Music library metadata generation: For large music collections without structured metadata, this model generates consistent, standardized data across tracks—BPM, segment boundaries, and functional labels. This is useful for DJs building playlists programmatically, music streaming services enriching metadata, or researchers building music corpora with reliable structural annotations.
Musicological research and analysis: The model outputs frame-level embeddings and raw activations (via the embed
and activ
options), enabling computational musicology studies. Researchers can extract learned representations of musical structure, analyze how different song sections cluster in embedding space, or study the model's internal decision-making through activation patterns across beats, downbeats, and functional segments.
Real-time music visualization and synchronization: The model produces sonifications (audio overlays of detected beats/downbeats) and visualizations showing the temporal structure. These outputs support live visualization tools, VJ software, or educational interfaces that teach musical structure by highlighting where beats and sections occur within the original audio.
The model requires GPU resources for reasonable inference speed; CPU-only execution is possible but slow for longer files. Input audio must be in a format compatible with standard audio libraries and file size affects processing time. The model is trained on the Harmonix Set, a specific music annotation dataset, so performance may degrade on musical styles, tempos, or structures underrepresented in that dataset. Real-time inference is not feasible—this is an offline analysis tool.
The demux functionality is not a full-featured source separation engine; it provides basic drum/bass/vocal/other separation but lacks the quality and flexibility of specialized stem-splitting models like those in all-in-one-audio. Segment labeling predicts a fixed set of functional categories (intro, verse, chorus, bridge, outro) and cannot extend to custom segment types. The output format is URI-based file lists rather than structured JSON with precise numeric timestamps, requiring post-processing to extract exact beat times and segment boundaries for programmatic use. Functional segment boundaries are frame-level predictions that must be converted to time via the model's frame rate.
The model does not handle polyphonic audio perfectly for all music styles, particularly when musical content conflicts with expected structural patterns. Very long audio files may encounter memory constraints or timeout limits on Replicate. There is no explicit parameter for adjusting model confidence thresholds or post-processing strictness, limiting fine-tuning for domain-specific needs.
all-in-one-music-structure-analyzer by sakemin is an earlier Cog implementation of the same underlying research (mir-aidj's All-In-One model). Choose the cwalo
version (this model) for the latest updates and potential improvements; the sakemin
version may have older weights or less active maintenance. Both offer identical functionality, so the decision hinges on which maintainer you trust for ongoing support.
all-in-one-audio by erickluis00 combines structure analysis with more sophisticated stem splitting using Demucs and MDX-Net. Choose all-in-one-audio
if stem separation quality is your primary need and you can tolerate a less unified architecture; choose this model if you want tightly integrated multi-task learning where structure and demux predictions inform each other. The all-in-one-audio
approach is more modular but loses the accuracy gains from joint training.
sota-music-tagging-models by minzwon predicts semantic tags (genre, mood, instruments) rather than structural properties. Use sota-music-tagging-models
for understanding what a song is; use this model for understanding how a song is organized.
music-2.6 by minimax generates original music from text prompts. These tools are complementary—generate music with this model, then analyze its structure with all-in-one-music-structure-analysis
.
audiosep by cjwbw separates arbitrary described audio sources ("acoustic guitar," "traffic noise") via text conditioning. Use audiosep
for fine-grained separation of instrument categories you specify; use this model for fast, fixed-category demuxing integrated with structural analysis.
The model employs a unified attention-based architecture combining dilated neighborhood attention (for long-term temporal patterns) and standard attention (for local instrumental relationships). It processes source-separated spectrograms as input features, enabling the model to learn instrument-specific beat and structure patterns.
The architecture is designed for multi-task learning: simultaneously predicting beats (temporal positions of regular pulses), downbeats (the strongest beat in a measure), functional segment boundaries (where sections change), and segment labels (semantic category of each section). This joint training improves all four predictions compared to single-task models.
Training occurred on the Harmonix Set, a dataset of annotated music recordings with beat, downbeat, and functional segment labels. The model achieves state-of-the-art F-measure performance on all four tasks while maintaining lower parameter count than competing approaches.
The model supports two pretrained variants accessible via the model
parameter:
harmonix-all
(default): General-purpose model trained on Harmonix Set dataInput audio is converted to mel-spectrograms, optionally source-separated into stems, and processed at a frame rate determined by the STFT window parameters (typically 50ms frame hops, yielding 20 frames per second, though exact rate depends on the specific model weight configuration).
Outputs are generated at frame-level resolution, meaning segment boundaries and beat positions are initially predicted at frame indices rather than exact millisecond timestamps. Users must multiply frame indices by the frame duration (typically 50ms) to recover times in seconds.
The model can optionally output:
activ
option)embed
option)demux
option)Latest version updated 2024-10-28 and packaged with Cog 0.11.6 for containerization.
import replicate
client = replicate.Replicate(api_token="your-replicate-api-token")
output = client.run(
"cwalo/all-in-one-music-structure-analysis:6deeba047db17da69e9826c0285cd137cd2a81af05eb44ff496b7acd69b3a383",
input={
"music_input": "https://example.com/song.wav",
"visualize": True,
"sonify": False,
"demux": True,
"model": "harmonix-all",
"include_embeddings": False,
"include_activations": False,
}
)
for file_uri in output:
print(file_uri)
The example above runs the model on a remote audio file, requests visualizations and demuxed stems, and prints the resulting file URIs. Omit optional parameters to use defaults. You can also pass local files by up them first or using file:// URIs if running locally with Cog.
Q: What does demux output, and how good is it compared to dedicated stem separation tools?
A: The demux option separates audio into four stems: drums, bass, vocals, and other instruments. This functionality is built into the model but is less specialized than dedicated source separation tools like Demucs. Use demux for quick, integrated stem separation during structure analysis; use dedicated models if stem quality is critical.
Q: What format are the beat and downbeat predictions in the output files?
A: Outputs are returned as URIs pointing to files. The exact format (JSON, CSV, text) depends on the model implementation. Beat and downbeat predictions are frame-indexed, meaning you multiply the frame index by the frame duration (typically 50ms) to get time in seconds. You will need to parse the output files to extract structured timing data.
Q: Can I use this model commercially?
A: The model is licensed under the terms specified in the GitHub repository. Check the LICENSE file at the model's GitHub repo for commercial use restrictions. The underlying research is published and the model weights are available on Replicate, but verify licensing compliance for your specific use case.
Q: Which segment labels can the model predict?
A: The model predicts functional segment labels from a fixed set: intro, verse, chorus, bridge, and outro. It cannot be extended to custom categories without retraining. If you need different segment types, you must classify the model's outputs post-hoc or use a different labeling system.
Q: How does this model compare to alternatives for production use?
A: This is a research-oriented model balancing accuracy across four simultaneous tasks (beats, downbeats, segments, labels) rather than excelling at any single task. For production music work, consider whether you need all four outputs or could benefit from specialized tools: dedicated beat trackers for timing-critical work, or dedicated stem splitters for remix workflows. The integrated approach trades single-task performance for unified structure understanding.
Q: What happens if the model fails to detect clear beats or segments?
A: The model works best on clear, rhythmically regular music (pop, electronic, hip-hop). It may fail or produce low-confidence predictions on ambient, avant-garde, or highly syncopated music where traditional beat/downbeat concepts do not apply cleanly. Check the confidence outputs or activation values to assess reliability on edge cases.
Q: Is the model actively maintained?
A: The latest version on Replicate was updated in October 2024. The underlying research is published and the model is stable, but check the GitHub repository and maintainer profile for ongoing updates or bug fixes.
Q: What GPU and runtime requirements does this have?
A: The README recommends T4 GPU for Replicate deployment. Inference time depends on audio length; longer files take proportionally longer. CPU-only execution is possible but significantly slower. For files longer than 10–15 minutes, expect extended processing times even on GPU.
Click here to read the full guide to All-In-One-Music-Structure-Analysis