A beginner's guide to the Beat_this model by Xavriley on Replicate

Xavriley and collaborators at CPJKU released Beat_this, a beat and downbeat tracking model from the ISMIR 2024 paper 'Beat This! Accurate Beat Tracking Without DBN Postprocessing'. The model detects precise beat positions without Dynamic Bayesian Network postprocessing, achieving state-of-the-art F1 scores across diverse music genres. It is available on Replicate and suited for music information retrieval, production tools, transcription, and annotation.

This is a simplified guide to an AI model called Beat this maintained by Xavriley. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter. beat this is a beat and downbeat tracking model from the ISMIR 2024 paper "Beat This Accurate Beat Tracking Without DBN Postprocessing" by xavriley https://aimodels.fyi/creators/replicate/xavriley?utm source=devto&utm medium=referral and collaborators at CPJKU. The model detects precise beat positions and downbeat boundaries in audio files without relying on Dynamic Bayesian Network postprocessing, achieving state-of-the-art F1 scores while maintaining generality across diverse music genres. The architecture alternates convolutions with transformers operating either over frequency or time dimensions, and is trained on multiple datasets including solo instruments, pieces with time signature changes, and classical music with high tempo variations. The main model final0 , final1 , final2 weighs approximately 78 MB each, with a smaller variant available at 8.1 MB. The most critical detail before using it: the model achieves good results specifically because it avoids meter and tempo constraints that traditional systems impose, but this means it can still fail on difficult and underrepresented genres and performs worse on continuity metrics compared to methods using postprocessing. Music information retrieval and analysis workflows. If you build music analysis software that needs to segment tracks into beat-aligned sections for tempo detection, structural analysis, or synchronization with other modalities, beat this provides clean beat and downbeat annotations without requiring external postprocessing pipelines. The model outputs precise timestamps suitable for downstream music information retrieval tasks like onset detection or harmonic analysis. Rhythm-aware music production tools. For digital audio workstations, beat detection plugins, or metronome applications, this model provides frame-level accuracy suitable for real-time audio alignment and grid snapping. The two input parameters—constant tempo assumption and DBN postprocessing toggle—let you trade accuracy for continuity depending on whether your source material has stable rhythm or requires smoothing. Music transcription and notation systems. Beat and downbeat detection forms a foundational layer for automatic music transcription pipelines. Feeding the model's output into tempo curve estimation and time signature inference produces richer musical analysis than beat detection alone, particularly useful for music notation software or academic music analysis tools. Dataset annotation and validation. If you maintain collections of annotated music data, this model can automatically generate beat annotations for new tracks, which human annotators can then correct. The availability of multiple trained seeds final0 , final1 , final2 allows you to ensemble predictions for higher confidence annotation. Classical and complex music analysis. The model explicitly handles tempo variations, time signature changes, and solo instruments—scenarios where simpler beat trackers fail. If your music corpus includes Bach, contemporary classical, or experimental music, this model outperforms systems trained only on pop or dance music datasets. The model trains on multiple public datasets but explicitly excludes GTZAN for evaluation fairness; if you run inference on GTZAN files, results will be misleadingly high. The model struggles with difficult and underrepresented genres—the paper acknowledges this limitation directly. Performance on continuity metrics how smoothly beat timing progresses is worse than DBN-postprocessed systems, making it less suitable for applications requiring strictly monotonic beat sequences without jitter. The Replicate API defaults to using DBN postprocessing use dbn: true and constant tempo assumption constant tempo: true , which partially reverses the paper's main contribution of avoiding DBN; disabling use dbn requires setting it to false explicitly. The model uses PyTorch 2.0+, ffmpeg for non-WAV audio, and optional CUDA support—CPU inference is slower and lacks float16 optimization. The input schema accepts only audio as a URI, meaning you must upload files to a publicly accessible URL rather than passing raw bytes. Output format from Replicate's API is undocumented schema shows title: Output with no further specification , so the actual return structure requires inspection during your first API call. The model file size 78 MB for main variants requires downloading checkpoints on first use, which takes time on bandwidth-limited systems. demixing https://aimodels.fyi/models/replicate/demixing-jimothyjohn?utm source=devto&utm medium=referral by jimothyjohn separates instruments and vocals from audio—a different task than beat detection. Choose beat this if you need tempo and rhythm information; choose demixing if you need to isolate individual instruments before processing. all-in-one-music-structure-analysis https://aimodels.fyi/models/replicate/all-in-one-music-structure-analysis-cwalo?utm source=devto&utm medium=referral by cwalo performs comprehensive analysis including BPM, downbeats, and structure in one call. Use this alternative if you need multiple outputs structure, demuxing, BPM simultaneously; choose beat this if you want the most accurate beat and downbeat detection specifically, as it focuses on that single task and achieves state-of-the-art F1 scores. music https://aimodels.fyi/models/replicate/music-elevenlabs?utm source=devto&utm medium=referral by elevenlabs and music-2.6 https://aimodels.fyi/models/replicate/music-26-minimax?utm source=devto&utm medium=referral by minimax are generative models that compose music from prompts—the opposite of beat this , which analyzes existing audio. These are irrelevant if your goal is beat tracking, but useful if you need to create rhythmically structured content. musicgen https://aimodels.fyi/models/replicate/musicgen-charlesmccarthy?utm source=devto&utm medium=referral by charlesmccarthy generates music from text or composition plans, again a generative task rather than analysis. Not comparable to beat this unless you plan to analyze generated music afterward. The model uses a hybrid convolutional-transformer architecture that alternates between frequency-domain and time-domain transformer blocks. Training uses a custom loss function tolerant to small time shifts in beat annotations typically ±70 ms , addressing a core weakness in prior beat tracking systems where perfect frame alignment was unrealistic. The model trains on spectrograms 22 kHz sample rate, monophonic preprocessed from raw audio using pedalboard. Multiple model variants exist: the main final0/1/2 are trained on all datasets except GTZAN with three random seeds; small0/1/2 offer a 10x smaller model 8.1 MB with slight accuracy drops; single final0/1/2 use a single train/validation split; and fold0-7 provide 8-fold cross-validation variants for fair evaluation on datasets used in training. The model supports CPU and GPU inference CUDA recommended , with optional float16 precision for recent GPUs. Inference can be distributed across multiple GPUs using the --touch-first and --skip-existing command-line flags. The Replicate deployment defaults to using DBN postprocessing from madmom, which contradicts the paper's main claim about avoiding DBN—this requires explicit configuration to disable. Input audio supports any format readable by torchaudio with ffmpeg backend; output format is a .beats TSV file compatible with Sonic Visualizer. .beats file with beat and downbeat timestamps true : Assume the source material has constant tempo; use true for pop/electronic music, false for classical or live recordings with tempo drift true : Apply Dynamic Bayesian Network postprocessing for temporal smoothing; set to false to use the raw model output without DBN constraints title: Output with no documented structure; returns beat and downbeat annotations likely as a .beats file or JSON array of timestamps based on the command-line tool behavior python import replicate Initialize client assumes REPLICATE API TOKEN environment variable client = replicate.Replicate Run beat detection on an audio file output = client.run "xavriley/beat this:26142842c6dc94673820f0a9762214fa7109015d966031d8e6137bef4fd14323", input={ "audio": "https://example.com/path/to/your/audio.mp3", "constant tempo": True, "use dbn": True } print output For local Python use without Replicate, install the package and use directly: python from beat this.inference import File2Beats Load model downloads automatically on first run file2beats = File2Beats checkpoint path="final0", device="cuda", dbn=False Get beat and downbeat positions beats, downbeats = file2beats "path/to/audio.mp3" Save to Sonic Visualizer format from beat this.utils import save beat tsv save beat tsv beats, downbeats, "output.beats" Q: Should I disable use dbn when running on Replicate? A: It depends on your use case. The paper's main contribution is achieving state-of-the-art results without DBN, so set use dbn: false if you want the raw model predictions. However, DBN provides smoother, more musically consistent beat sequences at the cost of introducing meter constraints; use use dbn: true if you need temporal continuity or notice jittery output from the raw model. Q: What audio formats does beat this accept? A: The model accepts WAV, MP3, FLAC, and any format that ffmpeg can decode, provided ffmpeg is installed and torchaudio is configured to use it as a backend. For Replicate, pass audio as a URI string rather than raw bytes. Q: How accurate is this model compared to manual annotation? A: The paper reports state-of-the-art F1 scores on the GTZAN test set when using the final0/1/2 models. However, accuracy degrades on difficult and underrepresented genres, and it performs worse on continuity metrics how smoothly beat timing changes compared to DBN-postprocessed systems. For fair evaluation on datasets used in training, use the cross-validation variants fold0-7 or single final variants . Q: Can I use this model commercially? A: Check the LICENSE file in the GitHub repository for the specific open-source license terms. The code is publicly available and the paper is published, but license restrictions may apply depending on your use case. Q: Which model variant should I use: final, small, or fold? A: Use final0 the default for general-purpose beat tracking on unseen music. Use small0/1/2 if model size or inference speed matters more than accuracy—they sacrifice some F1 score for 10x smaller file size. Use fold or single final only for fair research evaluation on datasets that contributed to training data; otherwise results will be biased upward. Q: How long does inference take on CPU vs GPU? A: The README does not specify exact inference times, but indicates GPU is strongly recommended. The command-line tool defaults to GPU with CPU fallback; float16 mode on recent GPUs improves speed. For batch processing, distribute across multiple GPUs with separate process instances. Q: What happens if the audio has tempo changes or unusual time signatures? A: The model is specifically designed to handle tempo variations and time signature changes, as demonstrated on classical and contemporary music in the training data. However, disable constant tempo: true if you suspect tempo drift, and be aware that very extreme or rapid tempo changes may still cause failures. Q: Does the model work for non-Western music or underrepresented genres? A: The paper explicitly acknowledges that the model "can still fail, especially for difficult and underrepresented genres." The training datasets focus on Western music traditions. If you work with non-Western, experimental, or niche genres, test on a representative sample before production use.