Article: Beat-Aligned Mobile Audio Streaming with Virtual Chunks and Native Playback

Colossal, before pivoting to agentic commerce, built a mobile beat-discovery app that required beat-aligned, artifact-free audio switching under mobile constraints. The system stores one encoded MP3 per beat, fetches prioritized byte ranges, and schedules changes inside a native C++ playback engine, avoiding full-file preloading and standard player limitations.

Key Takeaways - Interactive audio apps often require a different streaming model than linear media playback, especially when users frequently jump between sections or tracks. - Beat-aligned switching is a scheduling problem as much as a buffering problem, so timing-critical decisions need to be made within the native playback execution path. - Deterministic prefetching based on a user's likely next actions is a practical first step before introducing more complex behavior-driven predictive models. - Virtual chunks let a system store one encoded file and fetch only prioritized byte ranges. - Selective MP3 decoding needs codec-boundary handling; otherwise, starting playback at arbitrary chunks can produce audible artifacts even when byte ranges are correct. At Colossal, before the company pivoted to agentic commerce, we were building a mobile beat-discovery app. It allowed producers to upload beats, and artists to swipe through a personalized feed ranked by a machine learning ML recommender using real-time in-session signals such as skip rate, listening time, repeated section locks, and replays. The core interaction model of the mobile app was to: - Play the current track from the start. - Jump to the previous or next section within the current track. - Lock a section, then keep that same section active while swiping through different tracks. That interaction model combined personalized feed navigation with hard audio constraints. Each swipe or section jump had to trigger an immediate, beat-aligned, artifact-free track or section switch despite mobile latency, network jitter, and bandwidth limits. The final design avoided full-file preloading by storing one encoded MP3 per beat, generating compact chunk descriptors, fetching only prioritized byte ranges, decoding chunks with MP3 warm-up context, and scheduling track or section changes inside a native audio control loop. For this project, I was responsible for the system end to end: backend audio analysis and chunk-metadata generation, binary descriptor format, mobile range-streaming strategy, and a native C++ playback engine integrated with React Native on iOS and Android. Figure 1: Beat discovery with section locks. Image Source: created by the author. Design Constraints For the system to work, all of these constraints had to be simultaneously held: - Section switches had to be immediate, seamless, and artifact-free. - Section loops had to avoid clicks or gaps at loop boundaries. - Track transitions had to be bar-aligned to preserve rhythmic continuity. - Playback had to stay seamless despite users spending only two to three seconds per beat and feed order changing in real time. - The implementation had to remain usable on weak 3G-class mobile connections. The difficult part was not any single requirement. It was making codec handling, transport, buffering, scheduling, and native playback behave as one system under mobile constraints. Why Existing Approaches Fell Short Before building a custom system, I evaluated three baseline approaches against the product's playback model. Standard Audio Players At first, I investigated whether standard audio player libraries could support the required playback model. While they were well-suited for basic linear playback, they were not designed for immediate, beat-aligned section and track switching. They also did not expose the low-level playback control or targeted prebuffering capabilities this use case required. The React Native RN bridge architecture introduced an additional limitation. A track switch could not reliably occur at the required moment because pausing the current player and starting the next one still had to cross the RN bridge. In my tests, this track switch added roughly five to ten milliseconds of delay plus jitter. By the time progress callbacks reached the JavaScript layer, playback had already advanced, making precise timing difficult to achieve. Figure 2: Delays in standard players flow. Image Source: created by the author. HLS/DASH I also evaluated HTTP Live Streaming HLS and Dynamic Adaptive Streaming over HTTP DASH , two widely used streaming protocols that deliver media as a sequence of small segments. These protocols are optimized for sequential playback, adaptive bitrate switching, and efficient streaming over varying network conditions. However, they were not designed for the kind of rapid, non-linear section hopping required by this app. Another challenge was that MP3 decoding depends on context from preceding frames. If playback starts at a physical segment boundary without that context, the first decoded samples can be incorrect, producing audible artifacts. While crossfades can mask some of these artifacts, they do not restore the missing decode context and were not acceptable for the interaction model we wanted. Another limitation was buffering behavior. Users could jump to a different section at any time, and if the target section was not already buffered, playback had to pause while the segment was fetched and decoded. That pausing introduced latency at the moment when an immediate response was required. Figure 3: HLS/DASH vs. MP3 frame decoding. Image Source: created by the author. Full File Downloads I also considered fully downloading each audio file before playback. However, the bandwidth requirements were far beyond the target operating conditions. The average beat size was about three megabytes, while users typically spent only two to three seconds on a beat before swiping away. Fully downloading the next beat within that window would require roughly eight megabits per second of sustained throughput for audio data alone. Once decoding overhead, network variability, and bandwidth shared with other app assets were included, the practical requirement was closer to fourteen megabits per second. Because the app needed to work reliably at around five hundred kilobits per second , even the optimistic estimate exceeded the available bandwidth by roughly sixteen times, making full-file downloads unviable. This evaluation made clear that meeting the product constraints required a custom transport model and a native playback engine. High-Level Architecture At a high level, the system had six stages: upload, server-side processing, descriptor generation, storage, mobile range fetching, and native playback. Figure 4: Architecture overview. Image Source: created by the author. Virtual Chunking When a producer uploaded a beat, a backend worker processed and analyzed it. One key design decision was how to support selective loading on mobile. There were two ways to chunk the file: - When using physical segments, HLS-style, each track becomes many small objects, which increases request count, metadata overhead, cache fragmentation, and boundary-handling complexity at segment edges. - When using virtual chunks, keep one MP3 file in storage, store byte ranges for each chunk, and fetch only the needed ranges with HTTP Range requests. I chose virtual chunks because they gave precise control over which bytes were fetched next, without the storage and request overhead of managing many physical segment files. Figure 5: Virtual chunks visualization. Image Source: created by the author. MP3 and CBR for v1 I evaluated MP3 and Advanced Audio Coding AAC for this pipeline and chose MP3 for v1 because frame-level inspection and chunk planning were simpler in the processing pipeline, decoder behavior was predictable across the target devices and libraries, and it let me move faster within the v1 delivery timeline. I also forced constant bitrate CBR during audio processing. Keeping most chunks close to the same size enabled a simple pool of preallocated buffers in the player, which reduced allocation complexity and made buffer reuse simpler at runtime. Chunk-Size Trade-Off The playback engine imposed a practical lower bound of roughly two seconds of audio per chunk. Smaller chunks were possible, but they did not make sense in this implementation because the player could not use less than that size, and splitting further only added requests. Smaller chunks improved responsiveness but increased request overhead. Larger chunks reduced request pressure, but on low-bandwidth connections they consumed too much throughput on audio the user might skip, which delayed chunks that actually needed to load next. Around 3.5 seconds was the best balance for this interaction model. MP3 Boundary Handling MP3 frames are not always independently decodable. Because of bit-reservoir behavior https://en.wikipedia.org/wiki/MP3 , the first valid samples of a chunk can depend on data from earlier frames. That idiosyncrasy made it harder to decode a chunk in isolation to stitch it into the playback buffer because it had to reproduce the same PCM output we would get at that point from continuous full-file decoding. To preserve that continuity in my decoder path, I prepended nine overlap frames to every non-initial chunk during chunk planning. Then I decoded with that warm-up context and discarded the warm-up samples before writing PCM into the playback buffer. If I skipped that overlap warm-up, chunk starts produced audible artifacts on real devices. The nine-frame value was based on MP3 decoder warm-up behavior and validated against the target decoder path. If this system had supported multiple codec implementations, I would have treated this situation as a per-decoder compatibility rule rather than a universal constant. Descriptor Format The mobile client had to fetch and parse chunk metadata quickly on weak connections, so I treated the descriptor as part of the transport design. I first used JSON for inspection, then added a compact binary format optimized for fast mobile parsing and low overhead. Each track had two descriptors: a JSON version for debugging and inspection, and a compact binary descriptor containing the chunk metadata required by the mobile client. Each chunk was described by this structure: @dataclass class ChunkData: start frame: int frames: int start byte: int bytes: int Track position was derived from frame index: position ms = frame index samples per frame 1000 / sample rate That mapping lets the client resolve a section timestamp to the correct chunk index. With a ten-minute max track length as a v1 product constraint, each chunk record fit into twelve bytes: - start frame: uint16 - frames: uint16 - start byte: uint32 - bytes: uint32 f.write struct.pack "<I", version f.write struct.pack "<I", len chunks uint32 count for chunk in chunks: f.write struct.pack "<HHII", chunk.start frame, chunk.frames, chunk.start byte, chunk.bytes I sized each field against explicit v1 constraints: - max track length: 10 minutes - sample rate: 44,100 Hz - MP3 samples per frame: 1,152 That sizing gives about twenty-three thousand MP3 frames per track, so uint16 was enough for start frame , and uint32 gave enough room for byte offsets and chunk payload sizes under the target file constraints. I considered both Protobuf https://github.com/protocolbuffers/protobuf and MessagePack https://github.com/msgpack/msgpack . I still chose a tiny custom format because the record shape was fixed, parsing logic was trivial in Python and C++, no additional serialization dependency was required, and I had exact control over byte layout and versioning. If the schema had become more complex, Protobuf would likely have been the better choice. Native Playback Engine As mentioned, off-the-shelf React Native players were not suitable for the playback behavior this app required, so I implemented the playback layer in native C++ on top of Superpowered http://superpowered.com . I packaged it as a native Expo module https://docs.expo.dev/workflow/customizing/ using-libraries-that-include-native-code , built with Builder Bob https://github.com/callstack/react-native-builder-bob at the time, and kept control-critical logic in native code. The timing-critical parts of playback had to run natively, and the core logic had to stay shared across iOS and Android. Audio callbacks run on strict deadlines, so even small stalls can produce audible artifacts. Keeping state transitions and buffer writes in shared C++ kept them off the React Native bridge path and avoided JS runtime pauses in playback control. The JS/TS interface was intentionally small: js const play = = … const pause = = … const loadTrack = track: TrackMetadata = … const unloadTrack = uid: string = … const setCurrentTrack = uid: string = … const seekTo = milliseconds: number = … const loopTrackSection = trackUid: string, sectionId: number = ... Getting audio to play was straightforward. The difficult part was keeping playback correct under timing, looping, and network pressure. Beat-Aligned Switching and Section Loops To avoid off-beat cuts and keep transitions rhythmically correct, the app did not allow users to jump to arbitrary points in the playback stream. To this aim, track and section switches were not executed immediately upon user input but were scheduled within the native control loop to occur at the next bar boundary. Likewise, section loops align exactly with section boundaries. A loop only activated after at least the first chunk of that section was already buffered, so the player did not click, gap, or start in silence. Figure 6: Beat-aligned switching. Image Source: created by the author. Native Transport and Descriptor Parsing Given our performance constraints, the native layer had to fetch binary descriptors and audio byte ranges directly from shared C++ code. Because React Native C++ modules do not ship with a full cross-platform HTTP stack by default, transport had to be implemented explicitly. There were two options: implementing a native networking stack per platform, with separate iOS and Android implementations or a cross-platform C++ HTTP layer. I based my implementation on libcurl https://github.com/curl/curl to ensure consistent handling of range requests, retries, and errors on both platforms. It required more initial setup, but it avoided duplicating transport logic and reduced the risk of iOS and Android behavior drifting over time. For parsing, the native reader had to match the backend wire format: pragma pack push, 1 struct ChunkData { uint16 t startFrame; uint16 t frames; uint32 t startByte; uint32 t bytes; }; pragma pack pop The binary descriptor stored chunk structs as tightly packed twelve-byte entries. The struct layout on the target platforms already matched the wire format, but I kept the packing explicit to guarantee that the in-memory representation remained identical to the serialized format. Runtime Decode Pipeline After implementing descriptor parsing, I defined the track's AudioInMemory structure. To avoid allocations later, I preallocated one PCM buffer slot per chunk, allowing worker threads to decode directly into the corresponding slot as data arrived. The runtime flow was: - Download compressed chunk bytes via HTTP Range . - Copy those bytes into a decoder-owned buffer. - Open the decoder on that buffer in a worker thread. - Find the matching preallocated PCM slot inside AudioInMemory . - Skip overlap samples for non-initial chunks. - Decode directly into that target PCM buffer. This approach kept both memory allocation and decoding out of the audio callback thread, minimizing the work performed on that thread and reducing the chance of audible artifacts. Buffer Sizing Once metadata was available in the native layer, I used the sample count for each chunk to compute PCM buffer sizes before any decode work started. // MP3 SAMPLES PER FRAME = 1152, OVERLAP FRAMES = 9 in the decoder path samplesInChunk = chunk.frames - OVERLAP FRAMES MP3 SAMPLES PER FRAME; chunkBufferSizeBytes = 4 samplesInChunk + 16384; The extra 16384 bytes came from Superpowered's buffer sizing requirements for this in-memory playback path. Decode and Buffer Fill After the native layer fetched a chunk with an HTTP Range request, its compressed bytes were first copied into a decoder-owned buffer, then a worker thread decoded them directly into the target PCM slot: auto compressedAudioBuffer = malloc dataSize ; memcpy compressedAudioBuffer, data.data , dataSize ; auto decoder = std::make unique<Decoder ; int openCode = decoder- openAudioFileInMemory compressedAudioBuffer, dataSize ; // Skip warm-up overlap for non-initial chunks decoder- setPositionPrecise OVERLAP SAMPLES ; auto payloadPtr = findPayloadPointerWithIndexInAudioInMemory trackInfo- audioInMemory, chunkIndex ; int decodeCode = decoder- decodeAudio static cast<short payloadPtr , chunkSampleCountWithoutOverlap ; Decoding directly into the target PCM slot avoided a second PCM copy before playback. Thread Model and Real-Time Constraints I split the runtime across three kinds of work: - Audio thread with audio callback, no blocking, no heavy allocations. - Network/decode workers with range fetch and MP3 decode. - Control logic with scheduling track and section changes, updating player state, and coordinating prefetch priorities. The audio thread had to remain real-time safe. Network fetches, decode work, allocations, and lock-heavy coordination could not run inside the playback callback. Even a small delay there could cause audible dropouts. For simple shared state, I used atomics and kept critical sections short, so the audio thread was not blocked by lower-priority work. A few runtime optimizations made a noticeable difference in real usage. First, I kept multiple players warm, including the current, previous, and upcoming tracks. As a result, switching tracks did not require player initialization at the moment of the switch. I also reused downloader connections on a per-track basis, allowing repeated HTTP range requests to avoid the overhead of establishing new connections. In addition, I replaced repeated PCM buffer allocations with a small reusable pool. Because the stream used CBR encoding, most chunks were similarly sized, making buffer reuse very effective. Finally, I kept shared state minimal and lock scope tight, which helped keep coordination overhead low even under load. Prefetch Priority Algorithm The prefetcher did not try to download everything at once. Instead, it continuously ranked a small set of possible next playback paths and spent bandwidth on the chunks most likely to be needed next. Priority depended first on whether section lock was enabled or not, second on the ordering between tracks and sections. When section lock was enabled, the highest priority was the same section of the next beat because that was the next likely playback target. When section lock was disabled, the highest priority was the first section of the next beat because that was the default next-track entry point. Next, the prefetcher prioritized neighboring track and section targets, such as previous track, next section, and previous section. Other chunks were loaded opportunistically after higher-priority playback paths were covered. The decision loop was intentionally deterministic: onPlaybackStateChanged state : candidates = candidates += chunks needed to continue current path state candidates += chunks needed for scheduled bar transition state if state.section lock enabled: candidates += same section in next track state candidates += same section in previous track state Else: candidates += first section in next track state candidates += first section in previous track state candidates += neighboring sections in current track state candidates += opportunistic remaining chunks state fetch missing chunks in priority order candidates This approach kept the most likely next playback targets ready before the user reached them, while avoiding long preload queues that would often be invalidated by swipes or feed re-ranking. Figure 7: Prioritization algorithm. Image Source: created by the author. Why Not Start with Predictive Prefetching? A probabilistic prefetch model was surely feasible, but we did not have enough behavior data yet to create it. Starting with a deterministic strategy based on section-lock state and neighboring track/section targets made the system understandable, testable, and useful before a learned model would have enough data to improve it. Reliability and Debugging The hardest issues we had to solve were cross-layer timing and state-synchronization failures. For example, by the time a UI action was processed, the audio engine could already be several milliseconds ahead. Boundary chunks exposed additional edge cases in the warm-up skip logic and required special handling before chunk-starts decoded cleanly. At the same time, prefetch scheduling that appeared correct in logs could still fail under packet jitter, starving the chunk that was needed immediately. Shared-state synchronization presented another challenge. Patterns that seemed harmless in normal code paths could introduce audible artifacts when executed on the audio thread, so mutex usage had to be minimized and state handoff kept lightweight. I relied heavily on instrumentation, timing traces, and controlled, repeatable network throttling tests to make the system behave reliably. Validation and Limits I do not have preserved production telemetry or benchmark logs for this project, so I refrain from presenting this implementation as a benchmarked general-purpose streaming system. The validation I discuss here is the development validation that we applied while building the product. The system was repeatedly tested with instrumentation, timing traces, audible checks at loop and switch boundaries, and controlled network throttling. In those tests, it held up on weak 3G-class connections and preserved the playback behavior the product required. Specifically, section loops did not introduce audible artifacts at loop boundaries. Section switches stayed clean and beat-aligned. Beat-to-beat switching remained usable under constrained mobile bandwidth. Additionally, prefetch and playback remained stable during swipe-heavy use. The design also had clear limits typical of a first implementation. For example, the descriptor integer sizes assumed a maximum track length of ten minutes, reflecting the requirements of the product at the time. Similarly, the MP3 overlap handling was validated only against the specific MP3 decoder used by this implementation and was not intended as a decoder-agnostic solution. The prefetcher was also intentionally simple. It followed a deterministic strategy rather than adapting based on production usage patterns or learned behavior. In addition, the system was optimized for short beat-preview interactions, not for long-form sequential media playback. Those constraints were acceptable for the product, but they are important if applying the design elsewhere. What Is Public vs. What Is Private This system was built inside a startup product, so the production code and internal infrastructure are not public. This writeup focuses on the architecture, algorithms, and implementation patterns that can be discussed openly. Broader Applicability Although the system was built for a beat marketplace, many of the underlying engineering patterns generalize to other domains. One example of this generalization is handling dependent codec boundaries when selective decoding must still preserve PCM continuity across segment transitions. Another example is designing prefetch strategies around the constrained set of user actions, allowing resources to be prioritized for the most likely paths. The same ideas also apply to native mobile playback pipelines that must remain responsive under weak or variable network conditions. Finally, beat and bar-aligned transition control is relevant to a broader class of interactive audio products where playback must remain tightly synchronized with user interactions. Conclusion The core system was implemented and working correctly during development. Treating codec boundaries, metadata format, transport, prefetch priority, native scheduling, and audio-thread safety as a single design problem rather than a collection of independent optimizations was ultimately what made the implementation successful. Solving any one of those pieces in isolation would not have been enough. Correct chunk boundaries needed decoder warm-up handling to avoid artifacts. Efficient range fetching depended on the right chunks being prioritized ahead of user actions. Low-latency playback required timing-critical decisions to run inside the native playback path. The desired outcome became possible only when all of those pieces worked together.