Distributing Large ML Assets (data/features) to a Separate Server - Using tar, scp, and MD5 Verification

An engineer at Workstyle Tech detailed a method for distributing large machine learning assets, such as precomputed features and model weights, to a separate server using tar, scp, and MD5 checksums. The approach involves excluding heavy files from git via .gitignore, bundling them into a compressed archive, transferring with scp, and verifying integrity with MD5 hashes before extraction. The post emphasizes the importance of checksum verification to prevent silent corruption during transfer.

📝 Originally published in Japanese at forge.workstyle.tech . When migrating a voice conversion app to a separate server, you'll inevitably face this dilemma: Precomputed features and anchor embeddings—those hundreds of megabytes of binary files—can't go in git. But the server absolutely needs them. While your code can live in git, placing heavy generated artifacts like data/ or features/ in the repository is bad practice. The repo grows bloated, clones slow down, and history becomes unmanageable. Yet manually copying files leaves ambiguity: Did we transfer all the right files without corruption? This article outlines a simple but reliable approach: bundle heavy assets with tar, distribute via scp, and verify integrity with MD5 checksums . It also covers the prerequisite .gitignore design that makes this possible. All hostnames and paths in this article are illustrative. Actual distribution targets and keys are not included. The rule is straightforward: data/ , features/ , output/ , model weights checkpoints/ , external repos external/ , virtual environmentsThe tricky part arises when exceptions mix in: "We want to exclude all heavy binaries like .wav or .npz , but we still need to include sample audio under web/public/personas/ in the repo." .gitignore Design: Exclude Broadly, Whitelist Exceptions The key to a clean .gitignore is "exclude broadly, then selectively whitelist exceptions" using . Here’s the actual configuration: Project-specific generated artifacts & data data/ features/ output/ .wav .npz But don't ignore documentation memo.md Include persona sample audio for the frontend exception to .wav exclusion web/public/personas/ web/public/personas/ .wav external/ Additional: Heavy artifacts from neural voice conversion .venv-seedvc/ .venv-vc/ checkpoints/ Key design points: .wav , you must first the directory web/public/personas/ before whitelisting the files inside web/public/personas/ .wav . If the parent directory remains ignored, the for individual files won’t work. / to limit scope. For example, /lib/ excludes only the repo root’s lib/ directory, avoiding unintended matches with web/src/lib .This setup ensures that heavy artifacts stay out of git while allowing only the minimal necessary files like sample audio to remain in the repo. Assets excluded from git are distributed as a single tar archive via scp. The critical step is integrity verification —network transfers can silently corrupt large files. Create a compressed tar archive containing only the heavy assets you want to distribute: Only include the heavy assets you want to distribute tar czf ml-assets.tar.gz data/ features/ Before sending, compute the MD5 hash of the archive use md5 on macOS, md5sum on Linux : md5 ml-assets.tar.gz macOS md5sum ml-assets.tar.gz Linux Use scp to send the archive to the target server hostnames and paths are illustrative : scp ml-assets.tar.gz user@example-host:/srv/app/ On the receiving end, always verify the checksum before unpacking . Only when the hash matches the sender’s should you proceed. On the destination server md5sum ml-assets.tar.gz Compare with sender's value If it matches, extract tar xzf ml-assets.tar.gz -C /srv/app/ If the checksums differ, the transfer was corrupted—do not unpack. Instead, request a retransmission. This simple step prevents "it seems to work" issues that are hard to debug later. .wav , first whitelist the parent directory web/public/personas/ , then the files inside web/public/personas/ .wav . .gitignore data/ , features/ , checkpoints/ , etc. .gitignore should follow the pattern: .wav globally while whitelisting only specific files like web/public/personas/ .wav .