{"slug": "distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp", "title": "Distributing Large ML Assets (data/features) to a Separate Server - Using tar, scp, and MD5 Verification", "summary": "An engineer at Workstyle Tech detailed a method for distributing large machine learning assets, such as precomputed features and model weights, to a separate server using tar, scp, and MD5 checksums. The approach involves excluding heavy files from git via .gitignore, bundling them into a compressed archive, transferring with scp, and verifying integrity with MD5 hashes before extraction. The post emphasizes the importance of checksum verification to prevent silent corruption during transfer.", "body_md": "📝 Originally published (in Japanese) at\n\n[forge.workstyle.tech].\n\nWhen migrating a voice conversion app to a separate server, you'll inevitably face this dilemma:\n\nPrecomputed features and anchor embeddings—those hundreds of megabytes of binary files—can't go in git. But the server absolutely needs them.\n\nWhile your code can live in git, placing heavy generated artifacts like `data/`\n\nor `features/`\n\nin the repository is bad practice. The repo grows bloated, clones slow down, and history becomes unmanageable. Yet manually copying files leaves ambiguity: *Did we transfer all the right files without corruption?*\n\nThis article outlines a simple but reliable approach: **bundle heavy assets with tar, distribute via scp, and verify integrity with MD5 checksums**. It also covers the prerequisite `.gitignore`\n\ndesign that makes this possible.\n\nAll hostnames and paths in this article are illustrative. Actual distribution targets and keys are not included.\n\nThe rule is straightforward:\n\n`data/`\n\n, `features/`\n\n, `output/`\n\n), model weights (`checkpoints/`\n\n), external repos (`external/`\n\n), virtual environmentsThe tricky part arises when exceptions mix in: *\"We want to exclude all heavy binaries like `*.wav` or `\n\n*.npz , but we still need to include sample audio under web/public/personas/` in the repo.\"*\n\n`.gitignore`\n\nDesign: Exclude Broadly, Whitelist Exceptions\nThe key to a clean `.gitignore`\n\nis **\"exclude broadly, then selectively whitelist exceptions\"** using `!`\n\n. Here’s the actual configuration:\n\n```\n# Project-specific (generated artifacts & data)\ndata/\nfeatures/\noutput/\n*.wav\n*.npz\n\n# But don't ignore documentation\n!memo.md\n# Include persona sample audio for the frontend (exception to *.wav exclusion)\n!web/public/personas/\n!web/public/personas/*.wav\nexternal/\n\n# Additional: Heavy artifacts from neural voice conversion\n.venv-seedvc/\n.venv-vc/\ncheckpoints/\n```\n\nKey design points:\n\n`*.wav`\n\n, you must first `!`\n\nthe directory (`web/public/personas/`\n\n) before whitelisting the files inside (`!web/public/personas/*.wav`\n\n). If the parent directory remains ignored, the `!`\n\nfor individual files won’t work.`/`\n\nto limit scope. For example, `/lib/`\n\nexcludes only the repo root’s `lib/`\n\ndirectory, avoiding unintended matches with `web/src/lib`\n\n.This setup ensures that heavy artifacts stay out of git while allowing only the minimal necessary files (like sample audio) to remain in the repo.\n\nAssets excluded from git are distributed as a single tar archive via scp. The critical step is **integrity verification**—network transfers can silently corrupt large files.\n\nCreate a compressed tar archive containing only the heavy assets you want to distribute:\n\n```\n# Only include the heavy assets you want to distribute\ntar czf ml-assets.tar.gz data/ features/\n```\n\nBefore sending, compute the MD5 hash of the archive (use `md5`\n\non macOS, `md5sum`\n\non Linux):\n\n```\nmd5 ml-assets.tar.gz          # macOS\n# md5sum ml-assets.tar.gz     # Linux\n```\n\nUse scp to send the archive to the target server (hostnames and paths are illustrative):\n\n```\nscp ml-assets.tar.gz user@example-host:/srv/app/\n```\n\nOn the receiving end, **always verify the checksum before unpacking**. Only when the hash matches the sender’s should you proceed.\n\n```\n# On the destination server\nmd5sum ml-assets.tar.gz    # Compare with sender's value\n# If it matches, extract\ntar xzf ml-assets.tar.gz -C /srv/app/\n```\n\nIf the checksums differ, the transfer was corrupted—do not unpack. Instead, request a retransmission. This simple step prevents \"it seems to work\" issues that are hard to debug later.\n\n`*.wav`\n\n), first whitelist the parent directory (`!web/public/personas/`\n\n), then the files inside (`!web/public/personas/*.wav`\n\n).`.gitignore`\n\n`data/`\n\n, `features/`\n\n, `checkpoints/`\n\n, etc.) `.gitignore`\n\nshould follow the pattern: `*.wav`\n\nglobally while whitelisting only specific files like `!web/public/personas/*.wav`\n\n.", "url": "https://wpnews.pro/news/distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp", "canonical_source": "https://dev.to/orca_forge/distributing-large-ml-assets-datafeatures-to-a-separate-server-using-tar-scp-and-md5-4phc", "published_at": "2026-08-02 17:03:59+00:00", "updated_at": "2026-08-02 17:18:16.809013+00:00", "lang": "en", "topics": ["mlops", "developer-tools"], "entities": ["Workstyle Tech"], "alternates": {"html": "https://wpnews.pro/news/distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp", "markdown": "https://wpnews.pro/news/distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp.md", "text": "https://wpnews.pro/news/distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp.txt", "jsonld": "https://wpnews.pro/news/distributing-large-ml-assets-data-features-to-a-separate-server-using-tar-scp.jsonld"}}