📝 Originally published (in Japanese) at
[forge.workstyle.tech].
When migrating a voice conversion app to a separate server, you'll inevitably face this dilemma:
Precomputed features and anchor embeddings—those hundreds of megabytes of binary files—can't go in git. But the server absolutely needs them.
While your code can live in git, placing heavy generated artifacts like data/
or features/
in the repository is bad practice. The repo grows bloated, clones slow down, and history becomes unmanageable. Yet manually copying files leaves ambiguity: Did we transfer all the right files without corruption?
This article outlines a simple but reliable approach: bundle heavy assets with tar, distribute via scp, and verify integrity with MD5 checksums. It also covers the prerequisite .gitignore
design that makes this possible.
All hostnames and paths in this article are illustrative. Actual distribution targets and keys are not included.
The rule is straightforward:
data/
, features/
, output/
), model weights (checkpoints/
), external repos (external/
), virtual environmentsThe tricky part arises when exceptions mix in: *"We want to exclude all heavy binaries like *.wav or `
.npz , but we still need to include sample audio under web/public/personas/` in the repo."
.gitignore
Design: Exclude Broadly, Whitelist Exceptions
The key to a clean .gitignore
is "exclude broadly, then selectively whitelist exceptions" using !
. Here’s the actual configuration:
data/
features/
output/
*.wav
*.npz
!memo.md
!web/public/personas/
!web/public/personas/*.wav
external/
.venv-seedvc/
.venv-vc/
checkpoints/
Key design points:
*.wav
, you must first !
the directory (web/public/personas/
) before whitelisting the files inside (!web/public/personas/*.wav
). If the parent directory remains ignored, the !
for individual files won’t work./
to limit scope. For example, /lib/
excludes only the repo root’s lib/
directory, avoiding unintended matches with web/src/lib
.This setup ensures that heavy artifacts stay out of git while allowing only the minimal necessary files (like sample audio) to remain in the repo.
Assets excluded from git are distributed as a single tar archive via scp. The critical step is integrity verification—network transfers can silently corrupt large files.
Create a compressed tar archive containing only the heavy assets you want to distribute:
tar czf ml-assets.tar.gz data/ features/
Before sending, compute the MD5 hash of the archive (use md5
on macOS, md5sum
on Linux):
md5 ml-assets.tar.gz # macOS
Use scp to send the archive to the target server (hostnames and paths are illustrative):
scp ml-assets.tar.gz user@example-host:/srv/app/
On the receiving end, always verify the checksum before unpacking. Only when the hash matches the sender’s should you proceed.
md5sum ml-assets.tar.gz # Compare with sender's value
tar xzf ml-assets.tar.gz -C /srv/app/
If the checksums differ, the transfer was corrupted—do not unpack. Instead, request a retransmission. This simple step prevents "it seems to work" issues that are hard to debug later.
*.wav
), first whitelist the parent directory (!web/public/personas/
), then the files inside (!web/public/personas/*.wav
)..gitignore
data/
, features/
, checkpoints/
, etc.) .gitignore
should follow the pattern: *.wav
globally while whitelisting only specific files like !web/public/personas/*.wav
.