# Distributing Large ML Assets (data/features) to a Separate Server - Using tar, scp, and MD5 Verification

> Source: <https://dev.to/orca_forge/distributing-large-ml-assets-datafeatures-to-a-separate-server-using-tar-scp-and-md5-4phc>
> Published: 2026-08-02 17:03:59+00:00

📝 Originally published (in Japanese) at

[forge.workstyle.tech].

When migrating a voice conversion app to a separate server, you'll inevitably face this dilemma:

Precomputed features and anchor embeddings—those hundreds of megabytes of binary files—can't go in git. But the server absolutely needs them.

While your code can live in git, placing heavy generated artifacts like `data/`

or `features/`

in the repository is bad practice. The repo grows bloated, clones slow down, and history becomes unmanageable. Yet manually copying files leaves ambiguity: *Did we transfer all the right files without corruption?*

This article outlines a simple but reliable approach: **bundle heavy assets with tar, distribute via scp, and verify integrity with MD5 checksums**. It also covers the prerequisite `.gitignore`

design that makes this possible.

All hostnames and paths in this article are illustrative. Actual distribution targets and keys are not included.

The rule is straightforward:

`data/`

, `features/`

, `output/`

), model weights (`checkpoints/`

), external repos (`external/`

), virtual environmentsThe tricky part arises when exceptions mix in: *"We want to exclude all heavy binaries like `*.wav` or `

*.npz , but we still need to include sample audio under web/public/personas/` in the repo."*

`.gitignore`

Design: Exclude Broadly, Whitelist Exceptions
The key to a clean `.gitignore`

is **"exclude broadly, then selectively whitelist exceptions"** using `!`

. Here’s the actual configuration:

```
# Project-specific (generated artifacts & data)
data/
features/
output/
*.wav
*.npz

# But don't ignore documentation
!memo.md
# Include persona sample audio for the frontend (exception to *.wav exclusion)
!web/public/personas/
!web/public/personas/*.wav
external/

# Additional: Heavy artifacts from neural voice conversion
.venv-seedvc/
.venv-vc/
checkpoints/
```

Key design points:

`*.wav`

, you must first `!`

the directory (`web/public/personas/`

) before whitelisting the files inside (`!web/public/personas/*.wav`

). If the parent directory remains ignored, the `!`

for individual files won’t work.`/`

to limit scope. For example, `/lib/`

excludes only the repo root’s `lib/`

directory, avoiding unintended matches with `web/src/lib`

.This setup ensures that heavy artifacts stay out of git while allowing only the minimal necessary files (like sample audio) to remain in the repo.

Assets excluded from git are distributed as a single tar archive via scp. The critical step is **integrity verification**—network transfers can silently corrupt large files.

Create a compressed tar archive containing only the heavy assets you want to distribute:

```
# Only include the heavy assets you want to distribute
tar czf ml-assets.tar.gz data/ features/
```

Before sending, compute the MD5 hash of the archive (use `md5`

on macOS, `md5sum`

on Linux):

```
md5 ml-assets.tar.gz          # macOS
# md5sum ml-assets.tar.gz     # Linux
```

Use scp to send the archive to the target server (hostnames and paths are illustrative):

```
scp ml-assets.tar.gz user@example-host:/srv/app/
```

On the receiving end, **always verify the checksum before unpacking**. Only when the hash matches the sender’s should you proceed.

```
# On the destination server
md5sum ml-assets.tar.gz    # Compare with sender's value
# If it matches, extract
tar xzf ml-assets.tar.gz -C /srv/app/
```

If the checksums differ, the transfer was corrupted—do not unpack. Instead, request a retransmission. This simple step prevents "it seems to work" issues that are hard to debug later.

`*.wav`

), first whitelist the parent directory (`!web/public/personas/`

), then the files inside (`!web/public/personas/*.wav`

).`.gitignore`

`data/`

, `features/`

, `checkpoints/`

, etc.) `.gitignore`

should follow the pattern: `*.wav`

globally while whitelisting only specific files like `!web/public/personas/*.wav`

.