Originally published by AlphaPebble Labs — we build AI-powered intelligence pipelines.
Reading time: ~15 minutes. Audience: anyone who has to train or fine-tune a model on a rented GPU. New terms introduced: PTY, SFTP, gated repo, base image, pexpect, File Integrity Verification, end-to-end MD5.
This is the field guide I wish I had when I started: every command, every error message, every workaround that turned a 4-hour "wasted on infra" afternoon into a 30-minute "ship it" morning. It is dense. It is opinionated. It is war stories, not a tutorial.
You sign up, you spin up a "pod" (a Linux container with a GPU), and you connect to it.
The container comes with PyTorch + CUDA pre-installed. You can ssh
into it, pip install
what you need, and run your training script.
The catch: RunPod's SSH gateway is a console, not an exec channel. Every script you
have ever written that does ssh host "command"
will fail. Every CI/CD pipeline you have
ever used that does scp
or sftp
will fail. You cannot use the standard tools.
You have three things that do work:
Interactive— you log in and type commands by hand.ssh
— peer-to-peer file transfer between two machines that both haverunpodctl send
/runpodctl receive
runpodctl
installed. Generates a one-time code.HuggingFace Hub— if your model is on HF, you canhuggingface-cli download
it from the pod (the pod has internet).
That's it. There is no SCP, no rsync, no SFTP subsystem. If you want to move a file between your laptop and a pod, you use one of those three.
RunPod offers many "templates" (pre-built images). A solid default for a small (~1B
parameter) LoRA fine-tune is runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
. The numbers matter:
PyTorch 2.1.0: works withtransformers
up to 4.x out of the box. Fortransformers >= 5.0
you need PyTorch >= 2.4 (see the upgrade section below).CUDA 11.8: matches an RTX 3090's compute capability (8.6). CUDA 12.x is also fine on a 3090, but the matching 11.8 toolkit ships with the devel image, so you don't have to fiddle withnvcc
if you ever need to compile a CUDA extension.Ubuntu 22.04: LTS, well-supported, has Python 3.10 in the system. The image also haspython3.10
in/usr/local/bin
which is the one with PyTorch installed; don't try to usepython
(which is the system 3.10 without PyTorch) or you'll be debugging "torch not found" for 20 minutes.
python3 -c "import torch; print(f'torch: {torch.__version__}, cuda: {torch.cuda.is_available()}')"
For a small model (≤1-2B params + LoRA), the RTX 3090 at ~$0.22/hr is usually the right call over the RTX 4090 at ~$0.40/hr. Both have 24 GB VRAM; the 4090 has faster tensor cores, but for a small-model/few-epoch workload the time saved is minutes, not hours. The 3090 tends to be ~45% cheaper for ~5% less speed — the math is rarely close.
Save the A100 (40/80 GB) and H100 for fine-tuning 7B+ models — they're overkill and not worth the cost below that.
RunPod has two places to register SSH keys:
Account-level(Settings → SSH Public Keys): the key is added to every new pod you create. This is the right place to add your key.** Pod-level**(Connect → Add SSH Key): only applies to the current pod. Disappears when the pod is terminated.
Always use the account-level key. If you only add at the pod level, you have to re-add it every time you create a new pod, which is annoying when you are iterating.
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_runpod -C "runpod-$(whoami)"
cat ~/.ssh/id_ed25519_runpod.pub
The RunPod SSH connection string looks like:
ssh root@<pod-id>-<port>@ssh.runpod.io
Note on scope: the behavior below is what I observed on the pod templates and account tier I used at the time of writing. RunPod offers multiple connection paths (some templates/plans expose direct SSH to the container in addition to the proxied gateway), so treat this as "what to check for," not a universal guarantee — verify against your own pod before assuming it applies.
With that caveat, on the setup I used, the SSH connection was a gateway that forwarded a console to the pod, not a normal SSH server. The implications I hit:
Noexec
channel.ssh host 'command'
failed with*"Your SSH client doesn't support PTY"*. I could not run a command non-interactively.No SFTP.sftp host
failed with*"Channel closed"*— the SFTP subsystem wasn't advertised by the gateway.No Same reason.scp
.PTY required. Logging in needed a pseudo-terminal. Paramiko'sinvoke_shell(term="xterm")
was the only channel that worked reliably.
If you've been writing shell scripts that do ssh host "run this"
, or using fabric
or
ansible
to run commands on remote hosts, check this first — those tools assume a Unix-y SSH server, and a proxied gateway like the one described here is not.
Write a small Python wrapper around paramiko that opens a PTY-allocated shell, sends commands one at a time, waits for the prompt, and streams output back to the caller. The contract looks like this:
from runpod_ssh import run_remote
output, exit_code = run_remote(
pod_id="<pod-id>",
key_path="/Users/me/.ssh/id_ed25519_runpod",
commands=[
"cd /workspace/project",
"tar -czf /tmp/bundle.tar.gz results/checkpoint",
"ls -lh /tmp/bundle.tar.gz",
],
per_command_timeout=60,
)
The wrapper needs to handle three painful bits:
Allocate a PTY(invoke_shell(term="xterm")
).Wait for the prompt after each command (regex-matched).Capture the exit code by sendingecho $?
after the last command.
This is the only reliable way to run commands on RunPod from a script. If you try to
use subprocess.run(["ssh", host, cmd])
, it will fail in mysterious ways. If you try to
use paramiko.exec_command()
directly, it will fail with the PTY error. Write the wrapper once and reuse it.
You might think "I'll just wrap ssh in pexpect" and be done. The problem is that
pexpect
spawns a child process and reads its output. When the child writes colored
progress bars (which runpodctl
does), the output confuses pexpect's regex matching. The colored output includes control characters that look like prompt matches.
Paramiko, in contrast, is talking directly to the SSH protocol. It does not have a child process to manage. You control the bytes on the wire.
This is the part that took 4 hours the first time and 10 minutes the third time. The sequence of failures:
runpodctl send /tmp/bundle.tar.gz
runpodctl receive 1234-foo-bar-5
Should work. Did not work. After 30 seconds, the receive on the pod said
"room not ready". Why? The runpodctl
command uses colored progress bars. An
expect
script written to capture the code timed out silently because the colored output confused the prompt-matching.
Lesson: runpodctl
is not pexpect-friendly. Either pipe through script
to strip
colors, or capture the code by other means (the relay shows it in stdout — just read
stdout and parse for code is: <code>
).
client = paramiko.SSHClient()
client.connect(...)
sftp = client.open_sftp() # ← fails here
sftp.put(local, remote)
The SFTP subsystem is not advertised by the gateway. client.open_sftp()
raises
paramiko.ssh_exception.SSHException: Channel closed.
Lesson: SFTP is not available. Do not waste time on it.
base64 /tmp/bundle.tar.gz | wc -c
cat << 'EOF' | base64 -d > /tmp/bundle.tar.gz
... 7.5 MB of base64 ...
EOF
Works for small files. For a 5.6 MB file, the heredoc was truncated because the terminal session has a buffer limit. The file arrived 4.7 MB instead of 5.6 MB, and the checksum didn't match. This wasn't noticed for 10 minutes.
Lesson: Heredocs in interactive shells are limited. The limit is in the terminal emulator, not in the SSH layer. Modern terminals handle multi-MB inputs, but not all of them do. Test with a checksum before assuming success.
nohup runpodctl send /tmp/bundle.tar.gz > /tmp/send_out.log 2>&1 &
sleep 5
head -5 /tmp/send_out.log
runpodctl receive 4658-trade-gopher-under-3
Worked. The transfer completed at 99% in 12 seconds for a 5.6 MB file. The earlier failures had been looking at the wrong file. Always checksum:
sha1sum bundle.tar.gz
sha1sum /workspace/bundle.tar.gz
On the source side:nohup runpodctl send <file> > /tmp/send.log 2>&1 &
(in background, output to file).Wait 5 seconds for the code to be generated and written to the log.Read the code from the log withhead /tmp/send.log
.On the receiving side:runpodctl receive <code>
.Verify with on both sides.sha1sum
A 5.6 MB file takes ~30 seconds this way. A 1.6 GB file (model download) takes 15-45 minutes depending on RunPod's network load.
Many popular base models (e.g. Gemma, Llama) are gated repos on HuggingFace. Even with a valid token, down fails with a 401 or 403 unless two conditions are met:
You have accepted the license on the model's HuggingFace page (click "Agree and access repository").Your token has "Access to public gated repositories" enabled in the fine-grained token settings athttps://huggingface.co/settings/tokens.
The error messages are not helpful:
401 Unauthorized: You must have access to it and be authenticated to access it.
403 Forbidden: Please enable access to public gated repositories in your fine-grained
token settings to view this repository.
If you get 401, you have not accepted the license. If you get 403, you have the license but your token does not have the right scope. This is the most common 20-minute detour when first setting up a cloud training environment.
Once the token is set up, log in on the pod:
hf auth login --token hf_xxxxxxxxxxxxxxxx
If you set HF_DEBUG=1
before running, every HTTP request is logged with the full URL and headers. This is the only reliable way to debug auth issues.
A common starting image looks like:
python 3.10.12
torch 2.1.0+cu118
transformers 4.x
(whatever was latest at image build time)
If your training script needs transformers >= 5.0
for newer model features:
pip install --upgrade transformers trl peft
The fix: upgrade torch to a version that supports both CUDA 11.8 and the newer transformers. The 2.5.x line supports CUDA 11.8:
pip install --upgrade torch==2.5.1 \
--index-url https://download.pytorch.org/whl/cu118
After this, python3 -c "import torch; print(torch.__version__)"
should print
2.5.1+cu118
and the transformers warning goes away.
This is a model/eval-harness bug, not an infra bug, but it's common enough on rented GPUs — where every hour costs money and re-runs are expensive — to be worth including as a general failure mode, not tied to any specific project.
The shape of it: a training script's final step often evaluates the model in-process right after training, using a sample of held-out records, and reports some accuracy number. Later, a separate, standalone eval script re-evaluates the same trained model independently — and reports a wildly different, much worse number.
The usual root cause is a silent mismatch between the two eval paths, typically one or both of:
Input preprocessing differs. The standalone eval feeds the model raw input instead of running it through whatever cleaning step the training pipeline applied (e.g. stripping markup, normalizing whitespace, truncating to a fixed length).Output parsing differs. The standalone eval extracts the prediction naively (e.g. "take the first token") instead of matching the exact output format the model was trained to emit (e.g. a structured tag like<output>CLASS</output>
via regex).
Since the model was trained to expect a specific input shape and produce a specific output shape, feeding it something slightly different at eval time can make it emit off-format or unrelated continuations — and a naive parser will then confidently extract garbage as "the prediction," tanking the reported score even though the underlying model is fine.
The lesson: always reproduce the training format exactly in the eval format, on both the input-preprocessing and output-parsing sides. The smallest deviation between the two paths can produce wildly different, and misleading, accuracy numbers.
A pod at $0.22/hr adds up fast if you forget it's running. RunPod does not auto-terminate idle pods.
Always terminate when you are done. Two ways:
RunPod UI→ Pods → "Stop" button. Takes 30 seconds.** RunPod API**→POST /pods/{podId}/stop
. Faster but requires an API key.
runpodctl stop pod <pod-id>
Consider wiring a "terminate" step into your own training script so the workflow is
always: train → download → terminate
, with terminate as an explicit, hard-to-forget last step.
| Operation | Time | Cost |
|---|---|---|
| Pod spin-up (cold start) | 2-3 min | ~$0.01 |
| Pod spin-up (warm, image cached) | 30-60 sec | < $0.01 |
| 5-epoch LoRA training (~1B model) | ~2.5 hr | $0.55 |
| File upload (5.6 MB bundle) | 30 sec | $0.00 |
| File download (1.6 GB model) | 15-45 min | $0.06-$0.17 |
| Total per iteration | ||
| ~3-3.5 hr | ||
| $0.65-$0.75 | ||
| Idle pod (forgotten) | forever | $0.22/hr |
| Idle pod (1 day) | 24 hr | $5.28 |
For reference: the same workload on an M2 Pro Mac takes ~3 hours of wall time (free) but blocks the machine. Renting costs ~$0.65 per run but is 5-7x faster and doesn't block your laptop — roughly $0.22 per hour of saved time.
If you develop on an M-series Mac (MPS backend) but train on CUDA, you may hit a PyTorch error in certain layout/vision models that never shows up on the cloud pod:
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS
framework doesn't support float64. Please use float32 instead.
File ".../transformers/models/rt_detr_v2/modeling_rt_detr_v2.py",
line 988, in build_2d_sinusoidal_position_embedding
omega = torch.arange(pos_dim, dtype=torch.float64, device=device)
Why it only happens on MPS: some vision/layout models create a torch.float64
tensor for sinusoidal position embeddings. CUDA has full float64 support; MPS does not.
Why you won't catch it on the cloud: the training pod uses CUDA, where this works fine. The bug only bites when you run the inference pipeline locally on a Mac.
The fix (parameterize the device so it works everywhere):
-
A config value:
device: "auto" # auto | cuda | mps | cpu -
A pure function that resolves the device (env var → config → auto-detect), with graceful fallback to CPU if the requested device is unavailable.
-
Pass the resolved device into whatever accelerator options your inference library exposes.
-
Override at runtime with an env var when you need to, e.g.
DEVICE=cpu python run.py
.
Performance note: CPU is ~10x slower than MPS or CUDA for this kind of workload. If
it's only a fallback path (e.g. PDF layout parsing that's rarely hit), the overhead may
be negligible. If your workload is heavy on that path, set the device explicitly per
platform (mps
on Mac, cuda
on cloud).
If you are setting up RunPod for the first time, here is the checklist:
Generate an SSH key dedicated to RunPod (ssh-keygen -t ed25519
).Add the public key to your account-level RunPod settings.Create a pod with a PyTorch/CUDA image matching your target CUDA version and a GPU sized for your model.Connect via SSH in the RunPod web terminal to verify it works.Upgrade torch if you need a newertransformers
than the image ships with.Log in to HuggingFace withhf auth login --token <token>
if you need gated models.Verify CUDA with the one-liner above.Transfer files usingrunpodctl send
/receive
, with checksum verification.Run training withpython -u train.py ...
(the-u
makes stdout unbuffered so you see progress in real time).Download the trained model withrunpodctl
.**Terminate the pod.**Verify the checksum of the downloaded model matches the checksum on the pod.
If you skip step 12, you might find out days later that the model you're deploying is corrupted and have to re-train.
A field guide from the trenches, from the team at AlphaPebble Labs. The patterns documented here are the ones that actually worked, in the order they actually worked. The patterns that didn't work are documented too, because the most expensive thing in engineering is re-learning that a broken approach is broken.
We're AlphaPebble Labs — building AI-powered intelligence pipelines. More field notes at alphapebble.io.