# Cloud Training on RunPod: A Field Guide to the Edge Cases > Source: > Published: 2026-08-05 11:48:38+00:00 *Originally published by AlphaPebble Labs — we build AI-powered intelligence pipelines.* Reading time: ~15 minutes. Audience: anyone who has to train or fine-tune a model on a rented GPU. New terms introduced: PTY, SFTP, gated repo, base image, pexpect, File Integrity Verification, end-to-end MD5. This is the field guide I wish I had when I started: every command, every error message, every workaround that turned a 4-hour "wasted on infra" afternoon into a 30-minute "ship it" morning. It is dense. It is opinionated. It is war stories, not a tutorial. You sign up, you spin up a "pod" (a Linux container with a GPU), and you connect to it. The container comes with PyTorch + CUDA pre-installed. You can `ssh` into it, `pip install` what you need, and run your training script. The catch: **RunPod's SSH gateway is a console, not an exec channel.** Every script you have ever written that does `ssh host "command"` will fail. Every CI/CD pipeline you have ever used that does `scp` or `sftp` will fail. You cannot use the standard tools. You have three things that do work: **Interactive**— you log in and type commands by hand.`ssh` — peer-to-peer file transfer between two machines that both have`runpodctl send` /`runpodctl receive` `runpodctl` installed. Generates a one-time code.**HuggingFace Hub**— if your model is on HF, you can`huggingface-cli download` it from the pod (the pod has internet). That's it. There is no SCP, no rsync, no SFTP subsystem. If you want to move a file between your laptop and a pod, you use one of those three. RunPod offers many "templates" (pre-built images). A solid default for a small (~1B parameter) LoRA fine-tune is `runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04` . The numbers matter: **PyTorch 2.1.0**: works with`transformers` up to 4.x out of the box. For`transformers >= 5.0` you need PyTorch >= 2.4 (see the upgrade section below).**CUDA 11.8**: matches an RTX 3090's compute capability (8.6). CUDA 12.x is also fine on a 3090, but the matching 11.8 toolkit ships with the devel image, so you don't have to fiddle with`nvcc` if you ever need to compile a CUDA extension.**Ubuntu 22.04**: LTS, well-supported, has Python 3.10 in the system. The image also has`python3.10` in`/usr/local/bin` which is the one with PyTorch installed; don't try to use`python` (which is the system 3.10 without PyTorch) or you'll be debugging "torch not found" for 20 minutes. ``` python # Verify CUDA works after pod startup python3 -c "import torch; print(f'torch: {torch.__version__}, cuda: {torch.cuda.is_available()}')" # Expected: torch: 2.1.0+cu118, cuda: True ``` For a small model (≤1-2B params + LoRA), the **RTX 3090** at ~$0.22/hr is usually the right call over the RTX 4090 at ~$0.40/hr. Both have 24 GB VRAM; the 4090 has faster tensor cores, but for a small-model/few-epoch workload the time saved is minutes, not hours. The 3090 tends to be ~45% cheaper for ~5% less speed — the math is rarely close. Save the A100 (40/80 GB) and H100 for fine-tuning 7B+ models — they're overkill and not worth the cost below that. RunPod has *two* places to register SSH keys: **Account-level**(Settings → SSH Public Keys): the key is added to every new pod you create. This is the right place to add your key.** Pod-level**(Connect → Add SSH Key): only applies to the current pod. Disappears when the pod is terminated. Always use the account-level key. If you only add at the pod level, you have to re-add it every time you create a new pod, which is annoying when you are iterating. ``` # Generate a dedicated key for RunPod (don't reuse your GitHub key) ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_runpod -C "runpod-$(whoami)" # Copy the public key to RunPod Settings → SSH Public Keys cat ~/.ssh/id_ed25519_runpod.pub ``` The RunPod SSH connection string looks like: ``` ssh root@-@ssh.runpod.io ``` **Note on scope**: the behavior below is what I observed on the pod templates and account tier I used at the time of writing. RunPod offers multiple connection paths (some templates/plans expose direct SSH to the container in addition to the proxied gateway), so treat this as "what to check for," not a universal guarantee — verify against your own pod before assuming it applies. With that caveat, on the setup I used, the SSH connection was a *gateway* that forwarded a console to the pod, not a normal SSH server. The implications I hit: **No**`exec` channel.`ssh host 'command'` failed with*"Your SSH client doesn't support PTY"*. I could not run a command non-interactively.**No SFTP.**`sftp host` failed with*"Channel closed"*— the SFTP subsystem wasn't advertised by the gateway.**No** Same reason.`scp` .**PTY required.** Logging in needed a pseudo-terminal. Paramiko's`invoke_shell(term="xterm")` was the only channel that worked reliably. If you've been writing shell scripts that do `ssh host "run this"` , or using `fabric` or `ansible` to run commands on remote hosts, check this first — those tools assume a Unix-y SSH server, and a proxied gateway like the one described here is not. Write a small Python wrapper around paramiko that opens a PTY-allocated shell, sends commands one at a time, waits for the prompt, and streams output back to the caller. The contract looks like this: ``` python from runpod_ssh import run_remote output, exit_code = run_remote( pod_id="", key_path="/Users/me/.ssh/id_ed25519_runpod", commands=[ "cd /workspace/project", "tar -czf /tmp/bundle.tar.gz results/checkpoint", "ls -lh /tmp/bundle.tar.gz", ], per_command_timeout=60, ) ``` The wrapper needs to handle three painful bits: **Allocate a PTY**(`invoke_shell(term="xterm")` ).**Wait for the prompt** after each command (regex-matched).**Capture the exit code** by sending`echo $?` after the last command. This is the only reliable way to run commands on RunPod from a script. If you try to use `subprocess.run(["ssh", host, cmd])` , it will fail in mysterious ways. If you try to use `paramiko.exec_command()` directly, it will fail with the PTY error. Write the wrapper once and reuse it. You might think "I'll just wrap ssh in pexpect" and be done. The problem is that `pexpect` spawns a child process and reads its output. When the child writes colored progress bars (which `runpodctl` does), the output confuses pexpect's regex matching. The colored output includes control characters that look like prompt matches. Paramiko, in contrast, is talking directly to the SSH protocol. It does not have a child process to manage. You control the bytes on the wire. This is the part that took 4 hours the first time and 10 minutes the third time. The sequence of failures: ``` # local runpodctl send /tmp/bundle.tar.gz # Outputs: # 1234-foo-bar-5 # code is: 1234-foo-bar-5 # on the other computer run # runpodctl receive 1234-foo-bar-5 # pod runpodctl receive 1234-foo-bar-5 ``` Should work. *Did not work*. After 30 seconds, the receive on the pod said *"room not ready"*. Why? The `runpodctl` command uses colored progress bars. An `expect` script written to capture the code timed out silently because the colored output confused the prompt-matching. **Lesson**: `runpodctl` is not pexpect-friendly. Either pipe through `script` to strip colors, or capture the code by other means (the relay shows it in stdout — just read stdout and parse for `code is: ` ). ``` client = paramiko.SSHClient() client.connect(...) sftp = client.open_sftp() # ← fails here sftp.put(local, remote) ``` The SFTP subsystem is not advertised by the gateway. `client.open_sftp()` raises `paramiko.ssh_exception.SSHException: Channel closed.` **Lesson**: SFTP is not available. Do not waste time on it. ``` # local base64 /tmp/bundle.tar.gz | wc -c # 7680000 (for 5.6 MB file → 7.5 MB base64) # pod cat << 'EOF' | base64 -d > /tmp/bundle.tar.gz ... 7.5 MB of base64 ... EOF ``` Works for small files. For a 5.6 MB file, the heredoc was truncated because the terminal session has a buffer limit. The file arrived 4.7 MB instead of 5.6 MB, and the checksum didn't match. This wasn't noticed for 10 minutes. **Lesson**: Heredocs in interactive shells are limited. The limit is in the *terminal emulator*, not in the SSH layer. Modern terminals handle multi-MB inputs, but not all of them do. Test with a checksum before assuming success. ``` # pod (in background, output to a file) nohup runpodctl send /tmp/bundle.tar.gz > /tmp/send_out.log 2>&1 & # wait 5 seconds, then read the code from the log sleep 5 head -5 /tmp/send_out.log # code is: 4658-trade-gopher-under-3 # local runpodctl receive 4658-trade-gopher-under-3 ``` *Worked.* The transfer completed at 99% in 12 seconds for a 5.6 MB file. The earlier failures had been looking at the wrong file. Always checksum: ``` # local sha1sum bundle.tar.gz # dc0d8aa10aad4cc88b358098d9a126141c9b61e6 # pod sha1sum /workspace/bundle.tar.gz # dc0d8aa10aad4cc88b358098d9a126141c9b61e6 ← matches! ``` **On the source side**:`nohup runpodctl send > /tmp/send.log 2>&1 &` (in background, output to file).**Wait 5 seconds** for the code to be generated and written to the log.**Read the code** from the log with`head /tmp/send.log` .**On the receiving side**:`runpodctl receive ` .**Verify with** on both sides.`sha1sum` A 5.6 MB file takes ~30 seconds this way. A 1.6 GB file (model download) takes 15-45 minutes depending on RunPod's network load. Many popular base models (e.g. Gemma, Llama) are *gated repos* on HuggingFace. Even with a valid token, downloading fails with a 401 or 403 unless two conditions are met: **You have accepted the license** on the model's HuggingFace page (click "Agree and access repository").**Your token has "Access to public gated repositories" enabled** in the fine-grained token settings at[https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). The error messages are not helpful: ``` 401 Unauthorized: You must have access to it and be authenticated to access it. 403 Forbidden: Please enable access to public gated repositories in your fine-grained token settings to view this repository. ``` If you get 401, you have not accepted the license. If you get 403, you have the license but your token does not have the right scope. This is the most common 20-minute detour when first setting up a cloud training environment. Once the token is set up, log in on the pod: ``` hf auth login --token hf_xxxxxxxxxxxxxxxx # Token is valid (permission: read). # Login successful. ``` If you set `HF_DEBUG=1` before running, every HTTP request is logged with the full URL and headers. This is the only reliable way to debug auth issues. A common starting image looks like: `python 3.10.12` `torch 2.1.0+cu118` `transformers 4.x` (whatever was latest at image build time) If your training script needs `transformers >= 5.0` for newer model features: ``` pip install --upgrade transformers trl peft # After upgrade: # transformers 5.9.0 requires torch >= 2.4 but found 2.1.0 ``` The fix: upgrade torch to a version that supports both CUDA 11.8 and the newer transformers. The 2.5.x line supports CUDA 11.8: ``` pip install --upgrade torch==2.5.1 \ --index-url https://download.pytorch.org/whl/cu118 ``` After this, `python3 -c "import torch; print(torch.__version__)"` should print `2.5.1+cu118` and the transformers warning goes away. This is a model/eval-harness bug, not an infra bug, but it's common enough on rented GPUs — where every hour costs money and re-runs are expensive — to be worth including as a general failure mode, not tied to any specific project. The shape of it: a training script's final step often evaluates the model *in-process* right after training, using a sample of held-out records, and reports some accuracy number. Later, a *separate*, standalone eval script re-evaluates the same trained model independently — and reports a wildly different, much worse number. The usual root cause is a silent mismatch between the two eval paths, typically one or both of: **Input preprocessing differs.** The standalone eval feeds the model raw input instead of running it through whatever cleaning step the training pipeline applied (e.g. stripping markup, normalizing whitespace, truncating to a fixed length).**Output parsing differs.** The standalone eval extracts the prediction naively (e.g. "take the first token") instead of matching the exact output format the model was trained to emit (e.g. a structured tag like`CLASS` via regex). Since the model was *trained* to expect a specific input shape and produce a specific output shape, feeding it something slightly different at eval time can make it emit off-format or unrelated continuations — and a naive parser will then confidently extract garbage as "the prediction," tanking the reported score even though the underlying model is fine. **The lesson: always reproduce the training format exactly in the eval format**, on both the input-preprocessing and output-parsing sides. The smallest deviation between the two paths can produce wildly different, and misleading, accuracy numbers. A pod at $0.22/hr adds up fast if you forget it's running. RunPod does *not* auto-terminate idle pods. **Always terminate when you are done.** Two ways: **RunPod UI**→ Pods → "Stop" button. Takes 30 seconds.** RunPod API**→`POST /pods/{podId}/stop` . Faster but requires an API key. ``` # If you have the RunPod CLI installed runpodctl stop pod ``` Consider wiring a "terminate" step into your own training script so the workflow is always: `train → download → terminate` , with terminate as an explicit, hard-to-forget last step. | Operation | Time | Cost | |---|---|---| | Pod spin-up (cold start) | 2-3 min | ~$0.01 | | Pod spin-up (warm, image cached) | 30-60 sec | < $0.01 | | 5-epoch LoRA training (~1B model) | ~2.5 hr | $0.55 | | File upload (5.6 MB bundle) | 30 sec | $0.00 | | File download (1.6 GB model) | 15-45 min | $0.06-$0.17 | Total per iteration | ~3-3.5 hr | $0.65-$0.75 | | Idle pod (forgotten) | forever | $0.22/hr | | Idle pod (1 day) | 24 hr | $5.28 | For reference: the same workload on an M2 Pro Mac takes ~3 hours of wall time (free) but blocks the machine. Renting costs ~$0.65 per run but is 5-7x faster and doesn't block your laptop — roughly $0.22 per hour of saved time. If you develop on an M-series Mac (MPS backend) but train on CUDA, you may hit a PyTorch error in certain layout/vision models that never shows up on the cloud pod: ``` TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead. File ".../transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 988, in build_2d_sinusoidal_position_embedding omega = torch.arange(pos_dim, dtype=torch.float64, device=device) ``` **Why it only happens on MPS**: some vision/layout models create a `torch.float64` tensor for sinusoidal position embeddings. CUDA has full float64 support; MPS does not. **Why you won't catch it on the cloud**: the training pod uses CUDA, where this works fine. The bug only bites when you run the *inference* pipeline locally on a Mac. **The fix** (parameterize the device so it works everywhere): - A config value: `device: "auto" # auto | cuda | mps | cpu` - A pure function that resolves the device (env var → config → auto-detect), with graceful fallback to CPU if the requested device is unavailable. - Pass the resolved device into whatever accelerator options your inference library exposes. - Override at runtime with an env var when you need to, e.g. `DEVICE=cpu python run.py` . **Performance note**: CPU is ~10x slower than MPS or CUDA for this kind of workload. If it's only a fallback path (e.g. PDF layout parsing that's rarely hit), the overhead may be negligible. If your workload is heavy on that path, set the device explicitly per platform (`mps` on Mac, `cuda` on cloud). If you are setting up RunPod for the first time, here is the checklist: **Generate an SSH key** dedicated to RunPod (`ssh-keygen -t ed25519` ).**Add the public key** to your account-level RunPod settings.**Create a pod** with a PyTorch/CUDA image matching your target CUDA version and a GPU sized for your model.**Connect via SSH** in the RunPod web terminal to verify it works.**Upgrade torch** if you need a newer`transformers` than the image ships with.**Log in to HuggingFace** with`hf auth login --token ` if you need gated models.**Verify CUDA** with the one-liner above.**Transfer files** using`runpodctl send` /`receive` , with checksum verification.**Run training** with`python -u train.py ...` (the`-u` makes stdout unbuffered so you see progress in real time).**Download the trained model** with`runpodctl` .**Terminate the pod.****Verify the checksum** of the downloaded model matches the checksum on the pod. If you skip step 12, you might find out days later that the model you're deploying is corrupted and have to re-train. *A field guide from the trenches, from the team at AlphaPebble Labs. The patterns documented here are the ones that actually worked, in the order they actually worked. The patterns that didn't work are documented too, because the most expensive thing in engineering is re-learning that a broken approach is broken.* *We're AlphaPebble Labs — building AI-powered intelligence pipelines. More field notes at alphapebble.io.*