Cloud Training on RunPod: A Field Guide to the Edge Cases

AlphaPebble Labs engineers detailed a field guide for training AI models on RunPod's rented GPU infrastructure, highlighting edge cases such as the SSH gateway acting as a console rather than an exec channel, which breaks standard scp/sftp workflows. The guide recommends specific base images, account-level SSH key registration, and cost-effective GPU choices like the RTX 3090 for small-model fine-tuning.

Originally published by AlphaPebble Labs — we build AI-powered intelligence pipelines. Reading time: ~15 minutes. Audience: anyone who has to train or fine-tune a model on a rented GPU. New terms introduced: PTY, SFTP, gated repo, base image, pexpect, File Integrity Verification, end-to-end MD5. This is the field guide I wish I had when I started: every command, every error message, every workaround that turned a 4-hour "wasted on infra" afternoon into a 30-minute "ship it" morning. It is dense. It is opinionated. It is war stories, not a tutorial. You sign up, you spin up a "pod" a Linux container with a GPU , and you connect to it. The container comes with PyTorch + CUDA pre-installed. You can ssh into it, pip install what you need, and run your training script. The catch: RunPod's SSH gateway is a console, not an exec channel. Every script you have ever written that does ssh host "command" will fail. Every CI/CD pipeline you have ever used that does scp or sftp will fail. You cannot use the standard tools. You have three things that do work: Interactive — you log in and type commands by hand. ssh — peer-to-peer file transfer between two machines that both have runpodctl send / runpodctl receive runpodctl installed. Generates a one-time code. HuggingFace Hub — if your model is on HF, you can huggingface-cli download it from the pod the pod has internet . That's it. There is no SCP, no rsync, no SFTP subsystem. If you want to move a file between your laptop and a pod, you use one of those three. RunPod offers many "templates" pre-built images . A solid default for a small ~1B parameter LoRA fine-tune is runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 . The numbers matter: PyTorch 2.1.0 : works with transformers up to 4.x out of the box. For transformers = 5.0 you need PyTorch = 2.4 see the upgrade section below . CUDA 11.8 : matches an RTX 3090's compute capability 8.6 . CUDA 12.x is also fine on a 3090, but the matching 11.8 toolkit ships with the devel image, so you don't have to fiddle with nvcc if you ever need to compile a CUDA extension. Ubuntu 22.04 : LTS, well-supported, has Python 3.10 in the system. The image also has python3.10 in /usr/local/bin which is the one with PyTorch installed; don't try to use python which is the system 3.10 without PyTorch or you'll be debugging "torch not found" for 20 minutes. python Verify CUDA works after pod startup python3 -c "import torch; print f'torch: {torch. version }, cuda: {torch.cuda.is available }' " Expected: torch: 2.1.0+cu118, cuda: True For a small model ≤1-2B params + LoRA , the RTX 3090 at ~$0.22/hr is usually the right call over the RTX 4090 at ~$0.40/hr. Both have 24 GB VRAM; the 4090 has faster tensor cores, but for a small-model/few-epoch workload the time saved is minutes, not hours. The 3090 tends to be ~45% cheaper for ~5% less speed — the math is rarely close. Save the A100 40/80 GB and H100 for fine-tuning 7B+ models — they're overkill and not worth the cost below that. RunPod has two places to register SSH keys: Account-level Settings → SSH Public Keys : the key is added to every new pod you create. This is the right place to add your key. Pod-level Connect → Add SSH Key : only applies to the current pod. Disappears when the pod is terminated. Always use the account-level key. If you only add at the pod level, you have to re-add it every time you create a new pod, which is annoying when you are iterating. Generate a dedicated key for RunPod don't reuse your GitHub key ssh-keygen -t ed25519 -f ~/.ssh/id ed25519 runpod -C "runpod-$ whoami " Copy the public key to RunPod Settings → SSH Public Keys cat ~/.ssh/id ed25519 runpod.pub The RunPod SSH connection string looks like: ssh root@<pod-id -<port @ssh.runpod.io Note on scope : the behavior below is what I observed on the pod templates and account tier I used at the time of writing. RunPod offers multiple connection paths some templates/plans expose direct SSH to the container in addition to the proxied gateway , so treat this as "what to check for," not a universal guarantee — verify against your own pod before assuming it applies. With that caveat, on the setup I used, the SSH connection was a gateway that forwarded a console to the pod, not a normal SSH server. The implications I hit: No exec channel. ssh host 'command' failed with "Your SSH client doesn't support PTY" . I could not run a command non-interactively. No SFTP. sftp host failed with "Channel closed" — the SFTP subsystem wasn't advertised by the gateway. No Same reason. scp . PTY required. Logging in needed a pseudo-terminal. Paramiko's invoke shell term="xterm" was the only channel that worked reliably. If you've been writing shell scripts that do ssh host "run this" , or using fabric or ansible to run commands on remote hosts, check this first — those tools assume a Unix-y SSH server, and a proxied gateway like the one described here is not. Write a small Python wrapper around paramiko that opens a PTY-allocated shell, sends commands one at a time, waits for the prompt, and streams output back to the caller. The contract looks like this: python from runpod ssh import run remote output, exit code = run remote pod id="<pod-id ", key path="/Users/me/.ssh/id ed25519 runpod", commands= "cd /workspace/project", "tar -czf /tmp/bundle.tar.gz results/checkpoint", "ls -lh /tmp/bundle.tar.gz", , per command timeout=60, The wrapper needs to handle three painful bits: Allocate a PTY invoke shell term="xterm" . Wait for the prompt after each command regex-matched . Capture the exit code by sending echo $? after the last command. This is the only reliable way to run commands on RunPod from a script. If you try to use subprocess.run "ssh", host, cmd , it will fail in mysterious ways. If you try to use paramiko.exec command directly, it will fail with the PTY error. Write the wrapper once and reuse it. You might think "I'll just wrap ssh in pexpect" and be done. The problem is that pexpect spawns a child process and reads its output. When the child writes colored progress bars which runpodctl does , the output confuses pexpect's regex matching. The colored output includes control characters that look like prompt matches. Paramiko, in contrast, is talking directly to the SSH protocol. It does not have a child process to manage. You control the bytes on the wire. This is the part that took 4 hours the first time and 10 minutes the third time. The sequence of failures: local runpodctl send /tmp/bundle.tar.gz Outputs: 1234-foo-bar-5 code is: 1234-foo-bar-5 on the other computer run runpodctl receive 1234-foo-bar-5 pod runpodctl receive 1234-foo-bar-5 Should work. Did not work . After 30 seconds, the receive on the pod said "room not ready" . Why? The runpodctl command uses colored progress bars. An expect script written to capture the code timed out silently because the colored output confused the prompt-matching. Lesson : runpodctl is not pexpect-friendly. Either pipe through script to strip colors, or capture the code by other means the relay shows it in stdout — just read stdout and parse for code is: <code . client = paramiko.SSHClient client.connect ... sftp = client.open sftp ← fails here sftp.put local, remote The SFTP subsystem is not advertised by the gateway. client.open sftp raises paramiko.ssh exception.SSHException: Channel closed. Lesson : SFTP is not available. Do not waste time on it. local base64 /tmp/bundle.tar.gz | wc -c 7680000 for 5.6 MB file → 7.5 MB base64 pod cat << 'EOF' | base64 -d /tmp/bundle.tar.gz ... 7.5 MB of base64 ... EOF Works for small files. For a 5.6 MB file, the heredoc was truncated because the terminal session has a buffer limit. The file arrived 4.7 MB instead of 5.6 MB, and the checksum didn't match. This wasn't noticed for 10 minutes. Lesson : Heredocs in interactive shells are limited. The limit is in the terminal emulator , not in the SSH layer. Modern terminals handle multi-MB inputs, but not all of them do. Test with a checksum before assuming success. pod in background, output to a file nohup runpodctl send /tmp/bundle.tar.gz /tmp/send out.log 2 &1 & wait 5 seconds, then read the code from the log sleep 5 head -5 /tmp/send out.log code is: 4658-trade-gopher-under-3 local runpodctl receive 4658-trade-gopher-under-3 Worked. The transfer completed at 99% in 12 seconds for a 5.6 MB file. The earlier failures had been looking at the wrong file. Always checksum: local sha1sum bundle.tar.gz dc0d8aa10aad4cc88b358098d9a126141c9b61e6 pod sha1sum /workspace/bundle.tar.gz dc0d8aa10aad4cc88b358098d9a126141c9b61e6 ← matches On the source side : nohup runpodctl send <file /tmp/send.log 2 &1 & in background, output to file . Wait 5 seconds for the code to be generated and written to the log. Read the code from the log with head /tmp/send.log . On the receiving side : runpodctl receive <code . Verify with on both sides. sha1sum A 5.6 MB file takes ~30 seconds this way. A 1.6 GB file model download takes 15-45 minutes depending on RunPod's network load. Many popular base models e.g. Gemma, Llama are gated repos on HuggingFace. Even with a valid token, downloading fails with a 401 or 403 unless two conditions are met: You have accepted the license on the model's HuggingFace page click "Agree and access repository" . Your token has "Access to public gated repositories" enabled in the fine-grained token settings at https://huggingface.co/settings/tokens https://huggingface.co/settings/tokens . The error messages are not helpful: 401 Unauthorized: You must have access to it and be authenticated to access it. 403 Forbidden: Please enable access to public gated repositories in your fine-grained token settings to view this repository. If you get 401, you have not accepted the license. If you get 403, you have the license but your token does not have the right scope. This is the most common 20-minute detour when first setting up a cloud training environment. Once the token is set up, log in on the pod: hf auth login --token hf xxxxxxxxxxxxxxxx Token is valid permission: read . Login successful. If you set HF DEBUG=1 before running, every HTTP request is logged with the full URL and headers. This is the only reliable way to debug auth issues. A common starting image looks like: python 3.10.12 torch 2.1.0+cu118 transformers 4.x whatever was latest at image build time If your training script needs transformers = 5.0 for newer model features: pip install --upgrade transformers trl peft After upgrade: transformers 5.9.0 requires torch = 2.4 but found 2.1.0 The fix: upgrade torch to a version that supports both CUDA 11.8 and the newer transformers. The 2.5.x line supports CUDA 11.8: pip install --upgrade torch==2.5.1 \ --index-url https://download.pytorch.org/whl/cu118 After this, python3 -c "import torch; print torch. version " should print 2.5.1+cu118 and the transformers warning goes away. This is a model/eval-harness bug, not an infra bug, but it's common enough on rented GPUs — where every hour costs money and re-runs are expensive — to be worth including as a general failure mode, not tied to any specific project. The shape of it: a training script's final step often evaluates the model in-process right after training, using a sample of held-out records, and reports some accuracy number. Later, a separate , standalone eval script re-evaluates the same trained model independently — and reports a wildly different, much worse number. The usual root cause is a silent mismatch between the two eval paths, typically one or both of: Input preprocessing differs. The standalone eval feeds the model raw input instead of running it through whatever cleaning step the training pipeline applied e.g. stripping markup, normalizing whitespace, truncating to a fixed length . Output parsing differs. The standalone eval extracts the prediction naively e.g. "take the first token" instead of matching the exact output format the model was trained to emit e.g. a structured tag like <output CLASS</output via regex . Since the model was trained to expect a specific input shape and produce a specific output shape, feeding it something slightly different at eval time can make it emit off-format or unrelated continuations — and a naive parser will then confidently extract garbage as "the prediction," tanking the reported score even though the underlying model is fine. The lesson: always reproduce the training format exactly in the eval format , on both the input-preprocessing and output-parsing sides. The smallest deviation between the two paths can produce wildly different, and misleading, accuracy numbers. A pod at $0.22/hr adds up fast if you forget it's running. RunPod does not auto-terminate idle pods. Always terminate when you are done. Two ways: RunPod UI → Pods → "Stop" button. Takes 30 seconds. RunPod API → POST /pods/{podId}/stop . Faster but requires an API key. If you have the RunPod CLI installed runpodctl stop pod <pod-id Consider wiring a "terminate" step into your own training script so the workflow is always: train → download → terminate , with terminate as an explicit, hard-to-forget last step. | Operation | Time | Cost | |---|---|---| | Pod spin-up cold start | 2-3 min | ~$0.01 | | Pod spin-up warm, image cached | 30-60 sec | < $0.01 | | 5-epoch LoRA training ~1B model | ~2.5 hr | $0.55 | | File upload 5.6 MB bundle | 30 sec | $0.00 | | File download 1.6 GB model | 15-45 min | $0.06-$0.17 | Total per iteration | ~3-3.5 hr | $0.65-$0.75 | | Idle pod forgotten | forever | $0.22/hr | | Idle pod 1 day | 24 hr | $5.28 | For reference: the same workload on an M2 Pro Mac takes ~3 hours of wall time free but blocks the machine. Renting costs ~$0.65 per run but is 5-7x faster and doesn't block your laptop — roughly $0.22 per hour of saved time. If you develop on an M-series Mac MPS backend but train on CUDA, you may hit a PyTorch error in certain layout/vision models that never shows up on the cloud pod: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead. File ".../transformers/models/rt detr v2/modeling rt detr v2.py", line 988, in build 2d sinusoidal position embedding omega = torch.arange pos dim, dtype=torch.float64, device=device Why it only happens on MPS : some vision/layout models create a torch.float64 tensor for sinusoidal position embeddings. CUDA has full float64 support; MPS does not. Why you won't catch it on the cloud : the training pod uses CUDA, where this works fine. The bug only bites when you run the inference pipeline locally on a Mac. The fix parameterize the device so it works everywhere : - A config value: device: "auto" auto | cuda | mps | cpu - A pure function that resolves the device env var → config → auto-detect , with graceful fallback to CPU if the requested device is unavailable. - Pass the resolved device into whatever accelerator options your inference library exposes. - Override at runtime with an env var when you need to, e.g. DEVICE=cpu python run.py . Performance note : CPU is ~10x slower than MPS or CUDA for this kind of workload. If it's only a fallback path e.g. PDF layout parsing that's rarely hit , the overhead may be negligible. If your workload is heavy on that path, set the device explicitly per platform mps on Mac, cuda on cloud . If you are setting up RunPod for the first time, here is the checklist: Generate an SSH key dedicated to RunPod ssh-keygen -t ed25519 . Add the public key to your account-level RunPod settings. Create a pod with a PyTorch/CUDA image matching your target CUDA version and a GPU sized for your model. Connect via SSH in the RunPod web terminal to verify it works. Upgrade torch if you need a newer transformers than the image ships with. Log in to HuggingFace with hf auth login --token <token if you need gated models. Verify CUDA with the one-liner above. Transfer files using runpodctl send / receive , with checksum verification. Run training with python -u train.py ... the -u makes stdout unbuffered so you see progress in real time . Download the trained model with runpodctl . Terminate the pod. Verify the checksum of the downloaded model matches the checksum on the pod. If you skip step 12, you might find out days later that the model you're deploying is corrupted and have to re-train. A field guide from the trenches, from the team at AlphaPebble Labs. The patterns documented here are the ones that actually worked, in the order they actually worked. The patterns that didn't work are documented too, because the most expensive thing in engineering is re-learning that a broken approach is broken. We're AlphaPebble Labs — building AI-powered intelligence pipelines. More field notes at alphapebble.io.