{"slug": "cloud-training-on-runpod-a-field-guide-to-the-edge-cases", "title": "Cloud Training on RunPod: A Field Guide to the Edge Cases", "summary": "AlphaPebble Labs engineers detailed a field guide for training AI models on RunPod's rented GPU infrastructure, highlighting edge cases such as the SSH gateway acting as a console rather than an exec channel, which breaks standard scp/sftp workflows. The guide recommends specific base images, account-level SSH key registration, and cost-effective GPU choices like the RTX 3090 for small-model fine-tuning.", "body_md": "*Originally published by AlphaPebble Labs — we build AI-powered\nintelligence pipelines.*\n\nReading time: ~15 minutes. Audience: anyone who has to train or fine-tune a model on a rented GPU. New terms introduced: PTY, SFTP, gated repo, base image, pexpect, File Integrity Verification, end-to-end MD5.\n\nThis is the field guide I wish I had when I started: every command, every error message, every workaround that turned a 4-hour \"wasted on infra\" afternoon into a 30-minute \"ship it\" morning. It is dense. It is opinionated. It is war stories, not a tutorial.\n\nYou sign up, you spin up a \"pod\" (a Linux container with a GPU), and you connect to it.\nThe container comes with PyTorch + CUDA pre-installed. You can `ssh`\n\ninto it, `pip install`\n\nwhat you need, and run your training script.\n\nThe catch: **RunPod's SSH gateway is a console, not an exec channel.** Every script you\nhave ever written that does `ssh host \"command\"`\n\nwill fail. Every CI/CD pipeline you have\never used that does `scp`\n\nor `sftp`\n\nwill fail. You cannot use the standard tools.\n\nYou have three things that do work:\n\n**Interactive**— you log in and type commands by hand.`ssh`\n\n— peer-to-peer file transfer between two machines that both have`runpodctl send`\n\n/`runpodctl receive`\n\n`runpodctl`\n\ninstalled. Generates a one-time code.**HuggingFace Hub**— if your model is on HF, you can`huggingface-cli download`\n\nit from the pod (the pod has internet).\n\nThat's it. There is no SCP, no rsync, no SFTP subsystem. If you want to move a file between your laptop and a pod, you use one of those three.\n\nRunPod offers many \"templates\" (pre-built images). A solid default for a small (~1B\nparameter) LoRA fine-tune is `runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04`\n\n.\nThe numbers matter:\n\n**PyTorch 2.1.0**: works with`transformers`\n\nup to 4.x out of the box. For`transformers >= 5.0`\n\nyou need PyTorch >= 2.4 (see the upgrade section below).**CUDA 11.8**: matches an RTX 3090's compute capability (8.6). CUDA 12.x is also fine on a 3090, but the matching 11.8 toolkit ships with the devel image, so you don't have to fiddle with`nvcc`\n\nif you ever need to compile a CUDA extension.**Ubuntu 22.04**: LTS, well-supported, has Python 3.10 in the system. The image also has`python3.10`\n\nin`/usr/local/bin`\n\nwhich is the one with PyTorch installed; don't try to use`python`\n\n(which is the system 3.10 without PyTorch) or you'll be debugging \"torch not found\" for 20 minutes.\n\n``` python\n# Verify CUDA works after pod startup\npython3 -c \"import torch; print(f'torch: {torch.__version__}, cuda: {torch.cuda.is_available()}')\"\n# Expected: torch: 2.1.0+cu118, cuda: True\n```\n\nFor a small model (≤1-2B params + LoRA), the **RTX 3090** at ~$0.22/hr is usually the\nright call over the RTX 4090 at ~$0.40/hr. Both have 24 GB VRAM; the 4090 has faster\ntensor cores, but for a small-model/few-epoch workload the time saved is minutes, not\nhours. The 3090 tends to be ~45% cheaper for ~5% less speed — the math is rarely close.\n\nSave the A100 (40/80 GB) and H100 for fine-tuning 7B+ models — they're overkill and not worth the cost below that.\n\nRunPod has *two* places to register SSH keys:\n\n**Account-level**(Settings → SSH Public Keys): the key is added to every new pod you create. This is the right place to add your key.** Pod-level**(Connect → Add SSH Key): only applies to the current pod. Disappears when the pod is terminated.\n\nAlways use the account-level key. If you only add at the pod level, you have to re-add it every time you create a new pod, which is annoying when you are iterating.\n\n```\n# Generate a dedicated key for RunPod (don't reuse your GitHub key)\nssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_runpod -C \"runpod-$(whoami)\"\n# Copy the public key to RunPod Settings → SSH Public Keys\ncat ~/.ssh/id_ed25519_runpod.pub\n```\n\nThe RunPod SSH connection string looks like:\n\n```\nssh root@<pod-id>-<port>@ssh.runpod.io\n```\n\n**Note on scope**: the behavior below is what I observed on the pod templates and\naccount tier I used at the time of writing. RunPod offers multiple connection paths\n(some templates/plans expose direct SSH to the container in addition to the proxied\ngateway), so treat this as \"what to check for,\" not a universal guarantee — verify\nagainst your own pod before assuming it applies.\n\nWith that caveat, on the setup I used, the SSH connection was a *gateway* that forwarded\na console to the pod, not a normal SSH server. The implications I hit:\n\n**No**`exec`\n\nchannel.`ssh host 'command'`\n\nfailed with*\"Your SSH client doesn't support PTY\"*. I could not run a command non-interactively.**No SFTP.**`sftp host`\n\nfailed with*\"Channel closed\"*— the SFTP subsystem wasn't advertised by the gateway.**No** Same reason.`scp`\n\n.**PTY required.** Logging in needed a pseudo-terminal. Paramiko's`invoke_shell(term=\"xterm\")`\n\nwas the only channel that worked reliably.\n\nIf you've been writing shell scripts that do `ssh host \"run this\"`\n\n, or using `fabric`\n\nor\n`ansible`\n\nto run commands on remote hosts, check this first — those tools assume a\nUnix-y SSH server, and a proxied gateway like the one described here is not.\n\nWrite a small Python wrapper around paramiko that opens a PTY-allocated shell, sends commands one at a time, waits for the prompt, and streams output back to the caller. The contract looks like this:\n\n``` python\nfrom runpod_ssh import run_remote\n\noutput, exit_code = run_remote(\n    pod_id=\"<pod-id>\",\n    key_path=\"/Users/me/.ssh/id_ed25519_runpod\",\n    commands=[\n        \"cd /workspace/project\",\n        \"tar -czf /tmp/bundle.tar.gz results/checkpoint\",\n        \"ls -lh /tmp/bundle.tar.gz\",\n    ],\n    per_command_timeout=60,\n)\n```\n\nThe wrapper needs to handle three painful bits:\n\n**Allocate a PTY**(`invoke_shell(term=\"xterm\")`\n\n).**Wait for the prompt** after each command (regex-matched).**Capture the exit code** by sending`echo $?`\n\nafter the last command.\n\nThis is the only reliable way to run commands on RunPod from a script. If you try to\nuse `subprocess.run([\"ssh\", host, cmd])`\n\n, it will fail in mysterious ways. If you try to\nuse `paramiko.exec_command()`\n\ndirectly, it will fail with the PTY error. Write the\nwrapper once and reuse it.\n\nYou might think \"I'll just wrap ssh in pexpect\" and be done. The problem is that\n`pexpect`\n\nspawns a child process and reads its output. When the child writes colored\nprogress bars (which `runpodctl`\n\ndoes), the output confuses pexpect's regex matching.\nThe colored output includes control characters that look like prompt matches.\n\nParamiko, in contrast, is talking directly to the SSH protocol. It does not have a child process to manage. You control the bytes on the wire.\n\nThis is the part that took 4 hours the first time and 10 minutes the third time. The sequence of failures:\n\n```\n# local\nrunpodctl send /tmp/bundle.tar.gz\n\n# Outputs:\n# 1234-foo-bar-5\n# code is: 1234-foo-bar-5\n# on the other computer run\n# runpodctl receive 1234-foo-bar-5\n# pod\nrunpodctl receive 1234-foo-bar-5\n```\n\nShould work. *Did not work*. After 30 seconds, the receive on the pod said\n*\"room not ready\"*. Why? The `runpodctl`\n\ncommand uses colored progress bars. An\n`expect`\n\nscript written to capture the code timed out silently because the colored\noutput confused the prompt-matching.\n\n**Lesson**: `runpodctl`\n\nis not pexpect-friendly. Either pipe through `script`\n\nto strip\ncolors, or capture the code by other means (the relay shows it in stdout — just read\nstdout and parse for `code is: <code>`\n\n).\n\n```\nclient = paramiko.SSHClient()\nclient.connect(...)\nsftp = client.open_sftp()  # ← fails here\nsftp.put(local, remote)\n```\n\nThe SFTP subsystem is not advertised by the gateway. `client.open_sftp()`\n\nraises\n`paramiko.ssh_exception.SSHException: Channel closed.`\n\n**Lesson**: SFTP is not available. Do not waste time on it.\n\n```\n# local\nbase64 /tmp/bundle.tar.gz | wc -c\n# 7680000 (for 5.6 MB file → 7.5 MB base64)\n\n# pod\ncat << 'EOF' | base64 -d > /tmp/bundle.tar.gz\n... 7.5 MB of base64 ...\nEOF\n```\n\nWorks for small files. For a 5.6 MB file, the heredoc was truncated because the terminal session has a buffer limit. The file arrived 4.7 MB instead of 5.6 MB, and the checksum didn't match. This wasn't noticed for 10 minutes.\n\n**Lesson**: Heredocs in interactive shells are limited. The limit is in the *terminal\nemulator*, not in the SSH layer. Modern terminals handle multi-MB inputs, but not all of\nthem do. Test with a checksum before assuming success.\n\n```\n# pod (in background, output to a file)\nnohup runpodctl send /tmp/bundle.tar.gz > /tmp/send_out.log 2>&1 &\n\n# wait 5 seconds, then read the code from the log\nsleep 5\nhead -5 /tmp/send_out.log\n# code is: 4658-trade-gopher-under-3\n\n# local\nrunpodctl receive 4658-trade-gopher-under-3\n```\n\n*Worked.* The transfer completed at 99% in 12 seconds for a 5.6 MB file. The earlier\nfailures had been looking at the wrong file. Always checksum:\n\n```\n# local\nsha1sum bundle.tar.gz\n# dc0d8aa10aad4cc88b358098d9a126141c9b61e6\n\n# pod\nsha1sum /workspace/bundle.tar.gz\n# dc0d8aa10aad4cc88b358098d9a126141c9b61e6  ← matches!\n```\n\n**On the source side**:`nohup runpodctl send <file> > /tmp/send.log 2>&1 &`\n\n(in background, output to file).**Wait 5 seconds** for the code to be generated and written to the log.**Read the code** from the log with`head /tmp/send.log`\n\n.**On the receiving side**:`runpodctl receive <code>`\n\n.**Verify with** on both sides.`sha1sum`\n\nA 5.6 MB file takes ~30 seconds this way. A 1.6 GB file (model download) takes 15-45 minutes depending on RunPod's network load.\n\nMany popular base models (e.g. Gemma, Llama) are *gated repos* on HuggingFace. Even\nwith a valid token, downloading fails with a 401 or 403 unless two conditions are met:\n\n**You have accepted the license** on the model's HuggingFace page (click \"Agree and access repository\").**Your token has \"Access to public gated repositories\" enabled** in the fine-grained token settings at[https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).\n\nThe error messages are not helpful:\n\n```\n401 Unauthorized: You must have access to it and be authenticated to access it.\n403 Forbidden: Please enable access to public gated repositories in your fine-grained\ntoken settings to view this repository.\n```\n\nIf you get 401, you have not accepted the license. If you get 403, you have the license but your token does not have the right scope. This is the most common 20-minute detour when first setting up a cloud training environment.\n\nOnce the token is set up, log in on the pod:\n\n```\nhf auth login --token hf_xxxxxxxxxxxxxxxx\n# Token is valid (permission: read).\n# Login successful.\n```\n\nIf you set `HF_DEBUG=1`\n\nbefore running, every HTTP request is logged with the full URL\nand headers. This is the only reliable way to debug auth issues.\n\nA common starting image looks like:\n\n`python 3.10.12`\n\n`torch 2.1.0+cu118`\n\n`transformers 4.x`\n\n(whatever was latest at image build time)\n\nIf your training script needs `transformers >= 5.0`\n\nfor newer model features:\n\n```\npip install --upgrade transformers trl peft\n# After upgrade:\n# transformers 5.9.0 requires torch >= 2.4 but found 2.1.0\n```\n\nThe fix: upgrade torch to a version that supports both CUDA 11.8 and the newer transformers. The 2.5.x line supports CUDA 11.8:\n\n```\npip install --upgrade torch==2.5.1 \\\n  --index-url https://download.pytorch.org/whl/cu118\n```\n\nAfter this, `python3 -c \"import torch; print(torch.__version__)\"`\n\nshould print\n`2.5.1+cu118`\n\nand the transformers warning goes away.\n\nThis is a model/eval-harness bug, not an infra bug, but it's common enough on rented GPUs — where every hour costs money and re-runs are expensive — to be worth including as a general failure mode, not tied to any specific project.\n\nThe shape of it: a training script's final step often evaluates the model *in-process*\nright after training, using a sample of held-out records, and reports some accuracy\nnumber. Later, a *separate*, standalone eval script re-evaluates the same trained model\nindependently — and reports a wildly different, much worse number.\n\nThe usual root cause is a silent mismatch between the two eval paths, typically one or both of:\n\n**Input preprocessing differs.** The standalone eval feeds the model raw input instead of running it through whatever cleaning step the training pipeline applied (e.g. stripping markup, normalizing whitespace, truncating to a fixed length).**Output parsing differs.** The standalone eval extracts the prediction naively (e.g. \"take the first token\") instead of matching the exact output format the model was trained to emit (e.g. a structured tag like`<output>CLASS</output>`\n\nvia regex).\n\nSince the model was *trained* to expect a specific input shape and produce a specific\noutput shape, feeding it something slightly different at eval time can make it emit\noff-format or unrelated continuations — and a naive parser will then confidently extract\ngarbage as \"the prediction,\" tanking the reported score even though the underlying model\nis fine.\n\n**The lesson: always reproduce the training format exactly in the eval format**, on both\nthe input-preprocessing and output-parsing sides. The smallest deviation between the two\npaths can produce wildly different, and misleading, accuracy numbers.\n\nA pod at $0.22/hr adds up fast if you forget it's running. RunPod does *not*\nauto-terminate idle pods.\n\n**Always terminate when you are done.** Two ways:\n\n**RunPod UI**→ Pods → \"Stop\" button. Takes 30 seconds.** RunPod API**→`POST /pods/{podId}/stop`\n\n. Faster but requires an API key.\n\n```\n# If you have the RunPod CLI installed\nrunpodctl stop pod <pod-id>\n```\n\nConsider wiring a \"terminate\" step into your own training script so the workflow is\nalways: `train → download → terminate`\n\n, with terminate as an explicit, hard-to-forget\nlast step.\n\n| Operation | Time | Cost |\n|---|---|---|\n| Pod spin-up (cold start) | 2-3 min | ~$0.01 |\n| Pod spin-up (warm, image cached) | 30-60 sec | < $0.01 |\n| 5-epoch LoRA training (~1B model) | ~2.5 hr | $0.55 |\n| File upload (5.6 MB bundle) | 30 sec | $0.00 |\n| File download (1.6 GB model) | 15-45 min | $0.06-$0.17 |\nTotal per iteration |\n~3-3.5 hr |\n$0.65-$0.75 |\n| Idle pod (forgotten) | forever | $0.22/hr |\n| Idle pod (1 day) | 24 hr | $5.28 |\n\nFor reference: the same workload on an M2 Pro Mac takes ~3 hours of wall time (free) but blocks the machine. Renting costs ~$0.65 per run but is 5-7x faster and doesn't block your laptop — roughly $0.22 per hour of saved time.\n\nIf you develop on an M-series Mac (MPS backend) but train on CUDA, you may hit a PyTorch error in certain layout/vision models that never shows up on the cloud pod:\n\n```\nTypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS\nframework doesn't support float64. Please use float32 instead.\n\n  File \".../transformers/models/rt_detr_v2/modeling_rt_detr_v2.py\",\n        line 988, in build_2d_sinusoidal_position_embedding\n    omega = torch.arange(pos_dim, dtype=torch.float64, device=device)\n```\n\n**Why it only happens on MPS**: some vision/layout models create a `torch.float64`\n\ntensor for sinusoidal position embeddings. CUDA has full float64 support; MPS does not.\n\n**Why you won't catch it on the cloud**: the training pod uses CUDA, where this works\nfine. The bug only bites when you run the *inference* pipeline locally on a Mac.\n\n**The fix** (parameterize the device so it works everywhere):\n\n- A config value:\n`device: \"auto\" # auto | cuda | mps | cpu`\n\n- A pure function that resolves the device (env var → config → auto-detect), with graceful fallback to CPU if the requested device is unavailable.\n- Pass the resolved device into whatever accelerator options your inference library exposes.\n- Override at runtime with an env var when you need to, e.g.\n`DEVICE=cpu python run.py`\n\n.\n\n**Performance note**: CPU is ~10x slower than MPS or CUDA for this kind of workload. If\nit's only a fallback path (e.g. PDF layout parsing that's rarely hit), the overhead may\nbe negligible. If your workload is heavy on that path, set the device explicitly per\nplatform (`mps`\n\non Mac, `cuda`\n\non cloud).\n\nIf you are setting up RunPod for the first time, here is the checklist:\n\n**Generate an SSH key** dedicated to RunPod (`ssh-keygen -t ed25519`\n\n).**Add the public key** to your account-level RunPod settings.**Create a pod** with a PyTorch/CUDA image matching your target CUDA version and a GPU sized for your model.**Connect via SSH** in the RunPod web terminal to verify it works.**Upgrade torch** if you need a newer`transformers`\n\nthan the image ships with.**Log in to HuggingFace** with`hf auth login --token <token>`\n\nif you need gated models.**Verify CUDA** with the one-liner above.**Transfer files** using`runpodctl send`\n\n/`receive`\n\n, with checksum verification.**Run training** with`python -u train.py ...`\n\n(the`-u`\n\nmakes stdout unbuffered so you see progress in real time).**Download the trained model** with`runpodctl`\n\n.**Terminate the pod.****Verify the checksum** of the downloaded model matches the checksum on the pod.\n\nIf you skip step 12, you might find out days later that the model you're deploying is corrupted and have to re-train.\n\n*A field guide from the trenches, from the team at AlphaPebble Labs.\nThe patterns documented here are the ones that actually worked, in the order they\nactually worked. The patterns that didn't work are documented too, because the most\nexpensive thing in engineering is re-learning that a broken approach is broken.*\n\n*We're AlphaPebble Labs — building AI-powered\nintelligence pipelines. More field notes at alphapebble.io.*", "url": "https://wpnews.pro/news/cloud-training-on-runpod-a-field-guide-to-the-edge-cases", "canonical_source": "https://gist.github.com/balijepalli/b98bd3f2f8dbe4d6383a08af85d33e6f", "published_at": "2026-08-05 11:48:38+00:00", "updated_at": "2026-08-05 12:25:39.962718+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["RunPod", "AlphaPebble Labs", "PyTorch", "CUDA", "HuggingFace", "RTX 3090", "RTX 4090", "A100"], "alternates": {"html": "https://wpnews.pro/news/cloud-training-on-runpod-a-field-guide-to-the-edge-cases", "markdown": "https://wpnews.pro/news/cloud-training-on-runpod-a-field-guide-to-the-edge-cases.md", "text": "https://wpnews.pro/news/cloud-training-on-runpod-a-field-guide-to-the-edge-cases.txt", "jsonld": "https://wpnews.pro/news/cloud-training-on-runpod-a-field-guide-to-the-edge-cases.jsonld"}}