{"slug": "lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03", "title": "[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 03)", "summary": "A developer documented step 03 of a manual \"Kubernetes the Hard Way\" homelab build on Proxmox, working through SSH root-login restrictions on Ubuntu cloud images and distributing a dedicated keypair to three nodes. The engineer discovered that Ubuntu's cloud images embed a forced command in /root/.ssh/authorized_keys that blocks direct root SSH access, so administration was switched to the ubuntu user. Hostnames were then set to server, node-0, and node-1 via a looped script over machines.txt.", "body_md": "Continuing my [Kubernetes the Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) homelab build on Proxmox. Prerequisites and jumpbox setup ([steps 01-02](https://dev.to/lugerlogic/lab-notes-kubernetes-the-hard-way-for-real-this-time-steps-01-02-nog)) are already done, this covers step 03 only.\n\nSteps 01 and 02 were straightforward. Well, this one was a reminder that I'm doing this manually. No scripts or AI to automate all of the setup for me. Just me and the guide. And man, even if you have a guide but if you needed to tweak something in your setup. A whole bunch of problems pop up. It goes something like:\n\n\"Guide says do it this way, but I'm using a different tool, so I do it this way instead. Now something's broken because that tool needed extra config. Now the next step in the guide needs adjusting too.\"\n\nIt's like a row of light switches wired together. Flip one, and a few others flip with it. Now you've got to figure out which ones to switch back. But hey, thats exactly how Ops/Infra work is done(to me at least).\n\nAnyway, here's how it went.\n\nCreated `machines.txt` on the jumpbox, listing IP, FQDN, hostname, and pod subnet for each node:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# cat machines.txt\nXXX.XXX.XXX.XXX k8s-solo.home.lab k8s-solo\nXXX.XXX.XXX.XXX k8s-solo-w1.home.lab k8s-solo-w1 XXX.XXX.XXX.XXX/24\nXXX.XXX.XXX.XXX k8s-solo-w2.home.lab k8s-solo-w2 XXX.XXX.XXX.XXX/24\n```\n\n(IPs are masked cuz I'm this 🤏 close to wearing a tin foil hat... even if they're internal IPs)\n\n**Enabling root login:** the guide assumes Debian, where root SSH login just needs to be turned on. On Ubuntu, root login is disabled by default too, so the fix was the same. Just edit `sshd_config` and restart the service:\n\n``` bash\nubuntu@k8s-solo:~$ sudo sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config\nubuntu@k8s-solo:~$ sudo systemctl restart sshd\n```\n\n**Generating and distributing keys:** I originally planned to use root like the guide suggests, but kept hitting a `Please login as the user \"ubuntu\" rather than the user \"root\" prompt on every node`. I didn't bother digging into it right away, so it wasn't until the end of this step that I actually tracked down why. Ubuntu's cloud images bake a forced command into /root/.ssh/authorized_keys that blocks direct root SSH login and prints that message instead. Rather than fight it, I just switched to the ubuntu user for all admin access. So keys were copied to ubuntu@ on each node instead of [root@](mailto:root@). Also used a dedicated keypair instead of overwriting any existing id_rsa:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# ssh-keygen\nEnter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa_kthw\n\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n  ssh-copy-id -i /root/.ssh/id_rsa_kthw.pub ubuntu@${IP}\ndone < machines.txt\n```\n\nVerified access to all three nodes:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n  ssh -n ubuntu@${IP} hostname\ndone < machines.txt\nk8s-solo\nk8s-solo-w1\nk8s-solo-w2\n```\n\nThe guide sets each machine's hostname to match its role: `server`, `node-0`, `node-1`. I updated `machines.txt` accordingly and ran the rename commands against the `ubuntu` user instead of `root`:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n    CMD=\"sudo sed -i 's/^127.0.1.1.*/127.0.1.1\\t${FQDN} ${HOST}/' /etc/hosts\"\n    ssh -n ubuntu@${IP} \"$CMD\"\n    ssh -n ubuntu@${IP} sudo hostnamectl set-hostname ${HOST}\n    ssh -n ubuntu@${IP} sudo systemctl restart systemd-hostnamed\ndone < machines.txt\n```\n\nConfirmed:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n  ssh -n ubuntu@${IP} hostname --fqdn\ndone < machines.txt\nserver.kubernetes.local\nnode-0.kubernetes.local\nnode-1.kubernetes.local\n```\n\nFrom this point on, the nodes are addressed as `server`, `node-0`, and `node-1` per the guide's convention. Not the earlier `k8s-solo` / `k8s-solo-w1` / `k8s-solo-w2` names.\n\nNo surprises here, followed the guide as-is. Built a `hosts` file on the jumpbox and appended it to the jumpbox's own `/etc/hosts`:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# echo \"# Kubernetes The Hard Way\" >> hosts\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n    echo \"${IP} ${FQDN} ${HOST}\" >> hosts\ndone < machines.txt\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# cat hosts >> /etc/hosts\n```\n\nVerified hostname resolution works locally:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# for host in server node-0 node-1; do ssh ubuntu@${host} hostname; done\nserver\nnode-0\nnode-1\n```\n\n`/etc/hosts` entries to the remote machines\nHad to adjust the guide's command here too, same reason as before (`ubuntu` user instead of `root`), plus a gotcha with `sudo` and output redirection:\n\n```\nroot@luger-VirtualBox:~/kubernetes-the-hard-way# while read IP FQDN HOST SUBNET; do\n  scp hosts ubuntu@${HOST}:~/\n  ssh -n ubuntu@\"${IP}\" \"sudo sh -c 'cat hosts >> /etc/hosts'\"\ndone < machines.txt\n```\n\n**Interesting nugget of info I learned here:** `sudo somecommand >> file` doesn't work the way you'd expect, `sudo` only elevates the command itself, not the `>>` redirection. The redirection is set up by your *current* shell before `sudo` ever runs, so it still tries to write as your regular user. Wrapping the whole thing in `sudo sh -c '...'` fixes it, since now the shell doing the redirecting is the one running as root.\n\nStep 03 set up the actual infrastructure: root/admin SSH access between the jumpbox and all three nodes, and consistent hostnames (`server`, `node-0`, `node-1`) so every machine can address the others by name instead of IP.", "url": "https://wpnews.pro/news/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03", "canonical_source": "https://dev.to/lugerlogic/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03-3bc2", "published_at": "2026-09-10 16:04:45+00:00", "updated_at": "2026-09-10 16:15:02.066576+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Kubernetes", "Proxmox", "Ubuntu", "Debian", "Kelsey Hightower"], "alternates": {"html": "https://wpnews.pro/news/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03", "markdown": "https://wpnews.pro/news/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03.md", "text": "https://wpnews.pro/news/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03.txt", "jsonld": "https://wpnews.pro/news/lab-notes-kubernetes-the-hard-way-for-real-this-time-step-03.jsonld"}}