[Lab Notes] Kubernetes the Hard Way, For Real This Time (Step 03)

A developer documented step 03 of a manual "Kubernetes the Hard Way" homelab build on Proxmox, working through SSH root-login restrictions on Ubuntu cloud images and distributing a dedicated keypair to three nodes. The engineer discovered that Ubuntu's cloud images embed a forced command in /root/.ssh/authorized_keys that blocks direct root SSH access, so administration was switched to the ubuntu user. Hostnames were then set to server, node-0, and node-1 via a looped script over machines.txt.

Continuing my Kubernetes the Hard Way https://github.com/kelseyhightower/kubernetes-the-hard-way homelab build on Proxmox. Prerequisites and jumpbox setup steps 01-02 https://dev.to/lugerlogic/lab-notes-kubernetes-the-hard-way-for-real-this-time-steps-01-02-nog are already done, this covers step 03 only. Steps 01 and 02 were straightforward. Well, this one was a reminder that I'm doing this manually. No scripts or AI to automate all of the setup for me. Just me and the guide. And man, even if you have a guide but if you needed to tweak something in your setup. A whole bunch of problems pop up. It goes something like: "Guide says do it this way, but I'm using a different tool, so I do it this way instead. Now something's broken because that tool needed extra config. Now the next step in the guide needs adjusting too." It's like a row of light switches wired together. Flip one, and a few others flip with it. Now you've got to figure out which ones to switch back. But hey, thats exactly how Ops/Infra work is done to me at least . Anyway, here's how it went. Created machines.txt on the jumpbox, listing IP, FQDN, hostname, and pod subnet for each node: root@luger-VirtualBox:~/kubernetes-the-hard-way cat machines.txt XXX.XXX.XXX.XXX k8s-solo.home.lab k8s-solo XXX.XXX.XXX.XXX k8s-solo-w1.home.lab k8s-solo-w1 XXX.XXX.XXX.XXX/24 XXX.XXX.XXX.XXX k8s-solo-w2.home.lab k8s-solo-w2 XXX.XXX.XXX.XXX/24 IPs are masked cuz I'm this 🤏 close to wearing a tin foil hat... even if they're internal IPs Enabling root login: the guide assumes Debian, where root SSH login just needs to be turned on. On Ubuntu, root login is disabled by default too, so the fix was the same. Just edit sshd config and restart the service: bash ubuntu@k8s-solo:~$ sudo sed -i 's/^ PermitRootLogin. /PermitRootLogin yes/' /etc/ssh/sshd config ubuntu@k8s-solo:~$ sudo systemctl restart sshd Generating and distributing keys: I originally planned to use root like the guide suggests, but kept hitting a Please login as the user "ubuntu" rather than the user "root" prompt on every node . I didn't bother digging into it right away, so it wasn't until the end of this step that I actually tracked down why. Ubuntu's cloud images bake a forced command into /root/.ssh/authorized keys that blocks direct root SSH login and prints that message instead. Rather than fight it, I just switched to the ubuntu user for all admin access. So keys were copied to ubuntu@ on each node instead of root@ mailto:root@ . Also used a dedicated keypair instead of overwriting any existing id rsa: root@luger-VirtualBox:~/kubernetes-the-hard-way ssh-keygen Enter file in which to save the key /root/.ssh/id rsa : /root/.ssh/id rsa kthw root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do ssh-copy-id -i /root/.ssh/id rsa kthw.pub ubuntu@${IP} done < machines.txt Verified access to all three nodes: root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do ssh -n ubuntu@${IP} hostname done < machines.txt k8s-solo k8s-solo-w1 k8s-solo-w2 The guide sets each machine's hostname to match its role: server , node-0 , node-1 . I updated machines.txt accordingly and ran the rename commands against the ubuntu user instead of root : root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do CMD="sudo sed -i 's/^127.0.1.1. /127.0.1.1\t${FQDN} ${HOST}/' /etc/hosts" ssh -n ubuntu@${IP} "$CMD" ssh -n ubuntu@${IP} sudo hostnamectl set-hostname ${HOST} ssh -n ubuntu@${IP} sudo systemctl restart systemd-hostnamed done < machines.txt Confirmed: root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do ssh -n ubuntu@${IP} hostname --fqdn done < machines.txt server.kubernetes.local node-0.kubernetes.local node-1.kubernetes.local From this point on, the nodes are addressed as server , node-0 , and node-1 per the guide's convention. Not the earlier k8s-solo / k8s-solo-w1 / k8s-solo-w2 names. No surprises here, followed the guide as-is. Built a hosts file on the jumpbox and appended it to the jumpbox's own /etc/hosts : root@luger-VirtualBox:~/kubernetes-the-hard-way echo " Kubernetes The Hard Way" hosts root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do echo "${IP} ${FQDN} ${HOST}" hosts done < machines.txt root@luger-VirtualBox:~/kubernetes-the-hard-way cat hosts /etc/hosts Verified hostname resolution works locally: root@luger-VirtualBox:~/kubernetes-the-hard-way for host in server node-0 node-1; do ssh ubuntu@${host} hostname; done server node-0 node-1 /etc/hosts entries to the remote machines Had to adjust the guide's command here too, same reason as before ubuntu user instead of root , plus a gotcha with sudo and output redirection: root@luger-VirtualBox:~/kubernetes-the-hard-way while read IP FQDN HOST SUBNET; do scp hosts ubuntu@${HOST}:~/ ssh -n ubuntu@"${IP}" "sudo sh -c 'cat hosts /etc/hosts'" done < machines.txt Interesting nugget of info I learned here: sudo somecommand file doesn't work the way you'd expect, sudo only elevates the command itself, not the redirection. The redirection is set up by your current shell before sudo ever runs, so it still tries to write as your regular user. Wrapping the whole thing in sudo sh -c '...' fixes it, since now the shell doing the redirecting is the one running as root. Step 03 set up the actual infrastructure: root/admin SSH access between the jumpbox and all three nodes, and consistent hostnames server , node-0 , node-1 so every machine can address the others by name instead of IP.