Resource contraints with cgroups & pam on a multi-user system

A system administrator on the Level1Techs forum is seeking training materials to understand and configure cgroups v2 and PAM resource limits for multi-user HPC systems, citing a lack of comprehensive tutorials. The admin shared a working configuration using systemd slice settings to limit CPU, memory, I/O, and process counts, and noted the need for additional steps like installing systemd-oomd and enabling PSI.

mtwest https://forum.level1techs.com/u/mtwest 1 The Issue Does anyone know of well written / produced training materials to help someone understand resource controls in user space? Given that multi- human user interactive servers aren’t the major focus of commercial IT these days, it’s a little hard to find good material on how to manage such things. Over the past month, I have been trying piece together limits.conf files for both pam limits and user-.slice.d from bits scattered across multiple forums & docs pages and have found myself not understand the technology behind the various threshold variables as much as I’d like. Like, I have stuff in files that works, BUT I don’t feel like I could explain it to anyone else very well. Also I steadfastly refuse to use stochastic text extruders for this sort of work, so please don’t regurgitate something you got from LLM + code generation harness of choice. Some References Linux Weekly News at https://lwn.net/ https://lwn.net/ has been a resource I’ve followed for over two decades. It’s the best summary, look at the history of the development of cgroups v1 and v2 at https://lwn.net/Kernel/Index/ Memory management-Control groups https://lwn.net/Kernel/Index/ Memory management-Control groups . 1 Like mtwest https://forum.level1techs.com/u/mtwest 3 While the LWN Kernel page definitely does give a lot of nice historical context to so many different components within the Linux kernel, I must disagree with it being a good summary of anything but very deep technical discussions of implementation questions, and even then. I guess I am looking more for tutorials on how best to set resource flags, particular for human users, and the motivation behind each of the choices. Maybe that material just doesn’t exist and I need to write it myself. Multi-tennant systems aren’t uncommon, but it’s maybe worth clarifying what that actually means to you. “commercial IT” has been tacked more user isolation and introduced something called namespaces, and started giving users their own mount, process, and device namespaces. Congratulations, you’ve now just invented containers. Seriously though, a container is just a bunch of cgroups and namespaces in a trenchcoat. What are you looking for exactly? mtwest https://forum.level1techs.com/u/mtwest 6 Purpose I am trying to limit resource pressure on an HPC cluster submit host, especially as the number of users running LLM code-generator services with --dangerously-skip-permissions grows weekly. So the hope is to set up a bunch of constraints via CgroupsV2 so I don’t have to, as often, play whack-a-mole with processes in user space Config Best Practices I have been able to cobble together a bunch of settings from conversations with others Slice CPUQuota=200% IPAccounting=yes ; Memory management MemoryAccounting=yes MemoryMax=4G MemoryHigh=3G MemoryZSwapMax=1G ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=70% ManagedOOMMemoryPressureDurationSec=20s MemorySwapMax=1G ManagedOOMSwap=kill ; Process count limitation TasksAccounting=yes TasksMax=256 ; Local IO limits IOAccounting=yes IOWeight=10 Additional steps But having found all of these recommendations, then I run into a bunch of additional work that needs to be done that no one mentioned before. - installed systemd-oomd and start the service - how to set up zswap - turn on PSI in the kernel And there are definitely more that aren’t immediately related to managing slice resource usage via cgroups. mtwest https://forum.level1techs.com/u/mtwest 7 Each cluster maintainers will have a different tolerance to resource usage, so thresholds will be set differently. And that is fine. But I cannot be the only one wanting to manage user slices in these sorts of ways, right? Maybe this is just an indication that I need to start writing up this material myself. Just am a bit boggled this sort of stuff doesn’t exist already someplace else. mtwest https://forum.level1techs.com/u/mtwest 8 Persistence Issue While I can with ease get zswap working at runtime by editing sys/module/zswap/parameters/enabled trying to make it always come back up after a reboot is being more difficult. Using grubby for config updates Having used grubby https://linux.die.net/man/8/grubby before to add optional flag values to the used kernel, I figured this would be straightforward, as various webpages have indicated. sudo grubby \ --update-kernel=/boot/vmlinuz-$ uname -r \ --args="zswap.enabled=1" The value is taken up but upon reboot, I cannot SSH to the machine. Now I don’t think it’s doing anything with SSH perse. Just that is the most obvious indicator that something went wrong. In trying to debug this, I get pointed at /etc/initramfs-tools/modules but that doesn’t exist on my system. Also a few of these pages are a decade old and stuff shifts in all sorts of ways. Hence asking here if anyone knows more up-to-date info on how to make this work. mtwest https://forum.level1techs.com/u/mtwest 9 RHEL systems don’t have the initramfs-tools package as its a Debian-family thing. AFAICT, the RHEL equivalent is dracut but now it’s a matter of determining what the equivalent set of commands is as most of online docs are for debian/ubuntu/etc systems. Sigh…