mtwest 1 The Issue
Does anyone know of well written / produced training materials to help someone understand resource controls in user space? Given that multi-(human)user interactive servers aren’t the major focus of commercial IT these days, it’s a little hard to find good material on how to manage such things.
Over the past month, I have been trying piece together limits.conf
files for both pam_limits
and user-.slice.d
from bits scattered across multiple forums & docs pages and have found myself not understand the technology behind the various threshold variables as much as I’d like.
Like, I have stuff in files that works, BUT I don’t feel like I could explain it to anyone else very well. Also I steadfastly refuse to use stochastic text extruders for this sort of work, so please don’t regurgitate something you got from LLM + code generation harness of choice.
Some References
Linux Weekly News at https://lwn.net/ has been a resource I’ve followed for over two decades. It’s the best summary, look at the history of the development of cgroups v1 and v2 at https://lwn.net/Kernel/Index/#Memory_management-Control_groups.
1 Like
mtwest 3 While the LWN Kernel page definitely does give a lot of nice historical context to so many different components within the Linux kernel, I must disagree with it being a good summary of anything but very deep technical discussions of implementation questions, and even then.
I guess I am looking more for tutorials on how best to set resource flags, particular for human users, and the motivation behind each of the choices. Maybe that material just doesn’t exist and I need to write it myself.
Multi-tennant systems aren’t uncommon, but it’s maybe worth clarifying what that actually means to you. “commercial IT” has been tacked more user isolation and introduced something called namespaces, and started giving users their own mount, process, and device namespaces. Congratulations, you’ve now just invented containers.
Seriously though, a container is just a bunch of cgroups and namespaces in a trenchcoat. What are you looking for exactly?
mtwest 6 Purpose
I am trying to limit resource pressure on an HPC cluster submit host, especially as the number of users running LLM code-generator services with --dangerously-skip-permissions
grows weekly. So the hope is to set up a bunch of constraints via CgroupsV2 so I don’t have to, as often, play whack-a-mole with processes in user space
Config Best Practices
I have been able to cobble together a bunch of settings from conversations with others
[Slice]
CPUQuota=200%
IPAccounting=yes
; Memory management
MemoryAccounting=yes
MemoryMax=4G
MemoryHigh=3G
MemoryZSwapMax=1G
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=70%
ManagedOOMMemoryPressureDurationSec=20s
MemorySwapMax=1G
ManagedOOMSwap=kill
; Process count limitation
TasksAccounting=yes
TasksMax=256
; Local IO limits
IOAccounting=yes
IOWeight=10
Additional steps
But having found all of these recommendations, then I run into a bunch of additional work that needs to be done that no one mentioned before.
- installed
systemd-oomd
and start the service
-
how to set up
zswap -
turn on
PSI
in the kernel
And there are definitely more that aren’t immediately related to managing slice resource usage via cgroups.
mtwest 7 Each cluster maintainers will have a different tolerance to resource usage, so thresholds will be set differently. And that is fine. But I cannot be the only one wanting to manage user slices in these sorts of ways, right?
Maybe this is just an indication that I need to start writing up this material myself. Just am a bit boggled this sort of stuff doesn’t exist already someplace else.
mtwest 8 Persistence Issue
While I can with ease get zswap
working at runtime by editing
sys/module/zswap/parameters/enabled
trying to make it always come back up after a reboot is being more difficult.
Using grubby for config updates
Having used grubby before to add optional flag values to the used kernel, I figured this would be straightforward, as various webpages have indicated.
sudo grubby \
--update-kernel=/boot/vmlinuz-$(uname -r) \
--args="zswap.enabled=1"
The value is taken up but upon reboot, I cannot SSH to the machine. Now I don’t think it’s doing anything with SSH perse. Just that is the most obvious indicator that something went wrong. In trying to debug this, I get pointed at /etc/initramfs-tools/modules
but that doesn’t exist on my system. Also a few of these pages are a decade old and stuff shifts in all sorts of ways. Hence asking here if anyone knows more up-to-date info on how to make this work.
mtwest
9
RHEL systems don’t have the initramfs-tools
package as its a Debian-family thing. AFAICT, the RHEL equivalent is dracut
but now it’s a matter of determining what the equivalent set of commands is as most of online docs are for debian/ubuntu/etc systems. Sigh…