{"slug": "sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3", "title": "Sched_ext: A sketch of the scx_lavd scheduler (Part 3)", "summary": "The scx_lavd BPF CPU scheduler, built on sched_ext, is now deployed to Valve's Steam Frame VR headset and to Meta's datacenters, the scheduler's developer reported in the third post of a blog series based on a Kernel Recipes 2026 talk given with Gavin Guo. Since its first commit to the scx repository in March 2024, scx_lavd has grown to about 12,100 lines, 904 commits and 42 contributors, with development sponsored by Valve and Meta. The scheduler infers which threads are latency-critical from how they wait, wake others and run, rather than requiring annotation or configuration, and originated in 2022-2023 work for Valve's Steam Deck, where strace logs showed about 80% of system calls were futex.", "body_md": "# sched_ext: a sketch of the scx_lavd scheduler (Part 3)\n\nThis is the third blog post about sched_ext. It’s been a while since my\nlast blog post. I have been busy developing the `scx_lavd` scheduler. I am\nhappy to share that `scx_lavd` is now deployed to the\n[Steam Frame](https://store.steampowered.com/hardware/steamframe), a VR headset,\nand to [Meta’s datacenters](https://lpc.events/event/19/contributions/2099/).\n\nToday, let me sketch `scx_lavd`. This post is\nbased on the talk Gavin Guo and I gave at Kernel Recipes 2026,\n[“**LAVD: From Gaming to Server Fleets — a BPF Scheduler for Cloud, ML, and\nEdge**”](https://kernel-recipes.org/en/2026/schedule/lavd-from-gaming-to-server-fleets-a-bpf-scheduler-for-cloud-ml-and-edge/).\n\n[`scx_lavd`](https://github.com/sched-ext/scx/tree/main/scheds/rust/scx_lavd)\nstands for **L** atency-criticality **A** ware **V** irtual **D** eadline. It\nis a Linux CPU scheduler written in BPF on top of sched_ext, and it works\nout which of your threads are urgent **without anyone telling it**. Since its\nfirst commit in the scx repository in March 2024, it has grown to about 12,100 lines, 904\ncommits, and 42 contributors. The development has been sponsored by Valve\nand Meta.\n\nThe whole design is one idea and four consequences of taking it seriously:\n\n- **Which task is urgent?** Inferred from how it waits, wakes others, and\nruns. Nothing is annotated, nothing is configured.\n- **When does it run, and for how long?** That one number becomes a virtual\ndeadline, a time slice, and the right to preempt somebody.\n- **On which CPU?** Queues per cache domain, balanced by capacity rather\nthan by headcount, and a warm task left where it is.\n- **How few cores — and which ones?** Core compaction sizes the active set.\nThe kernel’s energy model chooses its members.\n- **How much may one tenant use?** cgroup bandwidth control, written as a\nlibrary any BPF scheduler can link.\n\nThe first two are the scheduler. The last three are what it took to run on real machines — and they are the reason the same code ends up in a handheld and in a server fleet. Let’s get started!\n\n## 1. It started with strace and a hunch [#](#1-it-started-with-strace-and-a-hunch)\n\nThe project started around the end of 2022 and the beginning of 2023, for\nValve’s handheld gaming device, the Steam Deck. SteamOS is a Linux\ndistribution that runs unmodified x86 Windows games on top of Proton and\nWine. At that time, SteamOS used CFS, the default scheduler back then, and\nusers complained about latency spikes and frame-time spikes from time to\ntime. We did our best to tune the scheduler’s debugfs knobs, which was not\ngreat. The more serious problem was that there was no universal setting of\nthe knobs that works well for all kinds of games. So we asked a blunter\nquestion: **can we build a gaming-optimized scheduler on sched_ext?**\n\nWhen I started working on this project, I had little knowledge of games and\nhow game engines work. So I ran games on SteamOS and collected `strace`\nlogs. About **80% of system calls were `futex`**. Most of what `strace` saw\nwas Wine synchronization: Wine maps Windows synchronization primitives onto\nfutexes, so what a game mostly does is **tasks waiting on each other**. When\nI traced who wakes whom (using Igalia’s\n[VaporMark](https://github.com/Igalia/vapormark)), a game looked like a\ngraph, not like one hot loop. For example, four worker threads all wake the\n`dxvk-cs` thread, which in turn wakes `dxvk-submit`; the `winepulse_main`\nand `winepulse_timer` audio threads wake each other.\n\nThink about what happens when you press a button on a gaming device. Your\ncharacter moves, a sound is generated, and the screen is updated. On the\ngraphics side, the game thread does some work and passes a request to a\ngraphics thread, which finally submits the frame update to the GPU. The\naudio side is similar: the game generates audio, which goes through an\naudio task in the Wine layer, then to PipeWire, and finally to the speaker.\nTo finish **one logical job**, many tasks have to interact with each other.\nThat was one of the key findings.\n\nDigging further, I found a few more characteristics of game workloads:\n\n- \nAbout 70-75% of scheduling in a game is initiated by a blocking system call ( `epoll` ,`pipe_read` ,`futex_wait` ). Only 25-30% is caused by\npreemption or the timer tick. So it makes sense to instrument the**sleep\nand wake edges** , not the tick.\n- \nAbout 300 tasks are scheduled while a game runs, ~90% of them never exit, and the top 30-40 tasks account for 95% of all scheduling.\n- \nCPU utilization is 65-95%, never 100%. The machine is busy but not out of capacity, so **stuttering is an ordering problem, not a throughput\nproblem** .\n\nIn short, a game is **communication-intensive**, **short-running** (most\ntasks run less than 1 msec at a time), and **semi-periodic**, so past\nbehavior predicts future behavior. The whole design of LAVD falls out of\nthese observations. (sched_ext itself reached mainline in 6.12, more than a\nyear later.)\n\n## 2. Average throughput is a lie your users can feel [#](#2-average-throughput-is-a-lie-your-users-can-feel)\n\nA steady 144 FPS and 144 FPS with one 40 msec hitch per second have the\nsame average, but only one of them is playable. I learned this the hard\nway: my first BPF scheduler sometimes hit 120 FPS, then dropped to 10 the\nvery next moment. Its average looked fine, but its 1% low FPS — the tail\nlatency, in gamers’ terms — collapsed, and the game felt jittery. The bad\nframes were spent **waiting in a run queue**, not rendering.\n\nA throughput-oriented scheduler cannot tell a game’s input, render, or\naudio thread apart from a compile job, and `nice` cannot help: it controls\n**how much** CPU time a task gets, not **when**. LAVD is a virtual-deadline\nscheduler like EEVDF, but with one key difference: it automatically infers\nhow **urgent** each task is — its **latency criticality** — and uses that to\ndecide the deadline, the time slice, and the CPU.\n\n## 3. From a handheld to a VR headset to a datacenter [#](#3-from-a-handheld-to-a-vr-headset-to-a-datacenter)\n\nThe development of LAVD went through three phases, and each phase brought new hardware that forced new features:\n\n1. \n**Steam Deck.** Initially, things were pretty simple. We focused only on\ngaming workloads on the Steam Deck, whose hardware is simple: an AMD x86\nCPU with 8 CPUs and a single LLC domain. So the development effort focused\non how to precisely infer urgency without any user annotation, and how to\nuse it in calculating the virtual deadline and the time slice.\n2. \n**Steam Frame.** Once that part matured, Valve started developing the\nSteam Frame, a VR headset that runs unmodified x86 Windows games on a\nQualcomm Snapdragon ARM CPU through Proton, Wine, and the[FEX](https://fex-emu.com/) binary translation layer. The Snapdragon is a\ntypical heterogeneous system with little, medium, and big cores. Around\nthat time, we introduced the concept of**compute domains** , so that\ndifferent types of cores are treated differently — and power efficiency\nbecame a first-class concern on a battery-backed device.\n3. \n**Datacenter.** Later, we found that our original design — minimizing\ntail latency — is also pretty useful for server workloads, like Meta’s.\nSo we expanded our scope to datacenter machines with 200 CPUs and 10 to 20\nLLC domains, with additional requirements such as cgroup bandwidth control.\n\nInitially, it was a one-person project; I was the only developer. So far, 42+ contributors have contributed to LAVD, and Gavin Guo and David Dai from Meta have been actively involved in the development.\n\n## 4. Which task is urgent? Latency criticality [#](#4-which-task-is-urgent-latency-criticality)\n\n### 4.1. Waker and wakee are the keys [#](#4-1-waker-and-wakee-are-the-keys)\n\nThe honest way to know which task is urgent is to ask the developer —\nlatency-nice, a cgroup knob, `sched_setattr()`. That works for exactly the\nsoftware you control. It does not work for a Windows game running under\nProton, a closed-source anti-cheat module, or a 30-year-old daemon.\n\nSo LAVD **infers** it. From behavior. At runtime. For free. The bet is that\n**how a task communicates tells you what it is for**. A thread that wakes up\nevery 5 msec and runs for 80 µsec is an audio mixer whether it says so or\nnot.\n\nEvent-driven systems are graphs. Every wake-up has two ends: a **waker**\nand a **wakee**. Consider a chain where task A wakes B, and B wakes C. If\nyou delay B, you delay C — and everything C wakes after that. The task in\nthe middle of a chain is the most expensive place to introduce a delay.\n\n### 4.2. Three behavioral signals [#](#4-2-three-behavioral-signals)\n\nLAVD measures three signals per task — two frequencies and one duration. That’s the whole sensor set:\n\n- \n**`wait_freq`** : How often the task sleeps waiting on something external\n— I/O, a timer, a packet, a mutex. It is measured at`ops.quiescent()` as\nan EWMA (exponentially weighted moving average). This is a**consumer** signal.\n- \n**`wake_freq`** : How often the task wakes somebody else. It is charged to\nthe waker, not the wakee, at`ops.runnable()` . A thread that unblocks\nothers is on somebody’s critical path. This is a**producer** signal.\n- \n**`runtime`** : The average invariant runtime per schedule. A 60 µsec task\ncannot absorb a 3 msec delay, but a 40 msec batch job barely notices. This\nis an**inverse** signal: the shorter, the more critical.\n\nBoth frequencies are clamped at 100,000 per second (i.e., a 10 µsec interval).\n\n### 4.3. Turning behavior into a number [#](#4-3-turning-behavior-into-a-number)\n\nThe intuition is simple:\n\n```\nlat_cri ∝ (wait_freq × wake_freq) / runtime\n```\n\nA task that waits often, that wakes others often, and that runs only\nbriefly sits in the middle of a chain — so it scores higher, and the\nscheduler runs it more urgently. In the code (`calc_lat_cri()` in\n`lat_cri.bpf.c`), it is log-compressed and then squared:\n\n```\nlat_cri = ( log2(wait_ft × wake_ft) + log2(runtime_ft × weight_ft) )^2\n\n  wait_ft    = min(wait_freq, 100k) + 1\n  wake_ft    = min(wake_freq, 100k) + 1\n  runtime_ft = (1 sec - avg_runtime) / 500 usec + 1   <- reverse runtime factor\n  weight_ft  = weight × boost + 1\n```\n\nIt looks a bit more complicated than the intuition, but the extra parts are nothing more than shaping the curve so that it works well for real workloads. Every piece of it is deliberate:\n\n- \n**Why multiply `wait` by `wake`, not add them?** The product is large\nonly when**both** are large, so it rewards sitting in the middle of a chain\nrather than at either end.\n- \n**Why `log2`?** A spinning thread wakes 100,000 times a second; a cron\njob wakes twice an hour. Raw values would let one pathological thread\ndominate every comparison.\n- \n**Why square?** Logs compress too well. Squaring re-expands the top of\nthe range so that genuinely urgent tasks separate from merely busy ones.\n- \n**What is `weight_ft`?** It keeps` nice` values meaningful, and it adds a\nboost ladder: 128 for a lock holder, 256 and 512 for medium and high\nboosts, and 1024 for kernel tasks and synchronous wake-ups.\n\n### 4.4. Urgency is contagious: latency criticality inheritance [#](#4-4-urgency-is-contagious-latency-criticality-inheritance)\n\nTasks do not form isolated entities; they form a connected (and often\ncyclic) graph carrying **one job**. Think about a key press: input interrupt\n→ input thread → game logic → render submit → display refresh → frame.\nChains start at an input interrupt and end at the display refresh\ninterrupt. Nothing on that path should wait its turn.\n\nSo LAVD propagates urgency along the graph in both directions:\n\n- \n**Forward propagation** : A waker lends some of its urgency to its wakee,\nso the whole chain keeps its**momentum** going forward.\n- \n**Backward propagation** : A highly latency-critical wakee lends some of\nits urgency to its waker if the waker is less critical than the wakee. So\nnext time, the waker gets scheduled sooner and won’t keep the wakee\nwaiting.\n\nInheritance must be bounded; otherwise, a cycle where A wakes B and B wakes\nA would inflate both without limit. So inheritance only applies when\n`lat_cri_waker + lat_cri_wakee` exceeds `2 × lat_cri`. The giver\ncontributes at most 12.5% of the surplus (`>> 3`), the receiver gains at\nmost 25% of its own score (`>> 2`), and both accumulators are reset\nwhenever `lat_cri` is recomputed. Being woken once does not make you urgent\nforever.\n\n## 5. When does it run, and for how long? [#](#5-when-does-it-run-and-for-how-long)\n\n### 5.1. The virtual deadline [#](#5-1-the-virtual-deadline)\n\nEverything the scheduler has learned collapses into one number, the\n**virtual deadline**, which is the only ordering key in the queue\n(`calc_when_to_run()`):\n\n```\ndeadline_delta = adjusted_runtime × greedy_penalty / lat_cri\nvdeadline      = logical_clk - compete_window + deadline_delta\n```\n\nRunnable tasks sit in a vtime-ordered DSQ, and `ops.dispatch()` pops the\none with the smallest deadline. The animation below shows three\nlatency-critical tasks and one batch task competing on one CPU: the\ncritical tasks keep winning, but each run nudges the logical clock forward,\nand eventually the batch task wins anyway.\n\n- \n`lat_cri`**divides** . Ten times more critical means one tenth of the\ndistance to wait.\n- \n`adjusted_runtime` is 5 msec plus the invariant time burned since the\nlast sleep (capped at 5 msec), so a task that just woke up beats one that\nhas been grinding.\n- \n`greedy_penalty` ranges from 1.0× to 2.0×, derived from the**fairness\nlag** — the gap between the CPU time a task should have received and what it\nactually got. Over-served tasks get pushed back.\n- \n`logical_clk` is not real time. It is a global cursor, advanced by\ncompare-and-swap in`ops.running()` , that steps toward the deadline of\nwhichever task actually ran. No wall-clock time appears anywhere in the\nordering.\n- \nNew arrivals start one `compete_window` behind the cursor, so a fresh\nwake-up genuinely competes. The window is not constant: once the queued\ntasks outnumber the online CPUs,`calc_compete_window()` shrinks it by`nr_online_cpus / nr_queued` . Otherwise, with many tasks waiting, most\ndeadlines would land behind the cursor and could not advance it, so the\nclock would stall and the waiting tasks would stop aging.\n\nDoes the least critical task starve? No. It loses every round until the\nlogical clock drifts past its deadline, and then it wins anyway. **Priority\nthat decays into fairness.**\n\n### 5.2. Time slice [#](#5-2-time-slice)\n\nThe virtual deadline decides **when**. The time slice decides **how long**:\n\n```\nslice = (targeted latency 10 msec × nr_active_cpus) / nr_queued_tasks\n        clamped to [500 usec, 5 msec]\n```\n\nThe goal is that every queued task gets a turn within 10 msec, served\nacross all the **active** CPUs (that’s the core compaction set — more on it\nlater). The more tasks are queued, the shorter the slices. Both clamps\nmatter: without the 5 msec ceiling, a lone task owns the whole window;\nwithout the 500 µsec floor, a loaded machine context-switches more than it\nruns. The result is also smoothed, so the slice drifts rather than jumping\neach tick.\n\nThere are a few adjustments on top:\n\n- \n**Slice boosting.** If a task’s average runtime already exceeds the slice\n(i.e., it**keeps getting cut off** ), LAVD boosts its slice. When the system\nis not heavily loaded, the slice can be extended up to 500 msec. Under\nload, only tasks more latency-critical than average get a boost, capped at\n2× the slice. Why? A long-running task in the middle of a chain that gets\ncut off at slice exhaustion delays everything downstream of it — that’s\nwhere tail latency comes from. It is not a reward for compute-bound work.\nAnd a more critical wake-up can still shorten a boosted slice.\n- \n**Fresh forks get half a slice.** Nothing is known about a new task yet,\nso information is cheap — and it brakes fork bombs.\n- \n**Pinned tasks.** When a pinned task is waiting, every task on that CPU\ndrops to`min(pinned_slice, slice)` (`--pinned-slice-us` , 5 msec by\ndefault).\n\n### 5.3. Preemption [#](#5-3-preemption)\n\nIdeally, the N most latency-critical tasks hold the N CPUs. So finding the\nCPU running the least critical task — the preemption victim — is really a\n**load balancing** problem, the classic “least-loaded server” problem. That’s\nwhy LAVD uses the **power of two random choices** (`preempt.bpf.c`):\n\n- \n**Two random probes, never a full scan.** A random start in a random\ndirection stops every CPU from converging on the same victim (the\nthundering herd).\n- \n**Only inside the task’s own compute domain** , among the CPUs it is\nallowed to run on.\n- \n**A candidate must be losing on both counts** : a lower`lat_cri`**and** a\nlater estimated stopping time. Otherwise, you would evict a task that was\nabout to release the CPU anyway.\n- \n**No IPI.**`scx_bpf_kick_cpu()` is avoided on purpose; IPIs are\nexpensive, and their cost varies with processor architecture and\ngeneration. Instead, the victim’s time slice is shortened (to 1, not 0,\nbecause the core replaces a zero slice with`SCX_SLICE_DFL` ), so it yields\nat its next scheduling point.\n- \n**Never a lock holder** , and only the top ~1.56% most latency-critical\ntasks (`--preempt-shift=6` ) may preempt at all.\n\nTwo CPUs may still pick the same victim. That is uncoordinated on purpose — the race is rare, and synchronizing would cost more. The victim’s estimated stopping clock is compare-and-swapped to zero, so whoever wins the swap preempts, and the victim is not chosen twice.\n\n### 5.4. Who holds the lock? [#](#5-4-who-holds-the-lock)\n\nPreemption has a tricky corner: a lock holder should not be preempted. If a\nlock holder is preempted and goes back to the run queue, a latency-critical\ntask that needs the lock has to wait for the lock holder to be scheduled\nback **and then** to finish its critical section and release the lock. A\nlatency-critical task is only as fast as whoever holds the lock it wants.\nSo LAVD traces futex operations to find the lock holder, then protects and\nboosts it: a lock holder is never preempted, and it gets a +128 weight\nboost on its next deadline after it releases the lock.\n\nThe hard part is knowing **cheaply**. A well-optimized user-space lock never\nenters the kernel when it is uncontended — NPTL skips `futex_wait()` if\nnobody is waiting. Uprobes on `pthread_mutex_lock()` would be exact, but\nthey trap into the kernel on every acquisition and would destroy the very\noptimization being measured. So LAVD uses the kernel-side view, which is\nincomplete **by design**, with four rules:\n\n1. \nNo `futex_wait()` means nobody is waiting, so there is nothing worth\nboosting. The missing information is exactly the information you do not\nneed.\n2. \nRepeated `futex_wait()` calls before a wake are spurious wake-up\nretries, so all but the first are ignored.\n3. \nNo `futex_wake()` within one time slice means we assume the lock was\nreleased. A critical section longer than a slice is not one you want to\nboost anyway.\n4. \nFutex addresses are deliberately not distinguished, to keep tracing cheap.\n\nThe attach point is chosen by cost. Measured per futex operation on an AMD\nRyzen 9 PRO 6950H: `fentry` costs 48 nsec, `fexit` 76 nsec, and a\ntracepoint 130 nsec. `fentry` is the fastest but breaks if the function is\ninlined; a tracepoint is a stable ABI; `fexit` is needed where the return\nvalue matters. The user-space part picks based on what the running kernel\nexposes.\n\n## 6. On which CPU? Topology [#](#6-on-which-cpu-topology)\n\n### 6.1. Dispatch queue layout [#](#6-1-dispatch-queue-layout)\n\nLAVD groups CPUs into **compute domains**. A compute domain is one unique\ncombination of LLC × core type × NUMA node (up to 128 of them). Aligning\nthe queue to the LLC is the point: tasks queued together are tasks that\nalready share a cache.\n\nThere are four kinds of queues: three that the BPF scheduler creates, and\none that the kernel owns. The **CPDOM** DSQ is the domain’s main queue,\nordered by virtual deadline. The **CPDOM_TURB** DSQ is an overflow tier for\nlatency-tolerant work (more on it below). The per-**CPU** DSQ holds pinned\ntasks that cannot migrate.\n\nNote that enqueueing and dispatching are asynchronous. `ops.enqueue()` only\nplaces a task in a DSQ; nothing runs it yet. The kernel calls\n`ops.dispatch()` only when a CPU’s local DSQ has run empty, and LAVD then\nmoves one task over with `scx_bpf_dsq_move_to_local()`. There is a\nshortcut, though: if the target CPU is idle with nothing queued, the task is\ndispatched directly to the local DSQ. At that load there is nothing to\narbitrate, so paying for deadline ordering would be a waste.\n\n### 6.2. Load balancing: stealer vs. stealee [#](#6-2-load-balancing-stealer-vs-stealee)\n\nSince every domain has one DSQ shared by its CPUs, the **domain**, not the\nCPU, is the unit of load balancing. Every 10 msec, each domain is\nclassified from scratch:\n\n- \n**Stealer domain** : An underloaded domain. It pulls tasks from stealees\nto keep its own CPUs busy.\n- \n**Stealee domain** : An overloaded domain, carrying more queued load than\nits fair share. Stealers may pull from it.\n- \nInside the band, a domain is neither.\n\nThe band narrows as the system loads up: ±25% when idle, ±12.5% at mild\nload, and ±6.25% when busy, so a busy system reacts to smaller imbalances.\nWhen a CPU’s local DSQ drains, `ops.dispatch()` tries, in order: (1) if\nthis domain is a stealer, steal from a remote stealee; (2) take from the\nlocal domain’s DSQ — the cache-warm common case; (3) force-steal from any\nnearby DSQ rather than let the CPU go idle. Balancing goes **first** on\npurpose. If the local queue were consulted first, a stealer would never get\naround to stealing while it still had anything of its own to run, so a\nmeasured imbalance would persist.\n\nWhich remote domain does a CPU try first? Each domain keeps its neighbors\nin a **neighboring order**, computed once at load time. Neighbors are first\ngrouped by distance: a domain of a different core type is the farthest,\nthen one on a different NUMA node, then one with a different LLC. Within\nthe same distance, neighbors are sorted in **circular distance order**\naround the domain’s own ID — the closest IDs first, alternating in both\ndirections.\n\nTake an AMD EPYC system with two NUMA nodes, each with eight CCXs. Each CCX has its own L3 cache, so each one is a compute domain: cpdom0–7 on node 0 and cpdom8–15 on node 1. cpdom0 first visits the other CCXs on its own node in the order 1, 7, 2, 6, 3, 5, 4, and only then the CCXs on the other node: 8, 15, 9, 14, 10, 13, 11, 12. cpdom1 visits 0, 2, 3, 7, 4, 6, 5 on its own node, then the other node. Compared with stealing in random order, this keeps task migration within the same NUMA node and, within it, in a small neighborhood of CCXs (e.g., 0, 1, and 7), which increases the chance of cache locality and reduces cacheline bouncing.\n\n### 6.3. What does “load” mean? [#](#6-3-what-does-load-mean)\n\nLoad is **not** the number of runnable tasks. A domain with two heavy tasks\nis busier than one with ten small ones, and counting tasks gets that\nexactly backwards. In LAVD:\n\n```\nload             = queued_load_invr + util_invr\nqueued_load_invr = Σ size_of(task), over everything queued in the domain\nutil_invr        = how busy the domain's CPUs are right now\n```\n\n`size_of(task)` is the task’s **duty cycle**: the fraction of recent time the\ntask has been runnable rather than sleeping, scaled to [0, 1024] and\ntracked as a running average with a 128 msec half-life. So a CPU-bound task\nthat never sleeps counts as 1024, while an I/O-bound task that runs briefly\nand then blocks counts as much less.\n\n`invr` stands for **invariant** runtime: runtime scaled by CPU capacity and\nfrequency. That is what makes a number from a little core comparable with a\nnumber from a big core, and it is why the same arithmetic works on a hybrid\nlaptop and on a homogeneous server.\n\n### 6.4. How much load should a domain carry? [#](#6-4-how-much-load-should-a-domain-carry)\n\nThe target is not an average. It is a capacity-proportional fair share:\n\n```\nfair_share = total_queued_load_invr × domain_capacity / total_capacity\n```\n\nA domain with 40% of the machine’s capacity should carry 40% of the queued\nload. So a domain holding **more** load than its neighbor can still be\nunderloaded and steal from it, if it has much more capacity. (On\nhomogeneous hardware, the fair share and the plain average coincide.)\n\nMoving load is also done in steps, not jumps. A stealee exports at most\nhalf of its excess per round, and a stealer imports at most half of its\ndeficit. Moving the whole excess would overshoot, so the imbalance would\nflip the next round (ping-pong); uncapped, every stealer would drain the\nsame domain in one round (thundering herd). Below 25% system utilization\n(`--lb-low-util-pct`), balancing is skipped entirely: when the machine is\nquiet, imbalance is cheap and migration is expensive. These budgets exist\nfor a reason — forcing steals under per-CCX pinning has been measured\nburning a large share of cycles in DSQ spinlock contention.\n\n### 6.5. Turbulent DSQs [#](#6-5-turbulent-dsqs)\n\nHave you ever thought about what happens if a latency-critical task runs on a CPU that receives a lot of IRQs (say, network interrupts), or that also runs RT or deadline tasks? Such CPUs are frequently interrupted by events out of LAVD’s control. Turbulent DSQs keep vulnerable tasks away from them.\n\n- \nA CPU is **steady** if its latency headroom is in the top 12.5%;\notherwise, it is**turbulent** . This is recomputed every tick.\n- \nNot every task suffers equally from being interrupted. A latency-critical task is hurt the most, because its delay cascades down the task chain; a task with high utilization is also more exposed, simply because it runs longer. LAVD ranks tasks by this **preemption vulnerability** : the more\nvulnerable ones are queued for the steady CPUs, and the rest — tasks that\ncan absorb an interruption — are queued for the turbulent CPUs. The split\npoint keeps the load of each group in proportion to the capacity of its CPUs.\n- \nDispatch is asymmetric, and that is the design. A turbulent CPU pulls only from the turbulent DSQ; a steady CPU pulls from both — the steady DSQ first. Steady CPUs are generally more responsive and have more capacity (for fair tasks), so letting them drain the turbulent queue keeps the boundary **soft** and ensures system-wide forward progress.\n\n### 6.6. Should a task wait for its old CPU? [#](#6-6-should-a-task-wait-for-its-old-cpu)\n\nWhen a task wakes up, LAVD, like other sched_ext schedulers, aggressively\nmigrates it to a completely idle core if one is available. That maximizes\nwork conservation, and it is the right call for latency-sensitive,\ncompute-intensive work. But a task leaves useful state behind on the CPU it\nlast ran on: its cache lines and TLB entries. For workloads bound on both\ncompute and memory throughput, such as caching services, migrating too\neagerly means refilling the cache and the TLB over and over. Most of the\nextra TLB misses come from application heap accesses: with user-space\nallocators like `jemalloc`, whose per-thread caches are typically hot, this\ncost about 3% of IPC at nominal load compared to EEVDF. **Sometimes it is\ncheaper to wait for the old CPU than to move to an idle one.**\n\nDeciding that needs two numbers: how warm the old CPU still is for this task, and how long the task would have to wait for it.\n\n**How warm is the old CPU?** LAVD keeps a per-task **heat** value (0 to\n1024) for the CPU the task last ran on, as a cheap proxy for how much of\nthe task’s cache and TLB state is still there.\n\n- \nHeat builds up while the task runs on that CPU and saturates after 500 µsec of residence.\n- \nOnce the task leaves — to sleep, or to run elsewhere — the heat fades linearly to zero over 5 msec, because other tasks running on that CPU gradually evict the task’s state.\n- \nIf the task runs on a different CPU, the heat starts over on the new CPU.\n\n**Wait or migrate?** When a latency-tolerant task wakes up and its old CPU\nis idle, it simply goes back there. When the old CPU is busy, LAVD compares\ntwo estimates:\n\n- \n**The wait budget** : a base budget (`--warm-cpu-us` ) stretched by the\nheat,`budget = base × (1 + heat / 1024)` . A CPU that is still warm is\nworth waiting up to twice as long for.\n- \n**The expected wait** : the time until the task currently running on the\nold CPU stops, plus the time to serve the tasks already queued on that\nCPU.\n\nIf the expected wait fits in the budget, the task is queued on the old\nCPU’s per-CPU DSQ (the one from §6.1) and waits there; otherwise, it\nmigrates to an idle CPU as usual. Latency-critical tasks never wait: if a\ntask’s latency criticality is above the system’s threshold, it takes an\nidle CPU right away, because for such a task, scheduling delay matters more\nthan cache warmth. The feature is off by default (`--warm-cpu-us=0`), and\nwe are still actively working on it.\n\n## 7. How few cores — and which ones? Power [#](#7-how-few-cores-and-which-ones-power)\n\n### 7.1. Spreading work across all cores can be the worst thing to do [#](#7-1-spreading-work-across-all-cores-can-be-the-worst-thing-to-do)\n\nLAVD was motivated by battery-backed gaming devices like the Steam Deck and the Steam Frame, and one thing we found interesting is that not all games are CPU-intensive. Especially, some old games are so light that fully utilizing one or two CPUs is more than enough to achieve 120 FPS. For example, I found that two or three CPUs were more than sufficient to play an old Tomb Raider game, which many users still want to play — and that was the motivation to pursue this idea further. Even the Steam Deck’s home screen with nothing running needs only about two CPUs.\n\nSo suppose the system-wide CPU utilization is 20% on an 8-CPU machine. The obvious placement is 20% on each CPU. Here is what actually happens:\n\n- \nEvery core sits at low utilization, so the frequency governor holds every core at a low frequency.\n- \nEvery core keeps entering and leaving C-states, paying transition energy and exit latency constantly.\n\nLow performance **and** high power. The worst of both. Idle is not free, and\nhalf-idle is the most expensive state a core can be in.\n\n**Core compaction** inverts it: use fewer CPUs, drive them properly, and\nlet the rest sleep deeply. You get higher clocks on the active cores,\ndeeper C-states elsewhere, and fewer P-state and C-state transitions. Core\ncompaction landed in June 2024, and it is where power entered LAVD’s model\nat all.\n\nCore compaction splits the CPUs into two sets. The **active set** is the\nCPUs that LAVD schedules tasks on in normal operation; its size follows the\ncurrent load. The **overflow set** is an elastic extension of it: when a\ntask cannot run on any active CPU — for example, a task pinned to an\ninactive CPU — LAVD adds more CPUs to the overflow set, following the CPU\npreference order. All the other CPUs stay idle and can sleep deeply.\n\n### 7.2. Not all cores cost the same [#](#7-2-not-all-cores-cost-the-same)\n\nCore compaction decides **how many** cores to wake. On a hybrid processor,\n**which** cores matters a lot, too. The kernel’s energy model describes each\nperformance domain as a table of OPPs (operating performance points) — the\nperformance/power pairs that the silicon can actually hold. Power grows\nroughly with V²f, and the voltage has to rise with the frequency, so power\nclimbs roughly with the cube of frequency. The top of a core’s range is\nwhere the energy goes.\n\nHere is a concrete example from a Snapdragon development board (two efficiency cores, five performance cores in two clusters, and one prime core). Suppose you need 1024 of compute capacity. The prime core can deliver it alone, but only at its last OPP, where it burns 2.33 W. Two performance cores at 512 each deliver the same compute for about 0.41 W in total — about 5.6× less.\n\n“**Use the fastest core**” is the wrong instinct; the fastest core is the\nright answer only when nothing else can reach the demand.\n\n### 7.3. Energy Model Aware Scheduling (EMAS) [#](#7-3-energy-model-aware-scheduling-emas)\n\nTurning those curves into scheduling decisions takes three steps:\n\n1. \n**Where does the model come from?** The kernel already publishes it at`/sys/kernel/debug/energy_model` . LAVD reads it\n(`scx_utils/src/energy_model.rs` ); it does not measure it.\n2. \n**Which CPUs for a given demand?** The user-space part\n(`EnergyModelOptimizer` in`cpu_order.rs` ) searches over every\ncombination of {performance domain, OPP, CPU count} and keeps the cheapest\nset at each performance level. This is solved once, at load time — not per\nwake-up.\n3. \n**How does BPF use the answer?** It is shipped into BPF as constant\ntables (`pco_bounds[]` ,`pco_table[][]` ,`pco_nr_primary[]` ), at most 11\nstates. At runtime, it is just an array index.\n\nHere is the real output for the Snapdragon board:\n\nNote that each row is not a prefix of the next one. At 20.2%, two\nperformance cores are woken **ahead of** the efficiency cores that were the\nwhole set in the row above. At 60.1%, the efficiency cores are dropped\noutright: adding a slow core to a set that already has fast ones can cost\nmore energy than it saves. And the prime core appears only in the last row,\nbecause it is the only way to reach the top of the capacity range.\n\nOn machines without an energy model (which includes most x86 machines), the\ntable is synthesized from the topology — turbo-capable cores first, then\nbig cores, SMT siblings last — and you can override it with\n`--cpu-pref-order`.\n\n### 7.4. How many cores are enough? [#](#7-4-how-many-cores-are-enough)\n\nWith the table in hand, all that is left is to measure the demand and index\nthe table (`calc_nr_active_cpus()`):\n\n```\nreq_cap = nr_cpus_online × avg_util_invr + 25%\n\nfor i in pco_states:                      <- the EMAS table\n    if pco_bounds[i] < req_cap: continue\n    sum = Σ eff_cap(cpu) × 50%  over pco_table[i]\n    if sum >= req_cap: ACTIVE = pco_nr_primary[i]\n```\n\nThe 25% is headroom for spikes, and the 50% per-CPU cap targets the active\nCPUs at half utilization, leaving real slack for latency-critical wake-ups.\nFor example, on the board above, an average invariant utilization of 90\ngives `req_cap = 8 × 90 = 720`, plus 25% = 900. The first state whose bound\ncovers 900 is 1138 (the 20.2% row). Summing half the capacities of cpu2,\ncpu3, cpu0 gives 430, 860, 965, crossing 900 at the third CPU — but all\nfour primary CPUs of that state (cpu2, cpu3, cpu0, cpu1) become active. The\nremaining CPUs stay in deep idle unless they are pulled into the overflow\nset.\n\nCompaction is not allowed to win everywhere. Kernel and pinned tasks are\nexempt, and the overflow set grows on demand: a non-active CPU that finds a\npinned task or queued work joins the overflow set and runs it. When utilization gets\nclose to 90%, the active set expands immediately; otherwise, it is\nre-planned every 250 msec. And `--performance` disables compaction\nentirely.\n\n### 7.5. Which task belongs on a big core? [#](#7-5-which-task-belongs-on-a-big-core)\n\nCompaction picks how many cores, and the energy model picks which ones. One\nquestion remains: which **task** goes on which kind of core? LAVD answers it\nwith the same three signals, read the other way around:\n\n```\nlog_wwf  = log2x(wait_freq_ft × wake_freq_ft)\nlat_cri  = log_wwf + log2x(runtime_ft × weight_ft)\nperf_cri = log_wwf + log2x(sum_runtime_ft)\n\n  runtime_ft     = reverse runtime factor (shorter runs score higher)\n  sum_runtime_ft = run_freq × invariant runtime × weight  (demand per second)\n```\n\nBoth share their first term — how central the task is in the wake-up graph.\nBeing in the middle of a chain makes a task both worth scheduling early and\nworth putting on a fast core. They differ in the second term: short, bursty\nwork is **latency-critical**; sustained demand is **performance-critical**. A\ntask whose `perf_cri` exceeds a threshold prefers a big core; otherwise, a\nlittle core.\n\nThe threshold is not fixed. It tracks the big cores’ share of the capacity\nof the **active** set, and the threshold is placed — by piecewise-linear\ninterpolation over the observed minimum, average, and maximum `perf_cri` —\nso that the fraction of tasks above it roughly follows that share. When\ncompaction shrinks to little cores only, nothing is performance-critical;\nwhen only big cores are active, everything is. In the 20.2% row above\n(cpu2, cpu3, cpu0, cpu1), 1722 of the 2142 capacity is big; in one trace,\nthe threshold landed at 276 and 15 of 26 tasks went big. Compaction chooses\nthe set, the set fixes the big-core share, the share moves the threshold,\nand tasks re-sort across core types — one loop on one 250 msec cadence, not\nthree separate mechanisms. On a homogeneous machine, this whole mechanism\nis skipped and costs nothing. In practice, it works pretty well and saves a\nlot of power, especially on the Steam Frame.\n\n## 8. How much may one tenant use? cgroup bandwidth control [#](#8-how-much-may-one-tenant-use-cgroup-bandwidth-control)\n\n### 8.1. cpu.max as a library [#](#8-1-cpu-max-as-a-library)\n\nServer fleets need hard quotas, not just good ordering. On a big server\nwith 200 CPUs, many containers are likely to run together, and we do not\nwant one noisy container to pollute the others. We want at least a minimum\nlevel of performance isolation. cgroup v2’s\n[`cpu.max`](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files)\nis the CPU bandwidth\ncap: all the tasks in a cgroup together may use at most `$QUOTA` µsec of\nCPU time per `$PERIOD` µsec, and an optional `$BURST` lets a cgroup carry\nunused quota over to the next period. Writing `50000 100000` gives that\ncgroup half a CPU’s worth. When a cgroup runs out of its quota, it is\n**throttled**: its tasks are parked until the next period.\n\nLAVD does not implement `cpu.max` itself. The mechanism lives in\n`lib/cgroup_bw.bpf.c` in the scx tree, as a BPF library (about three\nthousand lines, arena-allocated, entirely in BPF). Nothing in it is\nLAVD-specific; `scx_lavd` is simply its first user. Any sched_ext scheduler\ngets `cpu.max` by wiring a few callbacks, and the library owns the quota\npools, the timers, and the throttled-task queues:\n\nTo me, this is the argument for sched_ext as a **platform** rather than a\nscheduler. A feature this size would otherwise be reimplemented once per\nscheduler, or not at all. (The equivalent in-kernel machinery is spread\nacross the fair-class throttling path and is not reusable by anything\noutside CFS/EEVDF.)\n\n### 8.2. Why not just copy the kernel’s design? [#](#8-2-why-not-just-copy-the-kernel-s-design)\n\nThe kernel (CFS/EEVDF) implements `cpu.max`\n([CFS Bandwidth Control](https://docs.kernel.org/scheduler/sched-bwc.html))\nin a way that is accurate\nwithin every period, but it pays for that on the hottest paths:\n\n- \n**Task selection walks the cgroup hierarchy.** The hierarchy is mirrored\nas nested runqueues — each cgroup has its own red-black tree inside its\nparent’s — so picking the next task walks down every level. The cost grows\nfrom O(log N) to O(D × log N), where D is the depth of the hierarchy.\n- \n**Throttle detection is synchronous.** Each CPU borrows a small slice of\ntime (e.g., 5 msec) from the cgroup’s global quota pool. Running a task\nconsumes that local budget and those of its ancestors; when it runs dry,\nthe CPU pulls a new slice from the global pool, and if the pool is empty,\nthe cgroup and all its descendants are throttled. So the check costs O(D)\nand touches a global pool on the critical path.\n- \n**Two timers per cgroup.** A period timer refills the quota every`$PERIOD` , and a slack timer returns unused local budget to the global\npool. The number of timers grows with the number of cgroups.\n\nFor the sched_ext library, we set three goals instead:\n\n- \n**Task selection: O(log N), no extra overhead.** Task selection is\nperformance-critical, so supporting`cpu.max` should not add anything to\nit.\n- \n**Throttle detection: O(1), no global hot spot.** Checking whether a\ncgroup is throttled should be cheap and should not touch any memory that\nall CPUs contend for.\n- \n**Replenishment: only a handful of timers.** Two timers per cgroup are\ntoo many; a few timers in total should serve every cgroup.\n\n### 8.3. Key ideas: a flat runqueue and eventual enforcement [#](#8-3-key-ideas-a-flat-runqueue-and-eventual-enforcement)\n\n**No nested runqueue.** All unthrottled tasks share the same DSQ,\nregardless of their cgroup, so task selection is unchanged — still\nO(log N). Each cgroup instead keeps a separate queue only for its\n**throttled** tasks, the **backlogged task queue (BTQ)**. Throttling a task\nmeans moving it to its cgroup’s BTQ, and unthrottling means moving it back:\nboth are plain, unnested queue operations.\n\n**Eventual, not instant, enforcement.** The kernel’s synchronous model\nenforces the quota precisely within each period, which is exactly what\nmakes it expensive on the hot path. The library instead aims for correct\nbandwidth **across** periods. If a cgroup overspends, the overspend becomes\n**debt** that it pays back in the following periods:\n\n```\ndebt               = max(consumed - period_budget, 0)\nnext period_budget = nquota_ub + burst_credit - debt\n```\n\nA cgroup that overran by 3 msec gets 3 msec less in the next period. Nothing is forgiven, so the long-run average converges on the configured quota. This relaxation is what opens the door to moving all the expensive work off the critical path.\n\n**One timer for all cgroups.** Users configure arbitrary\n`{$QUOTA, $PERIOD, $BURST}` per cgroup, but the library normalizes every\nquota to one fixed 100 msec period (`nquota`). Now all cgroups speak the\nsame unit, and a single replenishment timer serves all of them. The\nhierarchy is folded in at the same time: a cgroup’s effective limit is\n`nquota_ub = min(nquota, tightest ancestor's nquota)`.\n\n### 8.4. How it works: a flag on the hot path, the rest in the background [#](#8-4-how-it-works-a-flag-on-the-hot-path-the-rest-in-the-background)\n\n- \n**Hot path.**`ops.enqueue()` asks the library whether the task’s cgroup\nis throttled, and the answer is a`READ_ONCE(is_throttled)` of a\nper-cgroup flag — no lock, no atomic, no hierarchy walk. If it is\nthrottled, the task goes to the cgroup’s BTQ instead of the DSQ.\n- \n**Accounting.**`ops.tick()` and`ops.stopping()` report the time a task\nconsumed with a single atomic add to an**LLC-local** counter of its\ncgroup, so there is no cross-LLC cacheline contention.\n- \n**Throttling.** An accounting timer sums the LLC-local counters up the\nhierarchy (bottom-up) and sets a cgroup’s throttle flag if it has\nconsumed its budget; then it propagates throttling down to the\ndescendants (top-down). The timer is adaptive: it tracks each cgroup’s\nconsumption rate, predicts when the cgroup will run out of budget, and\nfires at a quarter of that time, clamped to 1–20 msec. Checking about four\ntimes before the predicted exhaustion catches rate changes early and keeps\nthe overspend small.\n- \n**Replenishment.** Every 100 msec, the replenishment timer computes each\ncgroup’s new budget (with debt and burst credit) and marks the cgroups to\nunthrottle — that is the top half. The bottom half runs in`ops.dispatch()` : CPUs pop tasks from the BTQs and put them back to the\nDSQ, which keeps the timer callback short and naturally spreads the\ndraining work across CPUs.\n\nThe only thing traded away is **detection latency**: a cgroup may overrun\nfor up to one accounting interval before its flag is set. Accuracy is not\ntraded, because the debt carry-over pays the overrun back. If you need hard\ninstantaneous capping rather than a converging average, this is the wrong\ntrade-off — but for a fleet, it is a good one.\n\n## 9. Beyond gaming: from games, to fleets, to GPUs [#](#9-beyond-gaming-from-games-to-fleets-to-gpus)\n\nOne thing that is ironic: when I started the project, I didn’t expect much\nof LAVD. I just wanted to write a scheduler specialized for gaming. But it\nturns out that our intuition — wake frequencies and latency criticality —\nis not that specialized, and it is useful for other hardware and workloads,\ntoo. The signal LAVD measures — **who is on the critical path** — was never\ngaming-specific. Anything where tasks wait on each other has the same\nshape.\n\n- \n**Where it started: gaming.** SteamOS, handhelds, and VR headsets like\nthe Steam Frame — latency-critical, communication-heavy, and running on a\nbattery.\n- \n**Where it went: server fleets.**`scx_lavd` now runs in Meta’s\ndatacenters, for the range of hardware and use cases that do not need a\nspecialized scheduler. The same wake-up-frequency signal that found a\ngame’s critical path found a caching service’s; slice boosting, per-domain\nqueues with a budgeted load balancer, and the steady-vs-turbulent split all\nanswer the same “**tasks wait on each other** ” problem.\n- \n**The next question: AI serving.** Inference servers like vLLM and SGLang\nlive or die on keeping the GPU busy. We are looking at whether the\nscheduling objective should be GPU idle time, not CPU latency alone. This\nis a question we are asking.\n\n## 10. What’s next? [#](#10-what-s-next)\n\nIn this post, I sketched `scx_lavd` from a bird’s-eye view, so I skipped a\nlot of details along the way. In the upcoming posts of this series, I will\ngo deeper into some of the mechanisms where the details matter, and\nintroduce other interesting mechanisms of `scx_lavd` that I didn’t cover\ntoday.\n\nUntil then, if you want to dig deeper, I strongly encourage you to read the\nsource code.\n[`main.bpf.c`](https://github.com/sched-ext/scx/blob/main/scheds/rust/scx_lavd/src/bpf/main.bpf.c)\nopens with a 180-line design document, and the rest of the code is heavily\ncommented. The project lives at\n[github.com/sched-ext/scx](https://github.com/sched-ext/scx).\nContributions, bug reports, and benchmark results on your workloads are\nalways welcome. And if this kind of work excites you, [Igalia is\nhiring](https://www.igalia.com/jobs)!\n\nHappy scheduling until the next blog post!", "url": "https://wpnews.pro/news/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3", "canonical_source": "https://blogs.igalia.com/changwoo/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3/", "published_at": "2026-11-18 06:31:25+00:00", "updated_at": "2026-11-18 06:45:11.098579+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["scx_lavd", "sched_ext", "Valve", "Meta", "Steam Frame", "Steam Deck", "Gavin Guo", "Kernel Recipes 2026"], "also_reported_by": [], "takeaway": "scx_lavd infers latency-critical threads from futex wait/wake behavior instead of annotations, and pairs per-cache-domain queues, capacity-based balancing, core compaction and cgroup bandwidth control to run the same BPF scheduler on both a handheld and a server fleet.", "recommendation": {"advice": "When scheduling latency-sensitive workloads, infer urgency from futex wait/wake graphs rather than tuning static knobs, and balance per cache domain by capacity while compacting the active core set.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3", "markdown": "https://wpnews.pro/news/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3.md", "text": "https://wpnews.pro/news/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3.txt", "jsonld": "https://wpnews.pro/news/sched-ext-a-sketch-of-the-scx-lavd-scheduler-part-3.jsonld"}}