DPU Offload vs Host Cores: When the 100G NIC Stops Being Free

A developer's cost analysis of DPU network offload for AI inference serving nodes finds that offloading to a $1,800–$2,600 DPU only pays off when it frees roughly four to six cores per node, since a plain HTTP inference server with connection pooling typically burns just 1.3% to 2.1% of a single core at 300 requests per second. The analysis attributes the break-even to per-request CPU cost, which rises sharply with larger agent-style contexts (20–200 KB) and per-request TLS connection churn, and recommends offloading IPsec/WireGuard, TLS record processing and telemetry first while keeping model-aware routing and request-level authentication on the host.

A serving node that pushes 100 Gbps of model traffic does not do it for free. Depending on kernel, MTU and how many small requests you handle, the host burns somewhere between two and six cores on network stack work: interrupts, softirq, socket copies, TLS record processing. On a 64-core machine that is 3% to 9% of the machine, which sounds acceptable until you remember that inference is often memory-bandwidth-bound and those cores are also competing for the same uncore resources. The obvious answer is to offload the network to a DPU. The obvious answer is also where most teams waste $200,000. Here is the arithmetic that decides it. Take a realistic serving configuration: HTTPS, 4 KB average request body, 40 KB response, 300 requests per second per node. Per request the host performs a TLS handshake amortised across connections, parses HTTP headers, copies the body into user space, runs tokenisation, and copies the response back out. The network-attributable portion is roughly 45 to 70 microseconds of CPU time per request on a modern core with kernel bypass disabled. At 300 requests per second, that is 13 to 21 milliseconds of core time per second, or 1.3% to 2.1% of a single core. Multiply by 40 nodes and you have spent less than one core across the fleet. Offloading that to a $2,000 DPU per node is obviously wrong. Now change one variable: request size. Agent workloads send 20 KB to 200 KB of context per call. At 80 KB average, per-request network CPU rises to roughly 180 to 260 microseconds, and at 300 requests per second you are at 5% to 8% of a core per node, plus softirq contention that shows up as p99 jitter rather than throughput loss. Change a second variable: connection churn. If clients reconnect per request instead of pooling, TLS handshakes dominate and you can lose a full core per node. This is common in multi-tenant gateways and almost never measured. The offload decision is not about bandwidth, it is about CPU-seconds per byte and how much of them land on the critical path of your p99. A DPU is an Arm-based SoC with a NIC attached. It can host a full network stack: OVS, IPsec, TLS termination, load balancing, telemetry collection. When configured well, the host sees a virtio or SR-IOV device and the entire packet path disappears from its CPU accounting. Three things are worth offloading, in order of payoff. First, IPsec or WireGuard for inter-node traffic: this is pure per-packet cost with no application value, and it typically costs 0.8 to 1.5 cores per node at 100 Gbps. Second, TLS record processing at the edge: it removes both CPU and the associated cache pollution. Third, telemetry and flow logging, which is the classic death-by-a-thousand-cuts workload. Three things are usually not worth offloading. Application-layer load balancing, because you need model-aware routing that a DPU cannot do. Request-level authentication, because it needs the same state as the model server. And anything with a fast-changing policy, because every change becomes a DPU firmware deployment. A current-generation 200G DPU with dual 100G ports and reasonable Arm cores lists around $1,800 to $2,600, and consumes 25 to 45 watts. Amortised over three years that is roughly $55 to $80 per month per node, plus the switch port and the operational cost of a second operating system to patch. A cloud vCPU costs $18 to $35 per month at committed rates. On-prem, the fully loaded cost of a core is $12 to $20 per month. So a DPU pays for itself when it frees roughly four to six cores per node. Four to six cores is a lot. That threshold is only crossed when you are doing IPsec at line rate, terminating a high-churn TLS workload, or running a service mesh sidecar per pod. For a plain HTTP inference server with connection pooling, the DPU is a $2,000 device that saves one core. This is why the adoption pattern looks odd from the outside. Hyperscalers with custom silicon deploy DPUs everywhere, because they are also running overlay networks, encryption at line rate and per-tenant telemetry. Mid-size clusters buy DPUs, enable SR-IOV, and never turn on the offload features, which is the worst of both worlds: you paid for the device and still burn the cores. Two trends move the break-even point. Confidential computing moves it toward the DPU. Encrypting all inter-node traffic in memory is now a compliance requirement in several regulated industries, and doing that in software costs 10% to 25% of throughput. A DPU does it in hardware for close to nothing. As confidential inference becomes standard for healthcare and financial workloads, the DPU stops being optional. Smaller models move it away. A 3B-parameter model serving short requests has very little per-request network cost, and the fleet that serves it does not need a $2,000 NIC. If the industry's inference volume shifts toward many small models rather than a few large ones, the DPU market grows more slowly than the optimists expect. My estimate: DPU attach rate in AI serving clusters goes from roughly 15% today to 45% to 55% by 2030, with the growth concentrated in regulated and multi-tenant environments. The device market grows to $8 to $12 billion, but the more interesting number is the software market around it, which is where the operational pain actually lives. Every time you use a voice assistant and it responds before you finish your sentence, some part of that latency budget was spent moving audio frames through a network stack. Offload decisions made by an infrastructure team you will never meet determine whether that feels instant or slightly laggy. The same is true of a translation feature in a browser, or a photo feature that runs server-side. Users experience the sum of a hundred such decisions as a single quality judgement: "this app feels fast." For engineers, the actionable version is unglamorous. Instrument CPU time attributable to the network before you buy anything. On Linux, perf on softirq plus ss -ti for retransmit and queue stats gives you most of the answer in an afternoon. If you cannot attribute at least four cores per node to network work, a DPU is a purchase you will regret in the second year, when the firmware is three versions behind. Offload hardware follows a predictable lifecycle. It arrives expensive and specialised, gets adopted by the few operators whose workload genuinely justifies it, then drifts into general procurement as a default because nobody wants to be the team that skipped it. The teams that get value are the ones who keep measuring. The teams that do not are the ones who bought the device for the brochure and left the features off. In infrastructure, the difference between a 20% efficiency gain and a 20% cost increase is usually one afternoon of measurement, taken before the purchase order rather than after. SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS Creating this takes real effort — thank you for the support.