cent_runtime: C++17 Runtime and C ABI for the CENT Near-Bank PIM Architecture A developer built cent_runtime, a C++17 runtime with a C ABI and Python bindings that connects PyTorch linear layers from models such as Llama-2 7B to the CENT near-bank processing-in-memory instruction set architecture. The project targets the gap between the CENT paper's simulated gains for GPU-free LLM inference over CXL 3.0 and executable software, dispatching work to the official aim_simulator (Ramulator 2.0) and the GDDR6-AiM execution model. The report frames the effort around decoding's memory-bound GEMV phase, which accounts for roughly 98% of inference time. The CENT architecture PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference , Gu et al., ASPLOS 2025 proposes running Large Language Model LLM inference without GPUs, replacing HBM memory with GDDR6-AiM modules connected over CXL 3.0. While the analytical and simulation gains reported in the paper are substantial, a practical gap separates academic results from software execution: there is no system layer capable of taking PyTorch tensors and dispatching them onto the architecture's execution model. This technical report documents the architectural context, the physical bottlenecks motivating the project, the role of a hardware dispatcher, how the official simulator aim simulator / Ramulator 2.0 operates, and the implementation of cent runtime https://github.com/josedercysilva/cent-pim-runtime — a C++17, C ABI, and Python library connecting linear layers of models such as Llama-2 7B to the CENT instruction set architecture. The seminar slide deck accompanying this work is available at the repository root in apresentacao cent.pdf https://github.com/josedercysilva/cent-pim-runtime/blob/main/apresentacao cent.pdf . aim simulator Ramulator 2.0 cent runtime In classical computer architecture Patterson & Hennessy, Ch. 5 , the memory hierarchy balances speed, capacity, and cost across successive tiers: php CPU / GPU <--- L1 Cache <--- L2 Cache <--- L3 Cache <--- Main Memory DRAM <--- SSD / Disk fastest, smallest capacity slowest, largest capacity This organization relies on temporal and spatial locality: a block fetched from DRAM must be reused dozens or hundreds of times in caches before eviction. LLM inference breaks this assumption during its longest execution phase. Inside a DRAM chip, data is organized into two-dimensional cell arrays. Activating a row loads an entire 8,192-bit block at once into the internal Row Buffer in roughly 1 internal access cycle — meaning high internal bandwidth exists within the memory die itself. The bottleneck sits at the chip's output interface. Moving those 8,192 bits to the processor CPU or GPU requires crossing the external bus and physical I/O pins operating at roughly 64 bits per cycle or 32 bits in specific pin configurations . Transferring a single row that was already sitting in the Row Buffer takes roughly 128 bus cycles. While data crosses the bus, GPU compute units remain in a stall state waiting on memory. GEMM vs. Decoding GEMV LLM inference splits into two phases with opposite physical profiles: | Characteristic | Phase 1: Prefill Initial Prompt | Phase 2: Decoding Token-by-Token Generation | |---|---|---| | Share of total runtime | ~2% of inference time | ~98% of inference time | | Mathematical operation | Matrix × Matrix GEMM , e.g., 4K×4K × 4K×4K | Vector × Matrix GEMV , batch size = 1 | | Arithmetic intensity | High ~2,000 Ops/Byte | Low 1 to 16 Ops/Byte | | Execution regime | Compute-Bound saturates Tensor Cores | Memory-Bound limited by memory bandwidth | During decoding, generating each token requires reading entire weight matrices tens of megabytes per layer, totaling tens of gigabytes across the model only to compute a single vector-matrix product and discard the weights immediately after. On a 4× NVIDIA A100 node, this lack of reuse drops effective Tensor Core utilization to roughly 21% with 79% of execution time spent stalled on HBM reads . Processing-in-Memory PIM — specifically the GDDR6-AiM Accelerator-in-Memory design adopted in CENT — places multiply-accumulate execution units PUs/ALUs directly beside each DRAM bank, ahead of the narrow I/O pin interface. Each GDDR6-AiM chip therefore consumes weights directly at the Row Buffer output without streaming weight matrices over the external bus: Instead of dedicated GPUs and HBM interposers, CENT connects a Host CPU to 32 CXL 3.0 devices through a 48-port CXL Switch: +------------+ | Host CPU | +-----+------+ | v +--------------------+ | CXL 3.0 Switch | +---------+----------+ | +--------------------+--------------------+ | | | v v v +---------------+ +---------------+ +---------------+ | CXL Device 0 | | CXL Device 1 | .. | CXL Device 31| +-------+-------+ +---------------+ +---------------+ | +--- CXL Port + Inter-device Communication Controller +--- Instruction Buffer 2 MB SRAM - PC - Decoder +--- PNM Units + Shared Buffer 64 KB +--- 16x PIM Ctrl --- 16x GDDR6-PIM 32 channels per board Inside each CXL device, execution is divided across two tiers: Softmax , RMSNorm , SiLU , and RoPE , transferring only compact partial vectors <16 KB . In the paper's evaluation against a 4× NVIDIA A100 server 80 GB per GPU : At the same time, the architecture has three clear technical limitations: This third gap is the focus of our work: building the system layer that connects high-level tensors to the architecture's instruction format and cycle-accurate simulator. To validate the microarchitecture, the CENT authors released aim simulator , built on top of Ramulator 2.0 a cycle-accurate academic DRAM simulator . aim simulator compiles into a ramulator2 executable paired with libramulator.so . It is not a neural network runtime, but a discrete-event hardware simulator: configs/gddr6 aim.yaml ──┐ Physical parameters: │ 32 channels, 16 banks, ├──► ramulator2 aim simulator ──► Silicon Counters GDDR6 timings, FRFCFS │ e.g. CH31 AiM ISR WR GB cycles, │ memory system cycles file.trace ──────────────┘ Static list of textual CENT ISA commands When invoked from the command line ramulator2 -f configs/gddr6 aim.yaml -t file.trace , it models Row Buffer accesses, bank conflicts, FRFCFS command scheduling, AllBank refresh, and AiM instructions cycle by cycle across 32 GDDR6-AiM channels, printing per-instruction cycle counters on completion. .trace files: AiM WR GB , AiM MAC ABK , etc., already decomposed into physical rows and channel masks. y for the next model layer. In systems software, a dispatcher is the intermediary layer sitting between a high-level framework and the accelerator controller. When PyTorch runs torch.matmul x, W on a GPU, the call passes through the internal dispatcher into native C/C++ libraries libcudart.so , libcublas.so that configure and submit work to the device. For CENT, the hardware dispatcher fulfills four responsibilities: batch size = 1 . Prefill batch size 1 are rejected so they remain on the host. DVid hosts the layer and which of the 32 GDDR6 channels CHmask are enabled for dimension K . M × K weight matrix into blocks aligned to the 16 DRAM banks per channel, writes the binary descriptors into the 2 MB SRAM buffer, and emits the corresponding cent runtime structures this dispatcher into two main layers — Python PyTorch/NumPy and C++17 connected across a stable C ABI libcent runtime.so — plus an optional integration module for ramulator2 . +-----------------------------------------------------------------------------+ | 1. PYTHON LAYER python/cent runtime/ | | cent runtime.matmul x, W, layer id=0, trace path=None, simulate=False | | - Validates GEMV contract batch=1 in ops.py | | - Normalizes to contiguous float32 and extracts pointer via ctypes | +--------------------------------------+--------------------------------------+ | ctypes bindings.py v ======================= C ABI BOUNDARY extern "C" ======================= | 2. PUBLIC CONTRACT include/cent/runtime api.h - build/libcent runtime.so | | cent dispatch gemv x, M, K, W, y, layer id, trace path, &telemetry | +--------------------------------------+--------------------------------------+ | +----------------------------+----------------------------+ v v +-----------------------------------+ +-----------------------------------+ | 3. C++17 ENGINE src/ | | 4. NUMERICAL COMPUTATION | | - topology.cpp - DVid/CHmask | | - Row-major loop over W M x K | | - sram buffer.cpp - 2 MB SRAM | | - FP64 accumulation - FP32 out | | - isa emitter.cpp - PIM blocks | | - Writes directly to pointer 'y' | +-----------------+-----------------+ +-----------------------------------+ | +--- Binary packets 14 B in SRAM + Hex Dump +--- .trace file CENT ISA --- simulator.py - ramulator2 python/cent runtime/ The Python package exposes cent runtime.matmul , simulate on ramulator , and get instruction buffer hex : ops.py : torch.Tensor or numpy.ndarray falling back to NumPy when PyTorch is not installed . Validates x as K, or 1, K and W as M, K . 3D+ tensors or ValueError . It then ensures a contiguous float32 layout on CPU performing no extra copy when the tensor already satisfies this layout , allocates the output tensor ctypes . bindings.py : build/libcent runtime.so running make shared automatically if the library has not been built yet and maps the C struct CentDispatchTelemetry 56 bytes on x86 64 into a Python dictionary. simulator.py : simulate=True creating a temporary tempfile if trace path=None , resolves the ramulator2 binary in configurable order explicit ramulator bin → CENT RAMULATOR BIN environment variable → external/ramulator2 → ../CENT/aim simulator/build/ramulator2 → PATH , invokes the simulator via subprocess , and extracts wr gb cycles , mac abk cycles , and memory system cycles . .h and C++17 Core .hpp In C++, namespaced or overloaded functions undergo name mangling for instance, cent::compute dvid int becomes ZN4cent12compute dvidEi in the symbol table , and the C++ ABI differs across compilers. To guarantee direct FFI interoperability with ctypes , the codebase separates: isa.h , telemetry.h , runtime api.h : extern "C" , exporting unmangled symbols cent init runtime , cent dispatch gemv , cent get instruction buffer hex . topology.hpp , sram buffer.hpp , isa emitter.hpp : namespace cent , classes, RAII, and std::array . The native library compiles with -std=c++17 -O3 -fPIC -Wall -Wextra -Werror and uses no C++ exceptions throw / try-catch across the boundary. All exported functions are marked noexcept and return explicit status codes via enum CentStatus CENT OK = 0 , CENT ERR NULL PTR = -1 , CENT ERR INVALID SHAPE = -2 , CENT ERR SRAM OVERFLOW = -3 , CENT ERR TRACE IO = -4 . CentInstruction include/cent/isa.h defines the controller parameters CENT SRAM BUFFER BYTES = 2 MB , 32 CXL devices, 32 channels per device, 16 banks per channel, and opsize = 64 bursts , the 5 CENT ISA opcodes, and the packed binary instruction layout: enum CentOpcode : uint8 t { CENT OP WR GB = 0x02, // Writes activation vector to active channels' Global Buffer CENT OP WR BIAS = 0x03, // Initializes accumulator registers across DRAM banks CENT OP RD MAC = 0x05, // Reads accumulated result from banks to channel buffer CENT OP MAC ABK = 0x0F, // Parallel Multiply-Accumulate across all 16 banks All-Bank CENT OP EOC = 0xFF // End of Computation terminates the aim simulator trace }; pragma pack push, 1 typedef struct CentInstruction { uint8 t opcode; // 8 bits : CENT ISA opcode uint8 t dvid; // 8 bits : Target CXL device ID 0 to 31 uint32 t channel mask; // 32 bits : Bitmask of the 32 GDDR6 channels uint16 t bank index; // 16 bits : DRAM bank index 0 to 15 uint32 t row addr; // 32 bits : Physical row address Row Buffer uint16 t opsize; // 16 bits : Operation size in bursts } CentInstruction; pragma pack pop static assert sizeof CentInstruction == 14, "ABI Error: CentInstruction must be exactly 14 bytes packed" ; The pragma pack push, 1 directive verified by static assert eliminates alignment padding between fields, fixing each instruction at 14 contiguous bytes 1 + 1 + 4 + 2 + 4 + 2 . During cent dispatch gemv src/runtime api.cpp , four stages run in sequence: src/topology.cpp : DVid = layer id % 32 + 32 % 32 to distribute layers across the 32 CXL devices. The channel mask 0xFFFFFFFFu when K = 1024 , 16 channels 0x0000FFFFu when K = 512 , and 8 channels 0x000000FFu for smaller dimensions. src/sram buffer.cpp : SramBuffer manages a preallocated static 2 MB std::array