{"slug": "cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture", "title": "cent_runtime: C++17 Runtime and C ABI for the CENT Near-Bank PIM Architecture", "summary": "A developer built cent_runtime, a C++17 runtime with a C ABI and Python bindings that connects PyTorch linear layers from models such as Llama-2 7B to the CENT near-bank processing-in-memory instruction set architecture. The project targets the gap between the CENT paper's simulated gains for GPU-free LLM inference over CXL 3.0 and executable software, dispatching work to the official aim_simulator (Ramulator 2.0) and the GDDR6-AiM execution model. The report frames the effort around decoding's memory-bound GEMV phase, which accounts for roughly 98% of inference time.", "body_md": "The CENT architecture (*PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference*, Gu et al., ASPLOS 2025) proposes running Large Language Model (LLM) inference without GPUs, replacing HBM memory with GDDR6-AiM modules connected over CXL 3.0.\n\nWhile the analytical and simulation gains reported in the paper are substantial, a practical gap separates academic results from software execution: there is no system layer capable of taking PyTorch tensors and dispatching them onto the architecture's execution model.\n\nThis technical report documents the architectural context, the physical bottlenecks motivating the project, the role of a hardware dispatcher, how the official simulator (`aim_simulator` / Ramulator 2.0) operates, and the implementation of [`cent_runtime`](https://github.com/josedercysilva/cent-pim-runtime) — a C++17, C ABI, and Python library connecting linear layers of models such as Llama-2 7B to the CENT instruction set architecture. The seminar slide deck accompanying this work is available at the repository root in [`apresentacao_cent.pdf`](https://github.com/josedercysilva/cent-pim-runtime/blob/main/apresentacao_cent.pdf).\n\n`aim_simulator` (Ramulator 2.0)`cent_runtime`\nIn classical computer architecture (Patterson & Hennessy, Ch. 5), the memory hierarchy balances speed, capacity, and cost across successive tiers:\n\n``` php\nCPU / GPU  <--->  L1 Cache  <--->  L2 Cache  <--->  L3 Cache  <--->  Main Memory (DRAM)  <--->  SSD / Disk\n (fastest, smallest capacity)                                                        (slowest, largest capacity)\n```\n\nThis organization relies on temporal and spatial locality: a block fetched from DRAM must be reused dozens or hundreds of times in caches before eviction. LLM inference breaks this assumption during its longest execution phase.\n\nInside a DRAM chip, data is organized into two-dimensional cell arrays. Activating a row loads an entire 8,192-bit block at once into the internal Row Buffer in roughly 1 internal access cycle — meaning high internal bandwidth exists within the memory die itself.\n\nThe bottleneck sits at the chip's output interface. Moving those 8,192 bits to the processor (CPU or GPU) requires crossing the external bus and physical I/O pins operating at roughly 64 bits per cycle (or 32 bits in specific pin configurations). Transferring a single row that was already sitting in the Row Buffer takes roughly 128 bus cycles. While data crosses the bus, GPU compute units remain in a *stall* state waiting on memory.\n\n`GEMM`) vs. Decoding (` GEMV`)\nLLM inference splits into two phases with opposite physical profiles:\n\n| Characteristic | Phase 1: Prefill (Initial Prompt) | Phase 2: Decoding (Token-by-Token Generation) | \n|---|---|---|\n| Share of total runtime | ~2% of inference time | ~98% of inference time | \n| Mathematical operation | Matrix × Matrix ( `GEMM` , e.g.,`4K×4K` ×`4K×4K` ) | Vector × Matrix ( `GEMV` ,`batch_size = 1` ) | \n| Arithmetic intensity | High (~2,000 Ops/Byte) | Low (1 to 16 Ops/Byte) | \n| Execution regime | Compute-Bound (saturates Tensor Cores) | Memory-Bound (limited by memory bandwidth) | \n\nDuring decoding, generating each token requires reading entire weight matrices (tens of megabytes per layer, totaling tens of gigabytes across the model) only to compute a single vector-matrix product and discard the weights immediately after. On a 4× NVIDIA A100 node, this lack of reuse drops effective Tensor Core utilization to roughly 21% (with 79% of execution time spent stalled on HBM reads).\n\nProcessing-in-Memory (PIM) — specifically the GDDR6-AiM (*Accelerator-in-Memory*) design adopted in CENT — places multiply-accumulate execution units (PUs/ALUs) directly beside each DRAM bank, ahead of the narrow I/O pin interface.\n\nEach GDDR6-AiM chip therefore consumes weights directly at the Row Buffer output without streaming weight matrices over the external bus:\n\nInstead of dedicated GPUs and HBM interposers, CENT connects a Host CPU to 32 CXL 3.0 devices through a 48-port CXL Switch:\n\n```\n                       +------------+\n                       |  Host CPU  |\n                       +-----+------+\n                             |\n                             v\n                   +--------------------+\n                   |   CXL 3.0 Switch   |\n                   +---------+----------+\n                             |\n        +--------------------+--------------------+\n        |                    |                    |\n        v                    v                    v\n+---------------+    +---------------+    +---------------+\n| CXL Device #0 |    | CXL Device #1 | .. | CXL Device #31|\n+-------+-------+    +---------------+    +---------------+\n        |\n        +---> CXL Port + Inter-device Communication Controller\n        +---> Instruction Buffer (2 MB SRAM) -> PC -> Decoder\n        +---> PNM Units + Shared Buffer (64 KB)\n        +---> 16x PIM Ctrl ---> 16x GDDR6-PIM (32 channels per board)\n```\n\nInside each CXL device, execution is divided across two tiers:\n\n`Softmax`, `RMSNorm`, `SiLU`, and `RoPE`, transferring only compact partial vectors (<16 KB).\nIn the paper's evaluation against a 4× NVIDIA A100 server (80 GB per GPU):\n\nAt the same time, the architecture has three clear technical limitations:\n\nThis third gap is the focus of our work: building the system layer that connects high-level tensors to the architecture's instruction format and cycle-accurate simulator.\n\nTo validate the microarchitecture, the CENT authors released `aim_simulator`, built on top of Ramulator 2.0 (a cycle-accurate academic DRAM simulator).\n\n`aim_simulator` compiles into a `ramulator2` executable (paired with `libramulator.so`). It is not a neural network runtime, but a discrete-event hardware simulator:\n\n```\n  [ configs/gddr6_aim.yaml ] ──┐\n  (Physical parameters:        │\n   32 channels, 16 banks,      ├──► [ ramulator2 (aim_simulator) ] ──► Silicon Counters\n   GDDR6 timings, FRFCFS)      │                                       (e.g. CH31_AiM_ISR_WR_GB_cycles,\n                               │                                             memory_system_cycles)\n  [ file.trace ] ──────────────┘\n  (Static list of textual\n   CENT ISA commands)\n```\n\nWhen invoked from the command line (`ramulator2 -f configs/gddr6_aim.yaml -t file.trace`), it models Row Buffer accesses, bank conflicts, `FRFCFS` command scheduling, `AllBank` refresh, and AiM instructions cycle by cycle across 32 GDDR6-AiM channels, printing per-instruction cycle counters on completion.\n\n`.trace` files:`AiM WR_GB`, `AiM MAC_ABK`, etc., already decomposed into physical rows and channel masks.`y` for the next model layer.\nIn systems software, a dispatcher is the intermediary layer sitting between a high-level framework and the accelerator controller. When PyTorch runs `torch.matmul(x, W)` on a GPU, the call passes through the internal dispatcher into native C/C++ libraries (`libcudart.so`, `libcublas.so`) that configure and submit work to the device.\n\nFor CENT, the hardware dispatcher fulfills four responsibilities:\n\n`batch_size = 1`). Prefill `batch_size > 1`) are rejected so they remain on the host.` DVid`) hosts the layer and which of the 32 GDDR6 channels (` CHmask`) are enabled for dimension `K`.`[M × K]` weight matrix into blocks aligned to the 16 DRAM banks per channel, writes the binary descriptors into the 2 MB SRAM buffer, and emits the corresponding `cent_runtime` structures this dispatcher into two main layers — Python (PyTorch/NumPy) and C++17 connected across a stable C ABI (`libcent_runtime.so`) — plus an optional integration module for `ramulator2`.\n\n```\n+-----------------------------------------------------------------------------+\n| 1. PYTHON LAYER (python/cent_runtime/)                                      |\n|    cent_runtime.matmul(x, W, layer_id=0, trace_path=None, simulate=False)   |\n|    - Validates GEMV contract (batch=1) in ops.py                            |\n|    - Normalizes to contiguous float32 and extracts pointer via ctypes       |\n+--------------------------------------+--------------------------------------+\n                                       | ctypes (_bindings.py)\n                                       v\n======================= [ C ABI BOUNDARY (extern \"C\") ] =======================\n| 2. PUBLIC CONTRACT (include/cent/runtime_api.h -> build/libcent_runtime.so) |\n|    cent_dispatch_gemv(x, M, K, W, y, layer_id, trace_path, &telemetry)      |\n+--------------------------------------+--------------------------------------+\n                                       |\n          +----------------------------+----------------------------+\n          v                                                         v\n+-----------------------------------+     +-----------------------------------+\n| 3. C++17 ENGINE (src/)            |     | 4. NUMERICAL COMPUTATION          |\n|  - topology.cpp   -> DVid/CHmask  |     |  - Row-major loop over W [M x K]  |\n|  - sram_buffer.cpp -> 2 MB SRAM   |     |  - FP64 accumulation -> FP32 out  |\n|  - isa_emitter.cpp -> PIM blocks  |     |  - Writes directly to pointer 'y' |\n+-----------------+-----------------+     +-----------------------------------+\n                  |\n                  +---> Binary packets (14 B) in SRAM + Hex Dump\n                  +---> .trace file (CENT ISA) ---> simulator.py -> ramulator2\n```\n\n`python/cent_runtime/`)\nThe Python package exposes `cent_runtime.matmul`, `simulate_on_ramulator`, and `get_instruction_buffer_hex`:\n\n`ops.py`:` torch.Tensor` or `numpy.ndarray` (falling back to NumPy when PyTorch is not installed). Validates `x` as `(K,)` or `(1, K)` and `W` as `(M, K)`. 3D+ tensors or `ValueError`. It then ensures a contiguous `float32` layout on CPU (performing no extra copy when the tensor already satisfies this layout), allocates the output tensor `ctypes`.`_bindings.py`:` build/libcent_runtime.so` (running `make shared` automatically if the library has not been built yet) and maps the C struct `CentDispatchTelemetry` (56 bytes on x86_64) into a Python dictionary.`simulator.py`:` simulate=True` (creating a temporary `tempfile` if `trace_path=None`), resolves the `ramulator2` binary in configurable order (explicit `ramulator_bin` → `CENT_RAMULATOR_BIN` environment variable → `external/ramulator2` → `../CENT/aim_simulator/build/ramulator2` → `PATH`), invokes the simulator via `subprocess`, and extracts `wr_gb_cycles`, `mac_abk_cycles`, and `memory_system_cycles`.`.h`) and C++17 Core (`.hpp`)\nIn C++, namespaced or overloaded functions undergo *name mangling* (for instance, `cent::compute_dvid(int)` becomes `_ZN4cent12compute_dvidEi` in the symbol table), and the C++ ABI differs across compilers. To guarantee direct FFI interoperability with `ctypes`, the codebase separates:\n\n`isa.h`, `telemetry.h`, `runtime_api.h`):` extern \"C\"`, exporting unmangled symbols (` cent_init_runtime`, `cent_dispatch_gemv`, `cent_get_instruction_buffer_hex`).` topology.hpp`, `sram_buffer.hpp`, `isa_emitter.hpp`):` namespace cent`, classes, RAII, and `std::array`.\nThe native library compiles with `-std=c++17 -O3 -fPIC -Wall -Wextra -Werror` and uses no C++ exceptions (`throw` / `try-catch`) across the boundary. All exported functions are marked `noexcept` and return explicit status codes via `enum CentStatus` (`CENT_OK = 0`, `CENT_ERR_NULL_PTR = -1`, `CENT_ERR_INVALID_SHAPE = -2`, `CENT_ERR_SRAM_OVERFLOW = -3`, `CENT_ERR_TRACE_IO = -4`).\n\n`CentInstruction`)\n`include/cent/isa.h` defines the controller parameters (`CENT_SRAM_BUFFER_BYTES = 2 MB`, 32 CXL devices, 32 channels per device, 16 banks per channel, and `opsize = 64` bursts), the 5 CENT ISA opcodes, and the packed binary instruction layout:\n\n```\nenum CentOpcode : uint8_t {\n    CENT_OP_WR_GB   = 0x02, // Writes activation vector to active channels' Global Buffer\n    CENT_OP_WR_BIAS = 0x03, // Initializes accumulator registers across DRAM banks\n    CENT_OP_RD_MAC  = 0x05, // Reads accumulated result from banks to channel buffer\n    CENT_OP_MAC_ABK = 0x0F, // Parallel Multiply-Accumulate across all 16 banks (All-Bank)\n    CENT_OP_EOC     = 0xFF  // End of Computation (terminates the aim_simulator trace)\n};\n\n#pragma pack(push, 1)\ntypedef struct CentInstruction {\n    uint8_t  opcode;       //  8 bits : CENT ISA opcode\n    uint8_t  dvid;         //  8 bits : Target CXL device ID (0 to 31)\n    uint32_t channel_mask; // 32 bits : Bitmask of the 32 GDDR6 channels\n    uint16_t bank_index;   // 16 bits : DRAM bank index (0 to 15)\n    uint32_t row_addr;     // 32 bits : Physical row address (Row Buffer)\n    uint16_t opsize;       // 16 bits : Operation size in bursts\n} CentInstruction;\n#pragma pack(pop)\n\nstatic_assert(sizeof(CentInstruction) == 14,\n              \"ABI Error: CentInstruction must be exactly 14 bytes packed\");\n```\n\nThe `#pragma pack(push, 1)` directive verified by `static_assert` eliminates alignment padding between fields, fixing each instruction at 14 contiguous bytes (`1 + 1 + 4 + 2 + 4 + 2`).\n\nDuring `cent_dispatch_gemv` (`src/runtime_api.cpp`), four stages run in sequence:\n\n`src/topology.cpp`):` DVid = ((layer_id % 32) + 32) % 32` to distribute layers across the 32 CXL devices. The channel mask `0xFFFFFFFFu`) when `K >= 1024`, 16 channels (` 0x0000FFFFu`) when `K >= 512`, and 8 channels (` 0x000000FFu`) for smaller dimensions.` src/sram_buffer.cpp`):` SramBuffer` manages a preallocated static 2 MB `std::array<uint8_t, CENT_SRAM_BUFFER_BYTES>`, avoiding dynamic allocation (` new`/` delete`) during dispatch. Instructions are copied via `std::memcpy` with bounds checking (`CENT_ERR_SRAM_OVERFLOW`) and can be inspected in hexadecimal through `format_hex_dump`.` src/isa_emitter.cpp`):` M` is partitioned into `num_blocks = (M + 15) / 16` blocks. For each block `blk` (`row_addr = blk`), the emitter writes 4 instructions into SRAM (` WR_GB`, `WR_BIAS`, `MAC_ABK`, and `RD_MAC`) and, when `trace_path` is provided, writes the corresponding lines to the \n\n```\n   AiM WR_GB 64 0 0xffffffff\n   AiM WR_BIAS 0 0xffffffff\n   AiM MAC_ABK 64 0xffffffff 0\n   AiM RD_MAC 0 0xffffffff\n```\n\nAfter all blocks complete, it emits a `CENT_OP_EOC` instruction (`AiM EOC`), required to terminate simulation in `aim_simulator`.\n\n`src/runtime_api.cpp` and `y = x · W^T` in row-major order using a `double` (`FP64`) accumulator cast to `float` (`FP32`) in the output pointer `(M × K + K) × 2` bytes, transferring both `(K + M) × 2` bytes, where only The repository includes automated Python unit tests (`make test`), a native C++ testbench (` make test-core`), and a validation script exercising real Llama-2 7B layer dimensions (`examples/llama2_7b_demo.py`).\n\n```\ngit clone https://github.com/josedercysilva/cent-pim-runtime.git\ncd cent-pim-runtime\n\n./setup_env.sh\nsource .venv/bin/activate\n\nmake test        # Tests C ABI, numerical accuracy, GEMM rejection, SRAM, and .trace\nmake test-core   # Native C++ testbench\nmake demo        # Runs Llama-2 7B Attention and MLP layers\nmake simulate    # Optional: runs Ramulator 2.0 on the generated .trace files\npython\nimport torch\nimport torch.nn.functional as F\nimport cent_runtime\n\n# Decoding phase (batch_size = 1)\nx = torch.randn(1, 4096)        # Input activation\nW = torch.randn(4096, 4096)     # Stationary weights in GDDR6-AiM memory\n\n# Dispatch to the C++ runtime (returns PyTorch tensor + telemetry)\ny, telemetry = cent_runtime.matmul(x, W, layer_id=0)\n\n# Non-linear operations remain on CPU via PyTorch\nout = F.silu(y)\n\nprint(f\"Output: {out.shape}\")\nprint(f\"CXL Board: #{telemetry['dvid']} | Mask: {telemetry['channel_mask_hex']}\")\nprint(f\"Instructions: {telemetry['total_instructions']} ({telemetry['instruction_buffer_bytes']} B SRAM)\")\nprint(f\"Bus traffic saved: {telemetry['memory_traffic_saved_pct']:.2f}%\")\n```\n\nThe table below summarizes the results obtained for the Attention projection (`q/k/v/o_proj`, `M = 4096, K = 4096, layer_id = 0`) and the MLP projection (` gate/up/down_proj`, `M = 11008, K = 4096, layer_id = 1`):\n\n| Metric | Attention Layer ( `4096 × 4096` ) | MLP Layer ( `11008 × 4096` ) | \n|---|---|---|\n| CXL device ( `DVid` ) | Board `#0` | Board `#1` | \n| Active channels ( `CHmask` ) | `0xffffffff` (32 channels) | `0xffffffff` (32 channels) | \n| 16-bank blocks ( `ceil(M/16)` ) | 256 blocks | 688 blocks | \n| Total instructions ( `4/block + EOC` ) | 1,025 ( `256×4 + 1` ) | 2,753 ( `688×4 + 1` ) | \n| SRAM footprint ( `14 B/inst` ) | 14,350 bytes (14.01 KB) | 38,542 bytes (37.64 KB) | \n| GPU HBM vs. CENT CXL traffic (BF16) | ~32.01 MB vs. 16.00 KB (99.95% lower) | ~86.01 MB vs. 29.50 KB (99.97% lower) | \n| Ramulator 2.0 cycles ( `CH31_AiM_ISR_WR_GB_cycles` ) | 16,416 cycles | 44,064 cycles | \n\nThe ratio between the two layers reflects the matrix dimensions directly: the MLP output dimension is `11,008 / 4,096 = 2.6875×` larger than Attention, and both the total SRAM instruction count and the silicon cycle count reported by `aim_simulator` scale by the same `2.68×` factor (`44,064 / 16,416`).\n\nInspecting the first 16 SRAM bytes via `cent_get_instruction_buffer_hex` shows the first 14-byte packet serialized in *little-endian* order (`opcode = 0x02` for `WR_GB`, `dvid = 0x00`, `channel_mask = FF FF FF FF`, `bank = 00 00`, `row = 00 00 00 00`, `opsize = 64 = 0x0040 -> 40 00`), followed by the first two bytes of the second instruction (`03 00`, `WR_BIAS`):\n\n```\n[SRAM] 02 00 FF FF FF FF 00 00 00 00 00 00 40 00 03 00\n```\n\n`cent_runtime` connects the CENT architectural specification to an executable Python and C++17 workflow:\n\n`josedercysilva/cent-pim-runtime`` apresentacao_cent.pdf`", "url": "https://wpnews.pro/news/cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture", "canonical_source": "https://dev.to/josedercysilva/centruntime-c17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture-18l4", "published_at": "2026-11-20 04:05:37+00:00", "updated_at": "2026-11-20 04:13:21.657856+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-chips", "ai-research", "developer-tools"], "entities": ["cent_runtime", "CENT", "GDDR6-AiM", "CXL 3.0", "aim_simulator", "Ramulator 2.0", "Llama-2 7B", "NVIDIA A100"], "also_reported_by": [], "takeaway": "cent_runtime provides a C++17/C ABI/Python layer that maps PyTorch linear layers onto the CENT PIM instruction set, letting Llama-2 7B layers run against the aim_simulator instead of requiring a custom software stack.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture", "markdown": "https://wpnews.pro/news/cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture.md", "text": "https://wpnews.pro/news/cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture.txt", "jsonld": "https://wpnews.pro/news/cent-runtime-c-17-runtime-and-c-abi-for-the-cent-near-bank-pim-architecture.jsonld"}}