Building a Micro AI Code Reviewer in Rust: Lessons from 'ratatop' with Unsafe and System Metrics

A developer detailed the architecture of ratatop, a micro AI code reviewer built in Rust, emphasizing deterministic low-latency execution and zero-copy memory management for large diffs. The project leverages unsafe blocks for performance-critical paths and integrates prometheus and libbpf for real-time system metrics monitoring.

Originally published on tamiz.pro. In the world of CI/CD, AI-powered code review tools are becoming ubiquitous. However, most of these solutions are heavyweight Python or Node.js services that introduce significant latency into the pull request workflow. They often suffer from cold starts, high memory footprints, and non-deterministic execution times. This deep dive explores the architecture and engineering decisions behind ratatop , a micro AI code reviewer designed to run locally or in lightweight containers on every commit. Built entirely in Rust , the project prioritizes deterministic low-latency execution, zero-copy memory management for large diffs, and deep integration with system-level metrics. We will dissect how we leveraged unsafe blocks for performance-critical paths and how we integrated prometheus and libbpf to monitor the reviewer's impact on the host system in real-time. Before diving into the code, it is crucial to understand why Rust was chosen over more traditional languages for this specific use case. While Python is the lingua franca of AI/ML, it is often too slow and memory-inefficient for high-throughput, low-latency system tooling. Rust offers three distinct advantages for building a micro AI reviewer: libgit2 or unidiff are written in C or C++. Rust’s Foreign Function Interface FFI allows us to call these libraries directly, avoiding the need to rewrite complex low-level logic in Rust.One of the most performance-critical components of any code reviewer is the diff parser. When a developer pushes a commit with thousands of lines changed, parsing the diff, extracting context, and feeding it to an LLM can be expensive in terms of memory allocations. In Python, parsing a large diff often involves creating numerous string objects, leading to significant memory churn. In Rust, we can avoid this by using zero-copy techniques, primarily through unsafe blocks. Consider the following naive approach to extracting a changed line from a diff: // Naive approach - creates many allocations fn extract changes naive diff text: &str - Vec<String { diff text .lines .filter |line| line.starts with '+' || line.starts with '-' .map |line| line 1.. .to string // Allocates a new String for each line .collect } This function allocates memory for every changed line. In a large diff with 10,000 changes, this results in 10,000 heap allocations. While modern allocators are fast, this overhead adds up, especially when processing multiple files concurrently. Instead of creating new String objects, we can work directly with &str slices that point to the original buffer. This eliminates heap allocations entirely. However, if the diff data comes from a C library via FFI, we might receive a mut c char a raw pointer to a C string . Converting this safely requires unsafe code. Here is how we implemented a zero-copy diff extractor: use std::ffi::CStr; /// Extracts changed lines from a C-style diff buffer without allocating new strings. /// Returns slices pointing directly into the original buffer. fn extract changes zero copy diff buffer: mut libc::c char - Vec<&str { unsafe { // Safety: We assume diff buffer is a valid, null-terminated C string // and that the lifetime of the returned slices does not exceed the buffer's lifetime. let c str = CStr::from ptr diff buffer ; let diff text = c str.to str .unwrap or "\0" ; diff text .lines .filter |line| line.starts with '+' || line.starts with '-' .map |line| &line 1.. // Returns a &str slice, no allocation .collect } } unsafe is Justified Here The unsafe block is justified because: CStr::from ptr . &str that borrow from the original buffer. We must ensure that the buffer outlives these references. In our architecture, the buffer is owned by a Vec<u8 that lives for the duration of the review process, ensuring safety.The core of ratatop is its ability to send diffs to an LLM e.g., via OpenAI, Anthropic, or a local model like Llama 3 and parse the response. However, LLM APIs are inherently non-deterministic in terms of latency. A review that takes 2 seconds one time might take 10 seconds the next. To mitigate this, we implemented a circuit breaker pattern and streaming responses with timeout controls. Instead of waiting for the entire LLM response, we stream the tokens and parse them incrementally. This allows us to provide feedback to the user or the CI system faster. use async openai::config::OpenAIConfig; use async openai::types::{CreateChatCompletionRequestArgs, Role, Content}; use async openai::Client; async fn stream review diff content: String - Result<Vec<String , Box<dyn std::error::Error { let config = OpenAIConfig::from env ; let client = Client::with config config ; let request = CreateChatCompletionRequestArgs::default .model "gpt-4o-mini" .messages vec async openai::types::ChatCompletionRequestMessage::User async openai::types::ChatCompletionUserMessage { content: Content::Text diff content , name: None, } .max tokens 500 .build ?; let mut stream = client.chat .create stream request .await?; let mut reviews = Vec::new ; while let Some result = stream.next .await { match result { Ok response = { if let Some choice = response.choices.first { if let Some text = &choice.delta.content { reviews.push text.clone ; } } } Err e = { // Handle error, possibly with retry logic eprintln "Error in stream: {}", e ; break; } } } Ok reviews } We wrap the LLM call in a timeout to prevent hanging. If the LLM API is slow, we fall back to a cached review or a rule-based heuristic. use tokio::time::{timeout, Duration}; async fn review with timeout diff content: String - Result<Vec<String , Box<dyn std::error::Error { let timeout duration = Duration::from secs 5 ; match timeout timeout duration, stream review diff content .await { Ok Ok reviews = Ok reviews , Ok Err e = Err e , Err = { // Timeout occurred, fall back to heuristic eprintln "LLM call timed out. Using heuristic fallback." ; Ok vec " Heuristic Possible issue detected in diff.".to string } } } To monitor the performance of ratatop in production, we integrated libbpf , a Rust binding for eBPF Extended Berkeley Packet Filter . eBPF allows us to observe the behavior of the reviewer at the kernel level without modifying the kernel code. Traditional monitoring tools like Prometheus exporters rely on instrumentation within the application code. However, this can introduce overhead and may not capture system-level events like context switches or page faults. eBPF provides a low-overhead way to observe the system. We used eBPF to monitor the number of context switches performed by the ratatop process during a review. High context switches can indicate contention or inefficient scheduling. use libbpf rs::skel::SkelBuilder; use libbpf rs::OpenSkel; use libbpf rs::Skel; // Assume we have an eBPF skeleton generated from a C program // that counts context switches for a specific PID. fn setup bpf monitor pid: u32 - Result< , Box<dyn std::error::Error { let skel builder = MySkelBuilder::default ; let mut open skel = skel builder.open ?; // Set the PID to monitor open skel.maps .ro data .pid to monitor = pid; let mut skel = open skel.load ?; skel.attach ?; // Now, we can read the maps from the eBPF program // to get context switch counts println "eBPF monitor attached for PID {}", pid ; Ok } We exported the eBPF metrics to Prometheus using the prometheus crate. This allows us to create dashboards that show the relationship between eBPF metrics context switches, page faults and LLM latency. use prometheus::{register int counter, IntCounter}; static CONTEXT SWITCHES: Lazy<IntCounter = Lazy::new || { register int counter "ratatop context switches total", "Total context switches during review" .unwrap } ; fn record context switches count: u64 { CONTEXT SWITCHES.inc by count ; } Building ratatop taught us several valuable lessons about building micro AI services in Rust: unsafe is a Tool, Not a Crutch: unsafe sparingly, only where it provided clear performance benefits zero-copy parsing . Every unsafe block was thoroughly documented and tested. ratatop demonstrates that Rust is an excellent choice for building micro AI services that require low latency, high throughput, and system-level observability. By leveraging unsafe blocks for zero-copy memory management and eBPF for deep system monitoring, we were able to create a reviewer that is both fast and insightful. For developers looking to build similar tools, we recommend starting with a modular architecture, embracing Rust's type system for safety, and not being afraid to use unsafe where it provides clear benefits. Additionally, integrating eBPF can provide a level of observability that is difficult to achieve with traditional monitoring tools. Q: Is it safe to use unsafe blocks for zero-copy parsing? Q: How do I handle errors from LLM APIs? A: We recommend implementing a retry mechanism with exponential backoff, as well as a circuit breaker to fall back to heuristic-based reviews if the LLM API is consistently slow or unavailable. Q: Can I use eBPF on Windows or macOS? A: eBPF is primarily supported on Linux. For Windows and macOS, you may need to use alternative monitoring tools or containerize the reviewer on a Linux kernel. For more insights on building high-performance Rust applications, check out Tamiz's Insights https://tamiz.pro/insights .