Originally published on tamiz.pro.
In the world of CI/CD, AI-powered code review tools are becoming ubiquitous. However, most of these solutions are heavyweight Python or Node.js services that introduce significant latency into the pull request workflow. They often suffer from cold starts, high memory footprints, and non-deterministic execution times.
This deep dive explores the architecture and engineering decisions behind ratatop, a micro AI code reviewer designed to run locally or in lightweight containers on every commit. Built entirely in Rust, the project prioritizes deterministic low-latency execution, zero-copy memory management for large diffs, and deep integration with system-level metrics. We will dissect how we leveraged unsafe
blocks for performance-critical paths and how we integrated prometheus
and libbpf
to monitor the reviewer's impact on the host system in real-time.
Before diving into the code, it is crucial to understand why Rust was chosen over more traditional languages for this specific use case. While Python is the lingua franca of AI/ML, it is often too slow and memory-inefficient for high-throughput, low-latency system tooling.
Rust offers three distinct advantages for building a micro AI reviewer:
libgit2
or unidiff
) are written in C or C++. Rust’s Foreign Function Interface (FFI) allows us to call these libraries directly, avoiding the need to rewrite complex low-level logic in Rust.One of the most performance-critical components of any code reviewer is the diff parser. When a developer pushes a commit with thousands of lines changed, parsing the diff, extracting context, and feeding it to an LLM can be expensive in terms of memory allocations.
In Python, parsing a large diff often involves creating numerous string objects, leading to significant memory churn. In Rust, we can avoid this by using zero-copy techniques, primarily through unsafe
blocks.
Consider the following naive approach to extracting a changed line from a diff:
// Naive approach - creates many allocations
fn extract_changes_naive(diff_text: &str) -> Vec<String> {
diff_text
.lines()
.filter(|line| line.starts_with('+') || line.starts_with('-'))
.map(|line| line[1..].to_string()) // Allocates a new String for each line
.collect()
}
This function allocates memory for every changed line. In a large diff with 10,000 changes, this results in 10,000 heap allocations. While modern allocators are fast, this overhead adds up, especially when processing multiple files concurrently.
Instead of creating new String
objects, we can work directly with &str
slices that point to the original buffer. This eliminates heap allocations entirely. However, if the diff data comes from a C library via FFI, we might receive a *mut c_char
(a raw pointer to a C string). Converting this safely requires unsafe
code.
Here is how we implemented a zero-copy diff extractor:
use std::ffi::CStr;
/// Extracts changed lines from a C-style diff buffer without allocating new strings.
/// Returns slices pointing directly into the original buffer.
fn extract_changes_zero_copy(diff_buffer: *mut libc::c_char) -> Vec<&str> {
unsafe {
// Safety: We assume diff_buffer is a valid, null-terminated C string
// and that the lifetime of the returned slices does not exceed the buffer's lifetime.
let c_str = CStr::from_ptr(diff_buffer);
let diff_text = c_str.to_str().unwrap_or("\0");
diff_text
.lines()
.filter(|line| line.starts_with('+') || line.starts_with('-'))
.map(|line| &line[1..]) // Returns a &str slice, no allocation
.collect()
}
}
unsafe
is Justified Here
The unsafe
block is justified because:
CStr::from_ptr
.&str
) that borrow from the original buffer. We must ensure that the buffer outlives these references. In our architecture, the buffer is owned by a Vec<u8>
that lives for the duration of the review process, ensuring safety.The core of ratatop is its ability to send diffs to an LLM (e.g., via OpenAI, Anthropic, or a local model like Llama 3) and parse the response. However, LLM APIs are inherently non-deterministic in terms of latency. A review that takes 2 seconds one time might take 10 seconds the next.
To mitigate this, we implemented a circuit breaker pattern and streaming responses with timeout controls.
Instead of waiting for the entire LLM response, we stream the tokens and parse them incrementally. This allows us to provide feedback to the user (or the CI system) faster.
use async_openai::config::OpenAIConfig;
use async_openai::types::{CreateChatCompletionRequestArgs, Role, Content};
use async_openai::Client;
async fn stream_review(diff_content: String) -> Result<Vec<String>, Box<dyn std::error::Error>> {
let config = OpenAIConfig::from_env();
let client = Client::with_config(config);
let request = CreateChatCompletionRequestArgs::default()
.model("gpt-4o-mini")
.messages(vec![
async_openai::types::ChatCompletionRequestMessage::User(
async_openai::types::ChatCompletionUserMessage {
content: Content::Text(diff_content),
name: None,
}
)
])
.max_tokens(500)
.build()?;
let mut stream = client.chat().create_stream(request).await?;
let mut reviews = Vec::new();
while let Some(result) = stream.next().await {
match result {
Ok(response) => {
if let Some(choice) = response.choices.first() {
if let Some(text) = &choice.delta.content {
reviews.push(text.clone());
}
}
}
Err(e) => {
// Handle error, possibly with retry logic
eprintln!("Error in stream: {}", e);
break;
}
}
}
Ok(reviews)
}
We wrap the LLM call in a timeout to prevent hanging. If the LLM API is slow, we fall back to a cached review or a rule-based heuristic.
use tokio::time::{timeout, Duration};
async fn review_with_timeout(diff_content: String) -> Result<Vec<String>, Box<dyn std::error::Error>> {
let timeout_duration = Duration::from_secs(5);
match timeout(timeout_duration, stream_review(diff_content)).await {
Ok(Ok(reviews)) => Ok(reviews),
Ok(Err(e)) => Err(e),
Err(_) => {
// Timeout occurred, fall back to heuristic
eprintln!("LLM call timed out. Using heuristic fallback.");
Ok(vec!["[Heuristic] Possible issue detected in diff.".to_string()])
}
}
}
To monitor the performance of ratatop in production, we integrated libbpf, a Rust binding for eBPF (Extended Berkeley Packet Filter). eBPF allows us to observe the behavior of the reviewer at the kernel level without modifying the kernel code.
Traditional monitoring tools (like Prometheus exporters) rely on instrumentation within the application code. However, this can introduce overhead and may not capture system-level events like context switches or page faults. eBPF provides a low-overhead way to observe the system.
We used eBPF to monitor the number of context switches performed by the ratatop process during a review. High context switches can indicate contention or inefficient scheduling.
use libbpf_rs::skel::SkelBuilder;
use libbpf_rs::OpenSkel;
use libbpf_rs::Skel;
// Assume we have an eBPF skeleton generated from a C program
// that counts context switches for a specific PID.
fn setup_bpf_monitor(pid: u32) -> Result<(), Box<dyn std::error::Error>> {
let skel_builder = MySkelBuilder::default();
let mut open_skel = skel_builder.open()?;
// Set the PID to monitor
open_skel.maps().ro_data().pid_to_monitor = pid;
let mut skel = open_skel.load()?;
skel.attach()?;
// Now, we can read the maps from the eBPF program
// to get context switch counts
println!("eBPF monitor attached for PID {}", pid);
Ok(())
}
We exported the eBPF metrics to Prometheus using the prometheus
crate. This allows us to create dashboards that show the relationship between eBPF metrics (context switches, page faults) and LLM latency.
use prometheus::{register_int_counter, IntCounter};
static CONTEXT_SWITCHES: Lazy<IntCounter> = Lazy::new(|| {
register_int_counter!("ratatop_context_switches_total", "Total context switches during review").unwrap()
});
fn record_context_switches(count: u64) {
CONTEXT_SWITCHES.inc_by(count);
}
Building ratatop taught us several valuable lessons about building micro AI services in Rust:
unsafe
is a Tool, Not a Crutch:unsafe
sparingly, only where it provided clear performance benefits (zero-copy parsing). Every unsafe
block was thoroughly documented and tested.ratatop demonstrates that Rust is an excellent choice for building micro AI services that require low latency, high throughput, and system-level observability. By leveraging unsafe
blocks for zero-copy memory management and eBPF for deep system monitoring, we were able to create a reviewer that is both fast and insightful.
For developers looking to build similar tools, we recommend starting with a modular architecture, embracing Rust's type system for safety, and not being afraid to use unsafe
where it provides clear benefits. Additionally, integrating eBPF can provide a level of observability that is difficult to achieve with traditional monitoring tools.
Q: Is it safe to use unsafe blocks for zero-copy parsing?
Q: How do I handle errors from LLM APIs?
A: We recommend implementing a retry mechanism with exponential backoff, as well as a circuit breaker to fall back to heuristic-based reviews if the LLM API is consistently slow or unavailable.
Q: Can I use eBPF on Windows or macOS?
A: eBPF is primarily supported on Linux. For Windows and macOS, you may need to use alternative monitoring tools or containerize the reviewer on a Linux kernel.
For more insights on building high-performance Rust applications, check out Tamiz's Insights.