How to Profile Rust Performance Using Grafana and Prometheus

The hotpath-rs Rust profiling library added a hotpath-prometheus feature flag that exposes performance metrics on a Prometheus-compatible endpoint at port 6772, according to a tutorial published October 12, 2026. Enabling the flag in hotpath version 0.28 and running the process with --features='hotpath,hotpath-alloc,hotpath-prometheus' emits metrics such as hotpath_build_info, hotpath_uptime_seconds and hotpath_function_calls_total for scraping by Prometheus and display in Grafana. The tutorial argues raw function timings and allocation numbers lack value without context, citing an example where one function has a p95 of 10μs while another averages over 5 seconds.

How to Profile Rust Performance using Grafana and Prometheus how-to-profile-rust-performance-using-grafana-and-prometheus Published: October 12, 2026 Reading time: 21 minutes With agentic AI writing more code, production observability becomes critical to understanding what your app is actually doing. In this tutorial, I’ll show how to use the hotpath-rs Grafana & Prometheus integration to configure actionable performance metrics for your Rust application. We’ll discuss which signals are worth observing and how to ensure that profiling overhead does not impact your production deployment. Getting started with Grafana in Rust getting-started-with-grafana-in-rust In this article I won’t describe how to get started with Grafana itself. There are numerous tutorials and AI skills dedicated to that. I assume you already have Grafana and Prometheus up and running. Instead, we’ll focus on extracting metrics that provide useful insights without overwhelming you with distracting information. One issue I had when I started using Grafana was its almost infinite flexibility in how metrics can be queried and displayed. It makes deciding what’s actually worth monitoring difficult. Consider two functions: A has a p95 of 10μs , while B takes over 5 seconds on average. So A is fast and B is slow? What if A is a low-level processing hotpath called millions of times, while B represents a long-running background job? It all depends on CONTEXT . Raw function execution times or allocation metrics provide limited value without understanding how they relate to the application’s behavior. With AI writing increasingly more code, that context can be difficult to maintain even when you’re a supposed “author” of the feature. That’s why in this article I’ll show how to instrument a Rust application in a way that provides useful insights even with limited knowledge of the underlying codebase. And when you do know the codebase well, the same metrics become even more useful. How to configure Grafana and Prometheus metrics for Rust app? how-to-configure-grafana-and-prometheus-metrics-for-rust-app hotpath-rs makes it extremely easy to expose raw performance metrics through a Prometheus-compatible endpoint. See the Grafana & Prometheus docs https://hotpath.rs/prometheus grafana for detailed instructions and configuration options. If your Rust app is already instrumented with hotpath , all you need to do is enable the hotpath-prometheus feature flag: dependencies hotpath = "0.28" features hotpath = "hotpath/hotpath" hotpath-alloc = "hotpath/hotpath-alloc" hotpath-prometheus = "hotpath/hotpath-prometheus" run your process with this flag: cargo run --features='hotpath,hotpath-alloc,hotpath-prometheus' and your metrics will be exposed at the default 6772 port: curl http://127.0.0.1:6772/metrics | head TYPE hotpath build info gauge hotpath build info{hotpath version="0.28.3",rustc version="1.98.0",profile="debug",os="macos-aarch64"} 1 HELP hotpath uptime seconds Seconds since profiling started. TYPE hotpath uptime seconds gauge hotpath uptime seconds 3.309838291 HELP hotpath function calls total Total calls of each instrumented function, including calls skipped by time sampling. TYPE hotpath function calls total counter hotpath function calls total{function="hotpath::cmd::console::app::App::new"} 1 hotpath function calls total{function="hotpath::cmd::console::app::data::App::handle data response"} 8 Now point your Prometheus scraper at the endpoint and you’re good to go This could be the end of the tutorial. But read on if you’re interested in how hotpath-rs provides context-rich metrics that are easy to reason about. Layers of performance observability layers-of-performance-observability I’ve described my approach to optimizing Rust applications in the Rust performance profiling guide https://hotpath.rs/blog/profiling-rust-guide , so check it out to improve your context window. The core idea is to apply a Pareto principle mindset to performance optimization: focusing on high-level layers can take 5% of the effort while accounting for 95% of the potential improvements. Before migrating to Rust, I worked as a Ruby on Rails performance consultant for ~5 years and used the same approach when optimizing RoR backends. See My Recipe for Optimizing Performance of Rails Applications https://pawelurbanek.com/optimize-rails-performance for more details. This layered observability approach translates directly to how a useful Grafana dashboard can be structured. hotpath-rs lets you monitor different performance layers separately while using each one to provide context for the others. That’s enough “performance philosophy” for now. Let’s look at some code and charts. The sample metrics below come from the production deployment of hotpath-backend , the Axum app powering this blog and hotpath Cloud https://hotpath.rs/cloud app. SQL query performance in Grafana sql-query-performance-in-grafana Let’s start with SQL query performance. Database interactions are often among the biggest bottlenecks in backend applications. hotpath-rs offers SQL performance tracing https://hotpath.rs/sql tracing for popular SQL libraries. The instrumentation is agnostic to the underlying database engine, so it works with PostgreSQL, SQLite, and others. The following Grafana query shows where your app spends its database time. Each bar represents the percentage of total SQL execution time spent by a single query, accounting for both how slow the query is and how often it runs: sort desc topk 10, 100 sum by query histogram sum increase hotpath sql duration seconds $ range / on group left sum histogram sum increase hotpath sql duration seconds $ range Alternatively, you can use the following query to show only queries that cross a defined slowness threshold, e.g. p95 500ms , making it easy to spot candidates for optimization: sort desc histogram quantile 0.95, sum by query increase hotpath sql duration seconds $ range 0.500 Context-aware SQL performance metrics context-aware-sql-performance-metrics The metrics above can also be extracted directly from the database, for example with tools like pg-extras-rs https://github.com/pawurb/pg-extras-rs . You don’t need to run a dedicated profiler for that. Where hotpath-rs really starts to show its value is in contextualizing those metrics. If you’re using Axum https://github.com/tokio-rs/axum for your backend, you can analyze each SQL query in the context of the route that executed it https://hotpath.rs/axum tracing route-scoping-for-sql-and-http . Let’s start with a chart showing which Axum endpoints contribute most to the application’s total processing time: sort desc topk 10, 100 sum by route histogram sum increase hotpath server duration seconds $ range / on group left sum histogram sum increase hotpath server duration seconds $ range Now we can add a selector to inspect exactly which SQL queries are executed by each endpoint, and how much time each of them adds to a single request: sort desc topk 10, sum by query histogram sum increase hotpath sql duration seconds{route=~"$route"} $ range / scalar sum increase hotpath server scoped requests total{route=~"$route"} $ range $route is a Grafana query variable populated with label values hotpath server requests total, route . With this setup in place, you can easily discover which SQL queries meaningfully contribute to response time and which can be safely ignored, even without an in-depth knowledge of the underlying codebase. Detecting N+1 SQL queries in Rust detecting-n1-sql-queries-in-rust N+1 queries are one of the most common causes of slowdowns in backend applications. An endpoint loads a list of records and then runs a separate query for each one. This can be fast with a few rows in development but gets progressively slower as production data grows. Because each individual query is fast, it may never show up in a slow query report. Since hotpath-rs knows which Axum route executed each query, we can calculate how many SQL queries a single request triggers on average: sort desc topk 10, sum by route increase hotpath server sql calls total $ range / sum by route increase hotpath server scoped requests total $ range 0 Endpoints with an unexpectedly high number of queries per request are prime N+1 candidates. You can add a threshold, e.g. 5 , to show only the endpoints that cross it. Measuring HTTP call overhead measuring-http-call-overhead Next to SQL queries, HTTP calls https://hotpath.rs/http tracing are often another major contributor to a backend application’s response time. A useful graph can show how many HTTP calls each endpoint triggers on average. Multiple blocking HTTP calls can cause a problem similar to N+1 queries, with response times quickly growing as the number of calls increases: sort desc topk 10, sum by route increase hotpath server http calls total $ range 0 / sum by route increase hotpath server scoped requests total $ range 0 The next chart shows how much time a request to each endpoint spends waiting on blocking HTTP calls. A route that makes only one outbound call per request can still be the slowest if that call hits a slow external API: sort desc topk 10, sum by route histogram sum increase hotpath http duration seconds{route =""} $ range / sum by route increase hotpath server scoped requests total $ range 0 route ="" filters out HTTP calls made outside of a request, e.g. by background jobs. You can compare this HTTP overhead with the slowest endpoint response times to identify which blocking calls are worth optimizing or moving to the background. Memory usage per request memory-usage-per-request With the hotpath-alloc feature enabled, hotpath-rs also tracks how many bytes are allocated while serving each request https://hotpath.rs/axum tracing memory-per-route . The following chart shows how much memory a request to each endpoint allocates on average: sort desc topk 10, sum by route increase hotpath server alloc bytes total $ range / sum by route increase hotpath server scoped requests total $ range 0 If your app is using more memory than expected, this overview lets you quickly spot potential culprits. Endpoints allocating megabytes per request, e.g. by loading whole tables or large payloads into memory, are the first place to look. Once you’ve found a suspicious endpoint, you can narrow it down further. The following chart shows which instrumented functions contribute to its memory usage, measured as average bytes allocated per request: sort desc topk 10, sum by function increase hotpath function route alloc bytes total{route=~"$route"} $ range / scalar sum increase hotpath server scoped requests total{route=~"$route"} $ range By default, allocations are counted exclusively: each function reports only the memory it allocates itself, excluding nested instrumented calls. Here, comparisons controller::show does not include the bytes allocated by comparisons::build and render show , so each bar points directly to where the allocations happen. You can switch to cumulative accounting with HOTPATH ALLOC CUMULATIVE=true , but it double-counts recursive functions. Timing details per request timing-details-per-request The same approach works for response times. Let’s start with the slowest endpoints, ranked by their p95 response time: sort desc topk 10, histogram quantile 0.95, sum by route increase hotpath server duration seconds $ range And for a selected endpoint, we can check which instrumented functions contribute to its response time, measured as average time spent per request: sort desc topk 10, sum by function increase hotpath function route duration seconds total{route=~"$route"} $ range / sum by function increase hotpath function route timed calls total{route=~"$route"} $ range 0 sum by function increase hotpath function route calls total{route=~"$route"} $ range / scalar sum increase hotpath server scoped requests total{route=~"$route"} $ range Dividing by timed calls and multiplying by calls keeps the numbers correct when time sampling https://hotpath.rs/profiling overhead reducing-overhead-time-sampling is enabled. Keep in mind that this timing data is inclusive . A function’s time includes time spent in all the instrumented functions it calls, so comparisons controller::show above includes, for example, comparisons::build . Future releases of hotpath-rs will implement exclusive time attribution, showing only the time spent in each function’s own body. Discovering slow channel processing in Rust discovering-slow-channel-processing-in-rust Channels are one of the primary ways Rust programs communicate across threads and often represent important data flows in your application. The hotpath::channel https://hotpath.rs/data flow channels-monitoring macro lets you extract detailed performance metrics for any channel: Arguably one of the most useful metrics here is the maximum queue size. A growing queue means that consumers are not keeping up with incoming messages, providing a strong signal of a bottleneck in your application’s processing pipeline. hotpath-backend does not use any channels yet, so the following dashboard comes from the hotpath TUI demo. It shows the maximum queue size, throughput, and send-to-receive latency for each instrumented channel: demo-std-jobs is a clear bottleneck here. Its queue reached the full capacity of 8 messages, causing senders to block while waiting for a free slot. It’s also the busiest channel, and messages spend 296ms in the queue before being received, while other channels deliver them within microseconds. Maximum queue size since the process started, with channel capacity in the type label: sort desc topk 10, max by label, type hotpath channel max queue size Throughput as messages received per second: sort desc topk 10, sum by label, type rate hotpath channel received total $ range And p95 send-to-receive latency, measuring how long a message spends in the channel before being received: sort desc topk 10, histogram quantile 0.95, sum by label, type increase hotpath channel delay seconds $ range I like this instrumentation mode because it’s useful regardless of the type of Rust application or your knowledge of the codebase. A queue that keeps growing under constant load is a strong signal of congestion or an inefficient consumer. Lock contention lock-contention Similarly to channels, RwLocks and mutexes are commonly used to share data between threads. The hotpath::rw lock https://hotpath.rs/locks rw lock-macro and hotpath::mutex https://hotpath.rs/locks mutex-macro macros track two durations for every lock acquisition: - wait time - how long a thread was blocked before the lock was acquired. - hold time - how long the lock was held after it was acquired. hotpath-backend does not use any instrumented locks itself, but hotpath-rs profiles its own internals the same way. The following chart comes from production and shows the average hold time for reads and writes of each RwLock guarding the profiler’s internal state: sort desc topk 10, sum by label, op histogram sum increase hotpath rwlock acquire seconds $ range / sum by label, op histogram count increase hotpath rwlock acquire seconds $ range 0 While working on this post, I’ve realized that reads of server state hold the guard for over 10ms on average, several times longer than any other lock. That’s a sign of a too broad guard scope, e.g. doing expensive work like cloning or serializing data while the lock is still held. A read guard held that long blocks every writer in the meantime, so it’s something to look into. A long hold time only becomes a real problem when other threads have to wait for the lock. To check that, chart the p95 wait time per lock and side: sort desc topk 10, histogram quantile 0.95, sum by label, op increase hotpath rwlock wait seconds $ range A large wait time indicates that threads are competing for the same resource. Shortening the critical section, e.g. by preparing the new value before acquiring the lock, is a potential fix. For mutexes use hotpath mutex wait seconds and hotpath mutex acquire seconds instead, without the op label. Tokio runtime tokio-runtime Calling hotpath::tokio runtime in your async main exposes live Tokio runtime metrics https://hotpath.rs/tokio runtime , so you can display per-worker stats in Grafana: The table merges a separate query per column, each grouped by worker : Parks, Polls, Steals sum by worker increase hotpath tokio worker parks total $ range sum by worker increase hotpath tokio worker polls total $ range sum by worker increase hotpath tokio worker steals total $ range Busy, Busy % sum by worker increase hotpath tokio worker busy seconds total $ range sum by worker rate hotpath tokio worker busy seconds total $ range Local queue sum by worker hotpath tokio worker local queue depth Mean poll sum by worker increase hotpath tokio worker busy seconds total $ range / sum by worker increase hotpath tokio worker polls total $ range 0 - Parks - how many times a worker went idle waiting for work. - Busy - time a worker spent executing tasks, and its share of the selected time range. - Polls - how many times a worker polled a task. - Steals - tasks a worker took from other workers’ queues. Frequent stealing can suggest that work is not evenly distributed across workers. - Local queue - tasks waiting in a worker’s queue. A queue that keeps growing may indicate that the worker is falling behind. - Mean poll - busy time divided by polls, approximating how long a single poll takes. Long polls keep the worker from running other tasks. If one worker stays much busier than the others, it may be worth checking for blocking calls or long-running tasks. Some of these metrics require building with RUSTFLAGS="--cfg tokio unstable" . Per-thread CPU usage and memory allocations per-thread-cpu-usage-and-memory-allocations Standard monitoring usually provides process-level CPU and memory usage metrics. hotpath-rs provides more granular insights by letting you observe individual threads. This means you can spot a single thread constantly spinning at 100% while others are sitting idle. Thread monitoring https://hotpath.rs/threads enabling-per-thread-monitoring is enabled by default, while per-thread allocation metrics require the hotpath-alloc feature. The following query shows the peak CPU usage reached by each thread since the process started: sort desc topk 10, hotpath thread cpu percent max And this one, the total bytes allocated by each thread: sort desc topk 10, hotpath thread alloc bytes total Put them side by side to build a similar dashboard: This provides a granular overview of which threads are doing most of the work and can help you spread the load more evenly when bottlenecks surface. Both metrics are labeled with the thread name and OS tid , so give your threads names using std::thread::Builder::name . Otherwise, they will show up as generic thread N . To spot potential memory retention, compare allocated and deallocated bytes. If the difference keeps growing under a steady load, a thread may be holding on to memory: rate hotpath thread alloc bytes total $ rate interval - rate hotpath thread dealloc bytes total $ rate interval Correlating app metrics with system-level signals correlating-app-metrics-with-system-level-signals Host-level monitoring, e.g. node exporter https://github.com/prometheus/node exporter or your cloud provider’s metrics, can tell you that the CPU was maxed out or the network was saturated. It can’t tell you why . That’s where the hotpath-rs Grafana integration really pays off. Its metrics are regular Prometheus series, so you can put them on the same dashboard, next to the CPU and network throughput charts coming from other sources. When the CPU chart spikes, you look at what your routes, SQL queries, and HTTP calls were doing at the same moment. Being able to correlate signals across multiple app layers makes performance debugging a lot simpler. How to minimize hotpath-rs profiling overhead impact? how-to-minimize-hotpath-rs-profiling-overhead-impact Maybe the ideas above caught your attention, but you’re worried that adding such granular profiling will impact the performance of your production application. The good news is that there’s a simple way to verify that the profiling overhead stays negligible. hotpath::measure adds around 40ns per call see profiling overhead docs https://hotpath.rs/profiling overhead , which I’ll round up to 50ns . That’s 0.001% for a 5ms function, but 10% for a 500ns one. Fast functions only matter if they account for a meaningful share of total execution time, so the following query filters on both: sort desc 100 $overhead ns 1e-9 / sum by function histogram sum increase hotpath function duration seconds $ range / sum by function histogram count increase hotpath function duration seconds $ range 0 $threshold and on function 100 sum by function histogram sum increase hotpath function duration seconds $ range / scalar sum histogram sum increase hotpath function duration seconds $ range $min share The first part divides $overhead ns by each function’s average call duration and keeps only functions above $threshold . The second keeps only functions that account for more than $min share of the total instrumented time. All three values are Grafana textbox variables you can tweak directly from the dashboard. For the hotpath-backend binary running this blog, a 1% time share and 1% overhead threshold return no results. I had to lower them to 0.1% and 0.01% respectively to get anything to show up: The worst one is at 0.07% . Without the share filter, functions like report json::futures section reach 13-16% overhead, but each accounts for less than 0.01% of total execution time, so their impact is negligible. If a function does show up above your thresholds, exclude it from profiling. Remove its hotpath::measure attribute, or mark it with \ hotpath::skip\ https://hotpath.rs/functions hotpathskip-macro if it’s covered by measure all . Alternatively, you can configure time sampling https://hotpath.rs/profiling overhead reducing-overhead-time-sampling to reduce the instrumentation overhead. Disclaimer: hotpath-backend currently serves minimal traffic, so these numbers will differ for busier Rust applications. But the technique stays the same, letting you verify that profiling overhead remains negligible even in high-traffic production deployments. Get started with hotpath-rs Grafana using an AI skill get-started-with-hotpath-rs-grafana-using-an-ai-skill While working on this post, I considered providing ready-to-use Grafana dashboard templates. But since we’re in the AI era now, I think a more user-friendly way to share the setup is through an AI skill. I’ve updated the hotpath CLI to include a Grafana init command https://hotpath.rs/prometheus grafana ai-setup : cargo install hotpath --version '^0.28' hotpath init-grafana --agent claude or --agent codex / --agent opencode This command launches your own agent session with a skill https://github.com/pawurb/hotpath-rs/blob/main/skills/hotpath init grafana/SKILL.md containing everything it needs to configure a dedicated hotpath-rs Grafana dashboard for your project. Give it a try, and let me know in the comments if you manage to find any actionable insights. Use hotpath Cloud to spot issues before they reach production use-hotpath-cloud-to-spot-issues-before-they-reach-production Production observability is great, but it’s even better to catch performance issues before they get deployed. hotpath Cloud https://hotpath.rs/cloud lets you observe the same context-aware metrics in your CI benchmarks and put regression constraints https://hotpath.rs/regression policy and performance budgets https://hotpath.rs/performance budgets on them. For example, the N+1 check from the SQL section can become a hard limit enforced on every pull request: hotpath/policy.toml server.budgets match = " " timing = { sql per request = 5 } message = "too many SQL queries per request, check for N+1 calls" or you can ensure that a specific hot function never allocates: functions.budgets match = "my crate::router::route" alloc = { total = "0 B" } message = "route is on the hot path and must not allocate" Each pull request gets a comment summarizing meaningful performance changes, so regressions are caught in review instead of on a Grafana dashboard: You can set it up with your AI agent in a few minutes: cargo install hotpath hotpath init-ci --agent claude or --agent codex / --agent opencode With AI agents writing more and more code, manually reviewing every change for its performance impact no longer scales. hotpath Cloud adds deterministic constraints to your CI pipeline: regression limits and performance budgets that every pull request must pass, regardless of whether it was written by a human or an agent. This ensures your application stays performant even as the pace of changes keeps growing. hotpath Cloud is now in open beta and free to use, so give it a try After beta it will remain free for public repositories. I’ll elaborate on how to use it in the next post, so follow me on X https://x.com/pawelurbanekcom to get notified when it’s out. Summary summary hotpath-rs exposes performance metrics that are difficult to get from traditional observability tooling. These include SQL queries, HTTP calls, and allocations attributed to the exact Axum endpoint, as well as per-channel and per-lock latency. The real value comes from combining these signals with application context, making them easier to reason about and act on. I dogfood hotpath-rs extensively, using it to profile both the hotpath Cloud backend and the profiler itself. I’ll keep refining the production workflows and dashboards as I learn what works best. I invite you to check out hotpath-rs https://github.com/pawurb/hotpath-rs and integrate it into your Grafana setup. If you build a dashboard worth sharing, or if there’s a performance metric you’re missing from the lib, I’d love to hear from you.