Creating and consuming metrics with System.Diagnostics.Metrics APIs: System.Diagnostics.Metrics APIs - Part 1

The article introduces the `System.Diagnostics.Metrics` API, which was built into .NET 6 and is also available for earlier .NET versions via a NuGet package. It explains core concepts like `Instrument` and `Meter`, describes various metric types (such as `Counter`, `UpDownCounter`, `Gauge`, and `Histogram`), and demonstrates how to use the `dotnet-counters` tool for local monitoring of these metrics.

In this post I provide an introduction to the System.Diagnostics.Metrics API, show how to use dotnet-counters for local monitoring of metrics, and show how to add a custom metric to your application. The the-system-diagnostics-metrics-apis System.Diagnostics.Metrics APIs System.Diagnostics.Metrics APIs The System.Diagnostics.Metrics APIs were originally introduced as a built-in in feature in .NET 6, but are also supported in earlier versions of .NET Core and .NET Framework using the System.Diagnostics.DiagnosticSource https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/ NuGet package. The metrics APIs provide a way to both create and report on metrics generated by an application, such as simple counters, gauges, or histograms of values. I'll describe each of the available metric types later. The System.Diagnostics.Metrics APIs are designed to easily interoperate with OpenTelemetry, and so can be consumed by a large range of applications. You can also read the metrics using .NET SDK tools like dotnet-counters . The word "metric" is often used in multiple different ways. Is it a single "point" with associated "tags"? Is it the full set of the recorded values for a single concept? Is it the "aggregated" statistics for all of these points? It's common to see both meanings. In this post I mainly use "metric" to mean a stream of recordings for a single concept. There are two core concepts exposed by the System.Diagnostics.Metrics APIs. These are Instrument s and Meter s: Instrument : An instrument records the values for a single metric of interest. You might have separate Instrument s for "products sold", "invoices created", "invoice total", and "GC heap size". Meter : A Meter is a logical grouping of multiple instruments. For example, thecontains multiple System.Runtime Meter Instrument s about the workings of the runtime, while the https://learn.microsoft.com/en-us/aspnet/core/log-mon/metrics/built-in?view=aspnetcore-10.0 microsoftaspnetcorehosting contains Microsoft.AspNetCore.Hosting Meter Instrument s about the HTTP requests received by ASP.NET Core. There are also several different types of Instrument : Counter<T / ObservableCounter<T : These represent a count of occurrences, so return a non-negative values. For example, the number of requests received might be a Counter<T . UpDownCounter<T / ObservableUpDownCounter<T : These are similar to a counter, but can be used to record both positive and negative values. This may be used to report the change in queue size or the number of active requests. Gauge<T / ObservableGauge<T : These return a value that represents the "current value". The values it emits effectively "replace" the previous value. For example, the amount of memory used might be a Gauge<T . Histogram<T : Reports arbitrary values, which could be subsequently processed to calculate further statistics, or plot as a graph. For example, the duration of each request might be recorded as a Histogram . You'll note that the Counter<T , UpDownCounter<T , and Gauge<T all have observable versions. This difference relates to how the Instrument records and emits values; observable instruments only retrieve their values when explicitly requested, whereas the non-observable versions emit a value as soon as that value is recorded. The choice of whether an Instrument should be implemented as Observable is driven partly by performance considerations, and partly by how the value is obtained. I'll cover more about the implementation differences with observable Instrument s in a future post. Collecting metrics with collecting-metrics-with-dotnet-counters dotnet-counters dotnet-counters When running in production, you'll likely want to collect your metrics using an OpenTelemetry exporter integration https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/getting-started-prometheus-grafana collect-metrics-using-prometheus or another solution e.g. Datadog can collect these metrics without requiring application changes , but for local testing dotnet-counters is a very convenient tool. dotnet-counters is a .NET tool shipped by Microsoft that you can install by running: dotnet tool install -g dotnet-counters You can then run the tool by specifying a process ID or process name to monitor, using: dotnet-counters monitor -n MyApp or dotnet-counters monitor -p 123 Alternatively, you can specify a command to run when starting the tool, and it will monitor the target process: dotnet-counters monitor -- dotnet MyApp.dll When you run dotnet-counters in this "monitor" mode, the counter values are written to the console and periodically refresh: Press p to pause, r to resume, q to quit. Status: Running Name Current Value System.Runtime dotnet.assembly.count {assembly} 100 dotnet.gc.collections {collection} gc.heap.generation ------------------ gen0 67 gen1 6 gen2 1 dotnet.gc.heap.total allocated By 4,134,656 dotnet.gc.last collection.heap.fragmentation.size By gc.heap.generation ------------------ gen0 911,896 gen1 5,544 gen2 1,656 loh 0 poh 0 dotnet.gc.last collection.heap.size By gc.heap.generation ------------------ gen0 943,560 gen1 271,288 gen2 840,136 loh 0 poh 24,528 dotnet.gc.last collection.memory.committed size By 3,981,312 dotnet.gc.pause.time s 0.106 dotnet.jit.compilation.time s 1.096 dotnet.jit.compiled il.size By 199,280 dotnet.jit.compiled methods {method} 2,126 dotnet.monitor.lock contentions {contention} 1 dotnet.process.cpu.count {cpu} 4 dotnet.process.cpu.time s cpu.mode -------- system 5.453 user 9.313 dotnet.process.memory.working set By 51,384,320 dotnet.thread pool.queue.length {work item} 0 dotnet.thread pool.thread.count {thread} 4 dotnet.thread pool.work item.count {work item} 61,911 dotnet.timer.count {timer} 0 You can choose which Meter s to display by passing a comma-separated list of counters using --counters , for example to show the Microsoft.AspNetCore.Hosting Meter , you would use: dotnet-counters monitor --counters 'Microsoft.AspNetCore.Hosting' -- dotnet MyApp.dll Press p to pause, r to resume, q to quit. Status: Running Name Current Value Microsoft.AspNetCore.Hosting http.server.active requests {request} http.request.method url.scheme ------------------- ---------- GET http 0 http.server.request.duration s http.request.method http.response.status code http.route network.protocol.version url.scheme Percentile ------------------- ------------------------- ---------- ------------------------ ---------- ---------- GET 200 / 1.1 http 50 0 GET 200 / 1.1 http 95 0 GET 200 / 1.1 http 99 0 The metrics in the above image were created by hitting the same endpoint in a sample app several times, but they show some of the different features of the Instrument s available. Each metric has an associated unit {request} and s , and also an associated set of tags . Tags are an important aspect when recording metrics, as they allow you to more easily group and segregate data. For example, the http.server.active requests up/down counter has tags for http.request.method and url.scheme . Seeing as I only made GET requests to http://localhost:5000 , you only see one set of tags. But if I had made POST requests, or requests using https then you would have seen other values there. Similarly, the values in the http.server.request.duration histogram include tags for each value. Managing tag cardinality the number of possible values is an important aspect of dealing with tags in all observability data. Depending on how your data is stored, large tag cardinality could cause large data storage costs and an impact on performance. Those limits will generally be controlled by whatever system you're exporting your metrics to. As well as "immediate" monitoring approaches like the example above, which just outputs to the console, dotnet-counters also has options for just collecting the metrics and exporting them in a variety of formats https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters dotnet-counters-collYouect . You could drive a production monitoring system this way, but I suspect most usages of dotnet-counters are for the local testing scenario. Creating your own metrics creating-your-own-metrics The dotnet-counters example above demonstrates some of the built-in metrics available in .NET 10. The System.Runtime meter is available since .NET 9, and the Microsoft.AspNetCore.Routing meter is available since .NET 8, but there are many other additional built-in metrics available in different versions of .NET. You can find what's available here: These metrics can provide a reasonable overview of how your system is operating in general, but there might also be application-specific or business related metrics that would be useful to record from the application itself. As an example, we'll create a very simple counter metric that just records the number of requests sent to a particular API. To make it slightly less abstract, we'll imagine this to be a product pricing endpoint, and we want to track how often the details are checked for a given product. Creating the initial app creating-the-initial-app We'll start by creating the basic app using dotnet new web and updating the application to the following: js var builder = WebApplication.CreateBuilder args ; var app = builder.Build ; app.MapGet "/product/{id}", int id = { // This would return the real details // TODO: add metrics return $"Pricing for product {id}"; } ; app.Run ; We would obviously return real pricing details from this API, but this is just a demo after all. Creating our creating-our-instrument-and-meter Instrument and Meter Instrument and Meter Now let's add our metrics. We need to create two things: - An Instrument to track the number of requests. - A Meter to hold our instrument and any future related instruments . We need to be careful about the naming of both of these, as they essentially serve as the public API for subsequent consumers of our metrics. Seeing as this is an ASP.NET Core application and we generally avoid global static variables, the example below shows how we would create a class to encapsulate our Instrument and Meter , so that we can register it with the dependency injection container later. If you were creating an app that doesn't use DI, you could just as easily use new Meter , and save the variable in a global variable. public class ProductMetrics { private readonly Counter<long pricingDetailsViewed; public ProductMetrics IMeterFactory meterFactory { var meter = meterFactory.Create "MyApp.Products" ; pricingDetailsViewed = meter.CreateCounter<long "myapp.products.pricing page requests" ; } public void PricingPageViewed int id { pricingDetailsViewed.Add delta: 1, new KeyValuePair<string, object? "product id", id ; } } In the code above we: - Create a new Meter called MyApp.Products . This is named following similar guidelines to the built-in meters; we have "namespaced" using our app's name, and the broad category of the instruments it will include. - We create a Counter<long called myapp.products.pricing page requests . This is named using the OpenTelemetry naming guidelines https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/metrics.md general-guidelines . I opted for long because I anticipate that some pages will get a lot of reviews in the lifetime of the app more than int.MaxValue . - We added a convenience method for recording a view of a product's pricing page, tagging the view with the ID of the product we're viewing. We could add other tags&mash;maybe the product name would be more useful for example&smash;but this tag will do for our purposes. If we want to add additional Instrument s to the same Meter later, we would create them here, and likely add similar convenience methods. Hooking up the hooking-up-the-instrument-in-the-app Instrument in the app Instrument in the appNow that we have our metrics helper, we need to make use of it in our app. This involves both registering the helper in DI, and using it in our API: using System.Diagnostics.Metrics; using Microsoft.Extensions.Diagnostics.Metrics; var builder = WebApplication.CreateBuilder args ; // 👇 Register in DI builder.Services.AddSingleton<ProductMetrics ; var app = builder.Build ; // Inject in API handler 👇 app.MapGet "/product/{id}", int id, ProductMetrics metrics = { metrics.PricingPageViewed id ; // 👈 Record return $"Details for product {id}"; } ; app.Run ; We can now try it out using dotnet-counters to view the metrics. Testing our new metric testing-our-new-metric We'll start by running our app: dotnet run and then in a separate terminal window, we'll set dotnet-counters running using dotnet-counters monitor -n MyApp --counters MyApp.Products I've used the -n option to find the app by name, MyApp , and made sure to only show the MyApp.Products instrument. If we hit the product endpoint a few times with various IDs, we can see that the metrics are reported to dotnet-counters as expected With that, we have confirmed that we have a custom metric being successfully recorded 🎉 Adding extra information for consumers adding-extra-information-for-consumers In the dotnet-counters output above, we can see that the Instrument is reported with the unit Count , inferred from the instrument type. That's fine, but the Instrument API lets us provide additional details that can be optionally used by consumers to customise the display or metrics. For example, we could add some additional details to our instrument, as follows: pricingDetailsViewed = meter.CreateCounter<int "myapp.products.pricing page requests", unit: "requests", description: "The number of requests to the pricing details page for the product with the given product id" ; If we run and monitor the app again, the dotnet-counters output has changed slightly. The unit for myapp.products.pricing page requests has changed to requests instead of Count : Name Current Value MyApp.Products myapp.products.pricing page requests requests product id ---------- 1 1 234 1 5 4 That's a small nicity, and the description isn't used anywhere by dotnet-counters , but other exporters might choose to use it. Depending on how you're exporting your metrics out of process, your metric should now be available everywhere Summary summary In this post, I provided an introduction to the System.Diagnostics.Metrics APIs. I described some of the terminology used, such as Meter and Instrument , and the various different types of Instrument available. I then showed how you can use dotnet-counters to monitor the metrics produced by your app, primarily for local investigation. Finally, I showed how you could create a custom metric, customize it, hook it up to dependency injection, and report it in dotnet-counters .