In this post I provide an introduction to the System.Diagnostics.Metrics API, show how to use dotnet-counters
for local monitoring of metrics, and show how to add a custom metric to your application.
The System.Diagnostics.Metrics APIs
System.Diagnostics.MetricsAPIs
The System.Diagnostics.Metrics APIs were originally introduced as a built-in in feature in .NET 6, but are also supported in earlier versions of .NET Core and .NET Framework using the System.Diagnostics.DiagnosticSource NuGet package. The metrics APIs provide a way to both create and report on metrics generated by an application, such as simple counters, gauges, or histograms of values. I'll describe each of the available metric types later.
The System.Diagnostics.Metrics APIs are designed to easily interoperate with OpenTelemetry, and so can be consumed by a large range of applications. You can also read the metrics using .NET SDK tools like dotnet-counters
.
The word "metric" is often used in multiple different ways. Is it a single "point" with associated "tags"? Is it the full set of the recorded values for a single concept? Is it the "aggregated" statistics for all of these points? It's common to see both meanings. In this post I mainly use "metric" to mean a stream of recordings for a single concept.
There are two core concepts exposed by the System.Diagnostics.Metrics APIs. These are Instrument
s and Meter
s:
Instrument
: An instrument records the values for a single metric of interest. You might have separateInstrument
s for "products sold", "invoices created", "invoice total", and "GC heap size".Meter
: AMeter
is a logical grouping of multiple instruments. For example, thecontains multipleSystem.Runtime
Meter
Instrument
s about the workings of the runtime, whilethecontainsMicrosoft.AspNetCore.Hosting
Meter
Instrument
s about the HTTP requests received by ASP.NET Core.
There are also several different types of Instrument
:
Counter<T>
/ObservableCounter<T>
: These represent a count of occurrences, so return a non-negative values. For example, the number of requests received might be aCounter<T>
.UpDownCounter<T>
/ObservableUpDownCounter<T>
: These are similar to a counter, but can be used to record both positive and negative values. This may be used to report the change in queue size or the number ofactiverequests.Gauge<T>
/ObservableGauge<T>
: These return a value that represents the "current value". The values it emits effectively "replace" the previous value. For example, the amount of memory used might be aGauge<T>
.Histogram<T>
: Reports arbitrary values, which could be subsequently processed to calculate further statistics, or plot as a graph. For example, the duration of each request might be recorded as aHistogram
.
You'll note that the Counter<T>
, UpDownCounter<T>
, and Gauge<T>
all have observable versions. This difference relates to how the Instrument
records and emits values; observable instruments only retrieve their values when explicitly requested, whereas the non-observable versions emit a value as soon as that value is recorded.
The choice of whether an
Instrument
should be implemented asObservable*
is driven partly by performance considerations, and partly by how the value is obtained. I'll cover more about the implementation differences with observableInstrument
s in a future post.
Collecting metrics with dotnet-counters
dotnet-counters
When running in production, you'll likely want to collect your metrics using an OpenTelemetry exporter integration or another solution (e.g. Datadog can collect these metrics without requiring application changes), but for local testing
dotnet-counters
is a very convenient tool.dotnet-counters
is a .NET tool shipped by Microsoft that you can install by running:
dotnet tool install -g dotnet-counters
You can then run the tool by specifying a process ID or process name to monitor, using:
dotnet-counters monitor -n MyApp
dotnet-counters monitor -p 123
Alternatively, you can specify a command to run when starting the tool, and it will monitor the target process:
dotnet-counters monitor -- dotnet MyApp.dll
When you run dotnet-counters
in this "monitor" mode, the counter values are written to the console and periodically refresh:
Press p to , r to resume, q to quit.
Status: Running
Name Current Value
[System.Runtime]
dotnet.assembly.count ({assembly}) 100
dotnet.gc.collections ({collection})
gc.heap.generation
------------------
gen0 67
gen1 6
gen2 1
dotnet.gc.heap.total_allocated (By) 4,134,656
dotnet.gc.last_collection.heap.fragmentation.size (By)
gc.heap.generation
------------------
gen0 911,896
gen1 5,544
gen2 1,656
loh 0
poh 0
dotnet.gc.last_collection.heap.size (By)
gc.heap.generation
------------------
gen0 943,560
gen1 271,288
gen2 840,136
loh 0
poh 24,528
dotnet.gc.last_collection.memory.committed_size (By) 3,981,312
dotnet.gc..time (s) 0.106
dotnet.jit.compilation.time (s) 1.096
dotnet.jit.compiled_il.size (By) 199,280
dotnet.jit.compiled_methods ({method}) 2,126
dotnet.monitor.lock_contentions ({contention}) 1
dotnet.process.cpu.count ({cpu}) 4
dotnet.process.cpu.time (s)
cpu.mode
--------
system 5.453
user 9.313
dotnet.process.memory.working_set (By) 51,384,320
dotnet.thread_pool.queue.length ({work_item}) 0
dotnet.thread_pool.thread.count ({thread}) 4
dotnet.thread_pool.work_item.count ({work_item}) 61,911
dotnet.timer.count ({timer}) 0
You can choose which Meter
s to display by passing a comma-separated list of counters using --counters
, for example to show the Microsoft.AspNetCore.Hosting
Meter
, you would use:
> dotnet-counters monitor --counters 'Microsoft.AspNetCore.Hosting' -- dotnet MyApp.dll
Press p to , r to resume, q to quit.
Status: Running
Name Current Value
[Microsoft.AspNetCore.Hosting]
http.server.active_requests ({request})
http.request.method url.scheme
------------------- ----------
GET http 0
http.server.request.duration (s)
http.request.method http.response.status_code http.route network.protocol.version url.scheme Percentile
------------------- ------------------------- ---------- ------------------------ ---------- ----------
GET 200 / 1.1 http 50 0
GET 200 / 1.1 http 95 0
GET 200 / 1.1 http 99 0
The metrics in the above image were created by hitting the same endpoint in a sample app several times, but they show some of the different features of the Instrument
s available. Each metric has an associated unit ({request}
and s
), and also an associated set of tags. Tags are an important aspect when recording metrics, as they allow you to more easily group and segregate data.
For example, the http.server.active_requests
up/down counter has tags for http.request.method
and url.scheme
. Seeing as I only made GET
requests to http://localhost:5000
, you only see one set of tags. But if I had made POST
requests, or requests using https
then you would have seen other values there. Similarly, the values in the http.server.request.duration
histogram include tags for each value.
Managing tag
cardinality(the number of possible values) is an important aspect of dealing with tags in all observability data. Depending on how your data is stored, large tag cardinality could cause large data storage costs and an impact on performance. Those limits will generally be controlled by whatever system you're exporting your metrics to.
As well as "immediate" monitoring approaches like the example above, which just outputs to the console, dotnet-counters
also has options for just collecting the metrics and exporting them in a variety of formats. You could drive a production monitoring system this way, but I suspect most usages of dotnet-counters
are for the local testing scenario.
The dotnet-counters
example above demonstrates some of the built-in metrics available in .NET 10. The System.Runtime
meter is available since .NET 9, and the Microsoft.AspNetCore.Routing
meter is available since .NET 8, but there are many other additional built-in metrics available in different versions of .NET. You can find what's available here:
These metrics can provide a reasonable overview of how your system is operating in general, but there might also be application-specific or business related metrics that would be useful to record from the application itself.
As an example, we'll create a very simple counter metric that just records the number of requests sent to a particular API. To make it slightly less abstract, we'll imagine this to be a product pricing endpoint, and we want to track how often the details are checked for a given product.
We'll start by creating the basic app using
dotnet new web
and updating the application to the following:
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/product/{id}", (int id) =>
{
// This would return the real details
// TODO: add metrics
return $"Pricing for product {id}";
});
app.Run();
We would obviously return real pricing details from this API, but this is just a demo after all.
Creating our Instrument
and Meter
Instrument
and Meter
Now let's add our metrics. We need to create two things:
- An
Instrument
to track the number of requests. - A
Meter
to hold our instrument (and any future related instruments).
We need to be careful about the naming of both of these, as they essentially serve as the public API for subsequent consumers of our metrics.
Seeing as this is an ASP.NET Core application and we generally avoid global static
variables, the example below shows how we would create a class to encapsulate our Instrument
and Meter
, so that we can register it with the dependency injection container later. If you were creating an app that doesn't use DI, you could just as easily use new Meter()
, and save the variable in a global variable.
public class ProductMetrics
{
private readonly Counter<long> _pricingDetailsViewed;
public ProductMetrics(IMeterFactory meterFactory)
{
var meter = meterFactory.Create("MyApp.Products");
_pricingDetailsViewed = meter.CreateCounter<long>("myapp.products.pricing_page_requests");
}
public void PricingPageViewed(int id)
{
_pricingDetailsViewed.Add(delta: 1, new KeyValuePair<string, object?>("product_id", id));
}
}
In the code above we:
- Create a new
Meter
calledMyApp.Products
. This is named following similar guidelines to the built-in meters; we have "namespaced" using our app's name, and the broad category of the instruments it will include. - We create a
Counter<long>
calledmyapp.products.pricing_page_requests
. This is named using theOpenTelemetry naming guidelines. I opted forlong
because I anticipate that some pages will get alotof reviews in the lifetime of the app (more thanint.MaxValue
). - We added a convenience method for recording a view of a product's pricing page, tagging the view with the ID of the product we're viewing. We could add other tags&mash;maybe the product name would be more useful for example&smash;but this tag will do for our purposes.
If we want to add additional Instrument
s to the same Meter
later, we would create them here, and likely add similar convenience methods.
Hooking up the Instrument
in the app
Instrument
in the appNow that we have our metrics helper, we need to make use of it in our app. This involves both registering the helper in DI, and using it in our API:
using System.Diagnostics.Metrics;
using Microsoft.Extensions.Diagnostics.Metrics;
var builder = WebApplication.CreateBuilder(args);
// 👇 Register in DI
builder.Services.AddSingleton<ProductMetrics>();
var app = builder.Build();
// Inject in API handler 👇
app.MapGet("/product/{id}", (int id, ProductMetrics metrics) =>
{
metrics.PricingPageViewed(id); // 👈 Record
return $"Details for product {id}";
});
app.Run();
We can now try it out using dotnet-counters
to view the metrics.
We'll start by running our app:
dotnet run
and then in a separate terminal window, we'll set dotnet-counters
running using
dotnet-counters monitor -n MyApp --counters MyApp.Products
I've used the -n
option to find the app by name, MyApp
, and made sure to only show the MyApp.Products
instrument.
If we hit the product endpoint a few times with various IDs, we can see that the metrics are reported to dotnet-counters
as expected!
With that, we have confirmed that we have a custom metric being successfully recorded 🎉
Adding extra information for consumers
In the dotnet-counters
output above, we can see that the Instrument is reported with the unit Count
, inferred from the instrument type. That's fine, but the Instrument
API lets us provide additional details that can be optionally used by consumers to customise the display or metrics.
For example, we could add some additional details to our instrument, as follows:
_pricingDetailsViewed = meter.CreateCounter<int>(
"myapp.products.pricing_page_requests",
unit: "requests",
description: "The number of requests to the pricing details page for the product with the given product_id");
If we run and monitor the app again, the dotnet-counters
output has changed slightly. The unit for myapp.products.pricing_page_requests
has changed to requests
instead of Count
:
Name Current Value
[MyApp.Products]
myapp.products.pricing_page_requests (requests)
product_id
----------
1 1
234 1
5 4
That's a small nicity, and the description isn't used anywhere by dotnet-counters
, but other exporters might choose to use it. Depending on how you're exporting your metrics out of process, your metric should now be available everywhere!
In this post, I provided an introduction to the System.Diagnostics.Metrics APIs. I described some of the terminology used, such as Meter
and Instrument
, and the various different types of Instrument
available. I then showed how you can use dotnet-counters
to monitor the metrics produced by your app, primarily for local investigation. Finally, I showed how you could create a custom metric, customize it, hook it up to dependency injection, and report it in dotnet-counters
.