Creating standard and "observable" instruments: System.Diagnostics.Metrics APIs - Part 3

This article explains the `System.Diagnostics.Metrics` APIs in .NET, focusing on the distinction between "observable" and "normal" instruments. For normal instruments, the application (producer) actively emits metric values as events occur, whereas for observable instruments, the consumer (like `dotnet-counters`) polls for the value on demand, which is more efficient for continuous or expensive-to-track metrics. The post also provides a refresher on the seven instrument types (e.g., `Counter<T>`, `ObservableGauge<T>`) and how to create them.

In the first post in this series /creating-and-consuming-metrics-with-system-diagnostics-metrics-apis/ I provided an introduction to the System.Diagnostics.Metrics APIs introduced in .NET 6. I initially introduced the concept of "observable" Instrument s in that post, but didn't go into more details. In this post, we'll understand what being "observable" means, and how these Instrument s differ from non-observable Instrument s. I start the post with a quick refresher on the basics of the System.Diagnostics.Metrics APIs, such as the different types of instruments available. I then show how you can create each of the instrument types and produce values from them. System.Diagnostics.Metrics APIs system-diagnostics-metrics-apis The System.Diagnostics.Metrics APIs were introduced in .NET 6 but are available in earlier runtimes including .NET Framework by using the System.Diagnostics.DiagnosticSource https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/ NuGet package. There are two primary concepts exposed by these APIs: Instrument and Meter : Instrument : An instrument records the values for a single metric of interest. You might have separate Instrument s for "products sold", "invoices created", "invoice total", or "GC heap size". Meter : A Meter is a logical grouping of multiple instruments. For example, thecontains multiple System.Runtime Meter Instrument s about the workings of the runtime, while the https://learn.microsoft.com/en-us/aspnet/core/log-mon/metrics/built-in?view=aspnetcore-10.0 microsoftaspnetcorehosting contains Microsoft.AspNetCore.Hosting Meter Instrument s about the HTTP requests received by ASP.NET Core. There are also currently, as of .NET 10 7 different types of Instrument : Counter<T ObservableCounter<T UpDownCounter<T ObservableUpDownCounter<T Gauge<T ObservableGauge<T Histogram<T . To create a custom metric, you need to choose the type of Instrument to use, and associate it with a Meter . I'll discuss the differences between each of these instruments shortly, but first we'll look at the difference between "observable" instruments, and "normal" instruments. What is an what-is-an-observable-instrument- Observable instrument? Observable instrument?When using the System.Diagnostic.Metrics APIs there's a "producer" side and a "consumer" side. The producer of metrics is the app itself, recording values and details about how it's operating. The consumer could be an in-process consumer, such as the OpenTelemetry libraries https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel , or it could be an external process, such as dotnet-counters https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters or . https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-monitor dotnet-monitor The differences between a "normal" instrument and an "observable" instrument stem from who controls when and how a value is emitted: - For "normal" instruments, the producer emits values as they occur. For example, when a request is received, ASP.NET Core emits the http.server.active requests metric, indicating a new request is in-flight. - For "observable" instruments, the consumer side asks for the value. For example, the dotnet.gc.pause.time metric returns "The total amount of time paused in GC since the process has started", but only when you ask for it. In general, observable instruments are used when you have an effectively continuous value that you wouldn't make sense for the consumer to actively emit, such as the dotnet.gc.pause.time above, or where emitting all of the intermediate values would be too expensive from a performance point of view. Technically, you couldpotentially emit this metric every time the GC pauses, but given that these values are more fine-grained than you would likely wantanyway, it's much more efficient to allow the consumer to "poll" the values on demand, and therefore it makes the most sense as an observable instrument. Now we understand the difference between observable and normal instruments, let's walk through all the instrumentation types and see how they're used in the .NET base class libraries. Understanding the different understanding-the-different-instrument-types Instrument types Instrument typesSo far in this series we've used a simple Counter<T that records every time a given event occurs. In this post we'll look at each of the possible Instrument s in turn, showing how you create an instrument of that type to produce a given metric. Where possible, I'm showing places within the .NET or ASP.NET Core libraries that use each of these instruments, to give "real world" versions of how these are used. Counter<T Counter<T The Counter<T instrument is one of the simplest instruments conceptually. It is used to record how many times a given event occurs. For example, the aspnetcore.diagnostics.exceptions metric https://github.com/dotnet/aspnetcore/blob/102119ab7ceb911130fad4a485ec0a4828aa9e53/src/Middleware/Diagnostics/src/DiagnosticsMetrics.cs L24-L27 is a Counter<long which records the "Number of exceptions caught by exception handling middleware." handlerExceptionCounter = meter.CreateCounter<long "aspnetcore.diagnostics.exceptions", unit: "{exception}", description: "Number of exceptions caught by exception handling middleware." ; Every time the ExceptionHandlerMiddleware or DeveloperExceptionHandlerMiddleware catches an exception https://github.com/dotnet/aspnetcore/blob/102119ab7ceb911130fad4a485ec0a4828aa9e53/src/Middleware/Diagnostics/src/ExceptionHandler/ExceptionHandlerMiddlewareImpl.cs L126 , it adds 1 to this counter, first constructing an appropriate set of tags, and then calling Add 1, tags : private void RequestExceptionCore string exceptionName, ExceptionResult result, string? handler { var tags = new TagList ; tags.Add "error.type", exceptionName ; tags.Add "aspnetcore.diagnostics.exception.result", GetExceptionResult result ; if handler = null { tags.Add "aspnetcore.diagnostics.handler.type", handler ; } handlerExceptionCounter.Add 1, tags ; } As this Counter<T is tracking a number of occurrences, you're always adding positive values, never negative values, though you can increase by more than 1 at a time if needs be. ObservableCounter<T ObservableCounter<T The ObservableCounter<T is conceptually similar to a Counter<T , in that it records monotonically increasing values. Being an "observable" instrument, it only records the values when "observed" we'll look at how to observe the instruments in your own code in a subsequent post . For example, the dotnet.gc.heap.total allocated metric https://github.com/dotnet/runtime/blob/v10.0.1/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/RuntimeMetrics.cs L44-L48 is an ObservableCounter<long which records the "The approximate number of bytes allocated on the managed GC heap since the process has started" : js s meter.CreateObservableCounter "dotnet.gc.heap.total allocated", = GC.GetTotalAllocatedBytes , unit: "By", description: "The approximate number of bytes allocated on the managed GC heap since the process has started. The returned value does not include any native allocations." ; When observed, the lambda included in the definition is called, which invokes GC.GetTotalAllocatedBytes . Note that this value steadily increases during the lifetime of the app, so it's not returning the difference since last invocation, it's returning the current running total. UpDownCounter<T UpDownCounter<T The UpDownCounter<T is similar to the Counter<T , but it supports reporting positive or negative values. For example, the http.server.active requests metric https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs L24-L27 is an UpDownCounter<T that records the "Number of active HTTP server requests." : activeRequestsCounter = meter.CreateUpDownCounter<long "http.server.active requests", unit: "{request}", description: "Number of active HTTP server requests." ; When a request is started https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs L37 , the server calls Add and increments the value of the counter: public void RequestStart string scheme, string method { // Tags must match request end. var tags = new TagList ; InitializeRequestTags ref tags, scheme, method ; activeRequestsCounter.Add 1, tags ; } private static void InitializeRequestTags ref TagList tags, string scheme, string method { tags.Add HostingTelemetryHelpers.AttributeUrlScheme, scheme ; tags.Add HostingTelemetryHelpers.AttributeHttpRequestMethod, HostingTelemetryHelpers.GetNormalizedHttpMethod method ; } Similarly, when the request ends https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs L45C1-L54C10 , the server calls Add to decrement the value of the counter: public void RequestEnd string protocol, string scheme, string method, string? route, int statusCode, bool unhandledRequest, Exception? exception, List<KeyValuePair<string, object? ? customTags, long startTimestamp, long currentTimestamp, bool disableHttpRequestDurationMetric { var tags = new TagList ; InitializeRequestTags ref tags, scheme, method ; // Tags must match request start. if activeRequestsCounter.Enabled { activeRequestsCounter.Add -1, tags ; } // ... } Consequently, the UpDownCounter<T receives a series of increment/decrement values representing the movement of the metric. ObservableUpDownCounter<T ObservableUpDownCounter<T The ObservableUpDownCounter<T is similar to the UpDownCounter<T in that it reports increasing or decreasing values of a metric. The difference is that it returns the absolute value of the metric when observed, as opposed to a stream of deltas. For example, the dotnet.gc.last collection.heap.size metric is an ObservableUpDownCounter<long that reports "The managed GC heap size including fragmentation , as observed during the latest garbage collection" : s meter.CreateObservableUpDownCounter "dotnet.gc.last collection.heap.size", GetHeapSizes, unit: "By", description: "The managed GC heap size including fragmentation , as observed during the latest garbage collection." ; When observed, the GetHeapSizes method is invoked and returns a collection of Measurement s, each tagged by the heap generation name: private static readonly string s genNames = "gen0", "gen1", "gen2", "loh", "poh" ; private static readonly int s maxGenerations = Math.Min GC.GetGCMemoryInfo .GenerationInfo.Length, s genNames.Length ; private static IEnumerable<Measurement<long GetHeapSizes { GCMemoryInfo gcInfo = GC.GetGCMemoryInfo ; for int i = 0; i < s maxGenerations; ++i { yield return new Measurement<long gcInfo.GenerationInfo i .SizeAfterBytes, new KeyValuePair<string, object? "gc.heap.generation", s genNames i ; } } This returns the size of each heap at the last GC collection, the value of which may obviously increase or decrease. Gauge<T Gauge<T The Gauge<T is used to record "non-additive" values whenever they occur. These values can go up and down, and be positive or negative, but the point is that they "overwrite" all previous values. Interestingly, this Instrument type was only added in .NET 9, and I couldn't find a single case of Gauge<T being used in the .NET runtime, ASP.NET Core, or the .NET extensions packages 😅 So I made one up: for example, consider a gauge that reports the current room temperature when it changes: js var instrument = meter.CreateGauge<double name: "locations.room.temperature", unit: "°C", description: "Current room temperature" ; Then when the temperature of the room changes, you would report the new value: public void OnOfficeTemperatureChanged double newTemperature { instrument.Record newTemperature, new KeyValuePair<string, object? "room", "office" ; } The gauge values are record whenever the temperature changes. ObservableGauge<T ObservableGauge<T Conceptually the ObservableGauge<T is the same as a Gauge<T , except that it only produces a value when observed. ObservableGauge<T was added way back in .NET 6, and there are some examples of its use in this case. For example, the process.cpu.utilization metric https://github.com/dotnet/extensions/blob/9974fbf7a3fede68d7e5f22b9b249aebd819a26d/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Windows/WindowsSnapshotProvider.cs L94 is an ObservableGauge<double instrument which reports "The CPU consumption of the running application in range 0, 1 " . = meter.CreateObservableGauge name: "process.cpu.utilization", observeValue: CpuPercentage ; When observed, the CpuPercentage method https://github.com/dotnet/extensions/blob/9974fbf7a3fede68d7e5f22b9b249aebd819a26d/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Windows/WindowsSnapshotProvider.cs L168 is invoked, which returns a single value for the CPU usage as a value between 0 and 1 . private double CpuPercentage { // see above link for implementation } This Instrument is exposed in the Microsoft.Extensions.Diagnostics.ResourceMonitoring meter, and implemented in the Microsoft.Extensions.Diagnostics.ResourceMonitoring NuGet package https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.ResourceMonitoring . Histogram<T Histogram<T The final instrument type is Histogram<T , which is used to report arbitrary values, that you will typically want to aggregate using statistics. For example, the http.server.request.duration metric https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs L29 is a Histogram<double which records the "Duration of HTTP server requests." . Durations and latencies are a classic example of where you might want to use a histogram, so that you can calculate the p50, p90, p99 etc latencies, or to record all the values and plot them as a graph. requestDuration = meter.CreateHistogram<double "http.server.request.duration", unit: "s", description: "Duration of HTTP server requests.", advice: new InstrumentAdvice<double { HistogramBucketBoundaries = MetricsConstants.ShortSecondsBucketBoundaries } ; The example above also shows our first example of InstrumentAdvice<T . This type provides suggested configuration settings for consumers, indicating the best settings to use when processing Instrument values. In this case, the advice provides a suggested set of histogram bucket boundaries: 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 , which can be useful for consumers to know how best to plot the metric values. The requestDuration histogram instrument is called whenever an ASP.NET Core request ends https://github.com/dotnet/aspnetcore/blob/9ea8e2c28c695650c619a89b1edf2d6d6a75da67/src/Hosting/Hosting/src/Internal/HostingMetrics.cs L45 , recording the duration of the request, and a large associated number of tags. I've reproduced all the code below for completeness expanding tag constants for clarity but it's basically just building up a collection of tags which are recorded along with the duration of the request. public void RequestEnd string protocol, string scheme, string method, string? route, int statusCode, bool unhandledRequest, Exception? exception, List<KeyValuePair<string, object? ? customTags, long startTimestamp, long currentTimestamp, bool disableHttpRequestDurationMetric { var tags = new TagList ; InitializeRequestTags ref tags, scheme, method ; if disableHttpRequestDurationMetric && requestDuration.Enabled { if HostingTelemetryHelpers.TryGetHttpVersion protocol, out var httpVersion { tags.Add "network.protocol.version", httpVersion ; } if unhandledRequest { tags.Add "aspnetcore.request.is unhandled", true ; } // Add information gathered during request. tags.Add "http.response.status code", HostingTelemetryHelpers.GetBoxedStatusCode statusCode ; if route = null { tags.Add "http.route", RouteDiagnosticsHelpers.ResolveHttpRoute route ; } // Add before some built in tags so custom tags are prioritized when dealing with duplicates. if customTags = null { for var i = 0; i < customTags.Count; i++ { tags.Add customTags i ; } } // This exception is only present if there is an unhandled exception. // An exception caught by ExceptionHandlerMiddleware and DeveloperExceptionMiddleware isn't thrown to here. Instead, those middleware add error.type to custom tags. if exception = null { // Exception tag could have been added by middleware. If an exception is later thrown in request pipeline // then we don't want to add a duplicate tag here because that breaks some metrics systems. tags.TryAddTag "error.type", exception.GetType .FullName ; } else if HostingTelemetryHelpers.IsErrorStatusCode statusCode { // Add error.type for 5xx status codes when there's no exception. tags.TryAddTag "error.type", statusCode.ToString CultureInfo.InvariantCulture ; } var duration = Stopwatch.GetElapsedTime startTimestamp, currentTimestamp ; requestDuration.Record duration.TotalSeconds, tags ; } } It's an interesting point to note that while the histogram is strictly about request durations, the presence of the many tags could enable you to derive various other metrics. For example, you could determine the number of "successful" requests, the number of requests to a particular route, or with a given status code. And that's it, we've covered all of the Insturment types currently available in .NET 10. Note that there's no ObservableHistogram<T type, as that generally wouldn't be practical to implement. We now know how to create all the different types of Instrument , and in the first post of this series /creating-and-consuming-metrics-with-system-diagnostics-metrics-apis/ I showed how to record the metrics using dotnet-counters . In the following post in this series, we'll look at how to record these values in-process instead. Summary summary In this post, I described each of the different Instrument<T types exposed by the System.Diagnostics.Metrics APIs. For each type I described when you would use it and provided an example of both how to create the Instrument<T , and how to record values, using examples from the .NET base class libraries and ASP.NET Core. In the next post we'll look at how to record values produced by Instrument<T types in-process.