{"slug": "creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3", "title": "Creating standard and \"observable\" instruments: System.Diagnostics.Metrics APIs - Part 3", "summary": "This article explains the `System.Diagnostics.Metrics` APIs in .NET, focusing on the distinction between \"observable\" and \"normal\" instruments. For normal instruments, the application (producer) actively emits metric values as events occur, whereas for observable instruments, the consumer (like `dotnet-counters`) polls for the value on demand, which is more efficient for continuous or expensive-to-track metrics. The post also provides a refresher on the seven instrument types (e.g., `Counter<T>`, `ObservableGauge<T>`) and how to create them.", "body_md": "In the [first post in this series](/creating-and-consuming-metrics-with-system-diagnostics-metrics-apis/) I provided an introduction to the *System.Diagnostics.Metrics* APIs introduced in .NET 6. I initially introduced the concept of \"observable\" `Instrument`\n\ns in that post, but didn't go into more details. In this post, we'll understand what being \"observable\" means, and how these `Instrument`\n\ns differ from non-observable `Instrument`\n\ns.\n\nI start the post with a quick refresher on the basics of the *System.Diagnostics.Metrics* APIs, such as the different types of instruments available. I then show how you can create each of the instrument types and produce values from them.\n\n[System.Diagnostics.Metrics APIs](#system-diagnostics-metrics-apis)\n\nThe *System.Diagnostics.Metrics* APIs were introduced in .NET 6 but are available in earlier runtimes (including .NET Framework) by using the [ System.Diagnostics.DiagnosticSource](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/) NuGet package. There are two primary concepts exposed by these APIs:\n\n`Instrument`\n\nand `Meter`\n\n:`Instrument`\n\n: An instrument records the values for a single metric of interest. You might have separate`Instrument`\n\ns for \"products sold\", \"invoices created\", \"invoice total\", or \"GC heap size\".`Meter`\n\n: A`Meter`\n\nis a logical grouping of multiple instruments. For example, thecontains multiple`System.Runtime`\n\n`Meter`\n\n`Instrument`\n\ns about the workings of the runtime, while[the](https://learn.microsoft.com/en-us/aspnet/core/log-mon/metrics/built-in?view=aspnetcore-10.0#microsoftaspnetcorehosting)contains`Microsoft.AspNetCore.Hosting`\n\n`Meter`\n\n`Instrument`\n\ns about the HTTP requests received by ASP.NET Core.\n\nThere are also (currently, as of .NET 10) 7 different types of `Instrument`\n\n:\n\n`Counter<T>`\n\n`ObservableCounter<T>`\n\n`UpDownCounter<T>`\n\n`ObservableUpDownCounter<T>`\n\n`Gauge<T>`\n\n`ObservableGauge<T>`\n\n`Histogram<T>`\n\n.\n\nTo create a custom metric, you need to choose the type of `Instrument`\n\nto use, and associate it with a `Meter`\n\n. I'll discuss the differences between each of these instruments shortly, but first we'll look at the difference between \"observable\" instruments, and \"normal\" instruments.\n\n[What is an ](#what-is-an-observable-instrument-)`Observable*`\n\ninstrument?\n\n`Observable*`\n\ninstrument?When using the *System.Diagnostic.Metrics* APIs there's a \"producer\" side and a \"consumer\" side. The producer of metrics is the app itself, recording values and details about how it's operating. The consumer could be an in-process consumer, such as [the OpenTelemetry libraries](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel), or it could be an external process, such as [ dotnet-counters](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters) or\n\n[.](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-monitor)\n\n`dotnet-monitor`\n\nThe differences between a \"normal\" instrument and an \"observable\" instrument stem from who controls when and how a value is emitted:\n\n- For \"normal\" instruments, the\n*producer*emits values as they occur. For example, when a request is received, ASP.NET Core emits the`http.server.active_requests`\n\nmetric, indicating a new request is in-flight. - For \"observable\" instruments, the\n*consumer*side*asks*for the value. For example, the`dotnet.gc.pause.time`\n\nmetric returns \"The total amount of time paused in GC since the process has started\", but only when you*ask*for it.\n\nIn general, observable instruments are used when you have an effectively continuous value that you wouldn't make sense for the consumer to actively emit, such as the `dotnet.gc.pause.time`\n\nabove, or where emitting all of the intermediate values would be too expensive from a performance point of view.\n\nTechnically, you\n\ncouldpotentially emit this metric every time the GC pauses, but given that these values are more fine-grained than you would likely wantanyway, it's much more efficient to allow the consumer to \"poll\" the values on demand, and therefore it makes the most sense as an observable instrument.\n\nNow we understand the difference between observable and normal instruments, let's walk through all the instrumentation types and see how they're used in the .NET base class libraries.\n\n[Understanding the different ](#understanding-the-different-instrument-types)`Instrument`\n\ntypes\n\n`Instrument`\n\ntypesSo far in this series we've used a simple `Counter<T>`\n\nthat records every time a given event occurs. In this post we'll look at each of the possible `Instrument`\n\ns in turn, showing how you create an instrument of that type to produce a given metric. Where possible, I'm showing places within the .NET or ASP.NET Core libraries that use each of these instruments, to give \"real world\" versions of how these are used.\n\n`Counter<T>`\n\n`Counter<T>`\n\nThe `Counter<T>`\n\ninstrument is one of the simplest instruments conceptually. It is used to record how many times a given event occurs.\n\nFor example, [the aspnetcore.diagnostics.exceptions metric](https://github.com/dotnet/aspnetcore/blob/102119ab7ceb911130fad4a485ec0a4828aa9e53/src/Middleware/Diagnostics/src/DiagnosticsMetrics.cs#L24-L27) is a\n\n`Counter<long>`\n\nwhich records the `\"Number of exceptions caught by exception handling middleware.\"`\n\n```\n_handlerExceptionCounter = _meter.CreateCounter<long>(\n    \"aspnetcore.diagnostics.exceptions\",\n    unit: \"{exception}\",\n    description: \"Number of exceptions caught by exception handling middleware.\");\n```\n\nEvery time the `ExceptionHandlerMiddleware`\n\n(or `DeveloperExceptionHandlerMiddleware`\n\n) [catches an exception](https://github.com/dotnet/aspnetcore/blob/102119ab7ceb911130fad4a485ec0a4828aa9e53/src/Middleware/Diagnostics/src/ExceptionHandler/ExceptionHandlerMiddlewareImpl.cs#L126), it adds `1`\n\nto this counter, first constructing an appropriate set of tags, and then calling `Add(1, tags)`\n\n:\n\n```\n private void RequestExceptionCore(string exceptionName, ExceptionResult result, string? handler)\n{\n    var tags = new TagList();\n    tags.Add(\"error.type\", exceptionName);\n    tags.Add(\"aspnetcore.diagnostics.exception.result\", GetExceptionResult(result));\n    if (handler != null)\n    {\n        tags.Add(\"aspnetcore.diagnostics.handler.type\", handler);\n    }\n    _handlerExceptionCounter.Add(1, tags);\n}\n```\n\nAs this `Counter<T>`\n\nis tracking a number of occurrences, you're always adding positive values, never negative values, though you can increase by more than `1`\n\nat a time if needs be.\n\n`ObservableCounter<T>`\n\n`ObservableCounter<T>`\n\nThe `ObservableCounter<T>`\n\nis conceptually similar to a `Counter<T>`\n\n, in that it records monotonically increasing values. Being an \"observable\" instrument, it only records the values when \"observed\" (we'll look at how to observe the instruments in your own code in a subsequent post).\n\nFor example, [the dotnet.gc.heap.total_allocated metric](https://github.com/dotnet/runtime/blob/v10.0.1/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/RuntimeMetrics.cs#L44-L48) is an\n\n`ObservableCounter<long>`\n\nwhich records the `\"The approximate number of bytes allocated on the managed GC heap since the process has started\"`\n\n:\n\n``` js\ns_meter.CreateObservableCounter(\n    \"dotnet.gc.heap.total_allocated\",\n    () => GC.GetTotalAllocatedBytes(),\n    unit: \"By\",\n    description: \"The approximate number of bytes allocated on the managed GC heap since the process has started. The returned value does not include any native allocations.\");\n```\n\nWhen observed, the lambda included in the definition is called, which invokes `GC.GetTotalAllocatedBytes()`\n\n. Note that this value steadily increases during the lifetime of the app, so it's not returning the difference since *last* invocation, it's returning the current running total.\n\n`UpDownCounter<T>`\n\n`UpDownCounter<T>`\n\nThe `UpDownCounter<T>`\n\nis similar to the `Counter<T>`\n\n, but it supports reporting positive or negative values.\n\nFor example, [the http.server.active_requests metric](https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs#L24-L27) is an\n\n`UpDownCounter<T>`\n\nthat records the `\"Number of active HTTP server requests.\"`\n\n:\n\n```\n_activeRequestsCounter = _meter.CreateUpDownCounter<long>(\n    \"http.server.active_requests\",\n    unit: \"{request}\",\n    description: \"Number of active HTTP server requests.\");\n```\n\n[When a request is started](https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs#L37), the server calls `Add()`\n\nand increments the value of the counter:\n\n```\npublic void RequestStart(string scheme, string method)\n{\n    // Tags must match request end.\n    var tags = new TagList();\n    InitializeRequestTags(ref tags, scheme, method);\n    _activeRequestsCounter.Add(1, tags);\n}\n\nprivate static void InitializeRequestTags(ref TagList tags, string scheme, string method)\n{\n    tags.Add(HostingTelemetryHelpers.AttributeUrlScheme, scheme);\n    tags.Add(HostingTelemetryHelpers.AttributeHttpRequestMethod, HostingTelemetryHelpers.GetNormalizedHttpMethod(method));\n}\n```\n\nSimilarly, [when the request ends](https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs#L45C1-L54C10), the server calls `Add()`\n\nto *decrement* the value of the counter:\n\n```\npublic void RequestEnd(string protocol, string scheme, string method, string? route, int statusCode, bool unhandledRequest, Exception? exception, List<KeyValuePair<string, object?>>? customTags, long startTimestamp, long currentTimestamp, bool disableHttpRequestDurationMetric)\n{\n    var tags = new TagList();\n    InitializeRequestTags(ref tags, scheme, method);\n\n    // Tags must match request start.\n    if (_activeRequestsCounter.Enabled)\n    {\n        _activeRequestsCounter.Add(-1, tags);\n    }\n\n    // ...\n}\n```\n\nConsequently, the `UpDownCounter<T>`\n\nreceives a series of increment/decrement values representing the movement of the metric.\n\n`ObservableUpDownCounter<T>`\n\n`ObservableUpDownCounter<T>`\n\nThe `ObservableUpDownCounter<T>`\n\nis similar to the `UpDownCounter<T>`\n\nin that it reports increasing or decreasing values of a metric. The difference is that it returns the absolute value of the metric when observed, as opposed to a stream of deltas.\n\nFor example, the `dotnet.gc.last_collection.heap.size`\n\nmetric is an `ObservableUpDownCounter<long>`\n\nthat reports `\"The managed GC heap size (including fragmentation), as observed during the latest garbage collection\"`\n\n:\n\n```\ns_meter.CreateObservableUpDownCounter(\n    \"dotnet.gc.last_collection.heap.size\",\n    GetHeapSizes,\n    unit: \"By\",\n    description: \"The managed GC heap size (including fragmentation), as observed during the latest garbage collection.\");\n```\n\nWhen observed, the `GetHeapSizes()`\n\nmethod is invoked and returns a collection of `Measurement`\n\ns, each tagged by the heap generation name:\n\n```\nprivate static readonly string[] s_genNames = [\"gen0\", \"gen1\", \"gen2\", \"loh\", \"poh\"];\nprivate static readonly int s_maxGenerations = Math.Min(GC.GetGCMemoryInfo().GenerationInfo.Length, s_genNames.Length);\n\nprivate static IEnumerable<Measurement<long>> GetHeapSizes()\n{\n    GCMemoryInfo gcInfo = GC.GetGCMemoryInfo();\n\n    for (int i = 0; i < s_maxGenerations; ++i)\n    {\n        yield return new Measurement<long>(gcInfo.GenerationInfo[i].SizeAfterBytes, new KeyValuePair<string, object?>(\"gc.heap.generation\", s_genNames[i]));\n    }\n}\n```\n\nThis returns the size of each heap at the last GC collection, the value of which may obviously increase or decrease.\n\n`Gauge<T>`\n\n`Gauge<T>`\n\nThe `Gauge<T>`\n\nis used to record \"non-additive\" values whenever they occur. These values can go up and down, and be positive or negative, but the point is that they \"overwrite\" all previous values.\n\nInterestingly, this `Instrument`\n\ntype was only added in .NET 9, and I couldn't find a single case of `Gauge<T>`\n\nbeing used in the .NET runtime, ASP.NET Core, or the .NET extensions packages 😅 So I made one up: for example, consider a gauge that reports the current room temperature when it changes:\n\n``` js\nvar instrument = _meter.CreateGauge<double>(\n    name: \"locations.room.temperature\",\n    unit: \"°C\",\n    description: \"Current room temperature\"\n);\n```\n\nThen when the temperature of the room changes, you would report the new value:\n\n```\npublic void OnOfficeTemperatureChanged(double newTemperature)\n{\n    instrument.Record(newTemperature, new KeyValuePair<string, object?>(\"room\", \"office\"));\n}\n```\n\nThe gauge values are record whenever the temperature changes.\n\n`ObservableGauge<T>`\n\n`ObservableGauge<T>`\n\nConceptually the `ObservableGauge<T>`\n\nis the same as a `Gauge<T>`\n\n, except that it only produces a value when observed. `ObservableGauge<T>`\n\nwas added way back in .NET 6, and there are some examples of its use in this case.\n\nFor example, [the process.cpu.utilization metric](https://github.com/dotnet/extensions/blob/9974fbf7a3fede68d7e5f22b9b249aebd819a26d/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Windows/WindowsSnapshotProvider.cs#L94) is an\n\n`ObservableGauge<double>`\n\ninstrument which reports `\"The CPU consumption of the running application in range [0, 1]\"`\n\n.\n\n```\n_ = meter.CreateObservableGauge(\n    name: \"process.cpu.utilization\",\n    observeValue: CpuPercentage);\n```\n\nWhen observed, [the CpuPercentage() method](https://github.com/dotnet/extensions/blob/9974fbf7a3fede68d7e5f22b9b249aebd819a26d/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Windows/WindowsSnapshotProvider.cs#L168) is invoked, which returns a single value for the CPU usage as a value between\n\n`0`\n\nand `1`\n\n.\n\n```\nprivate double CpuPercentage()\n{\n    // see above link for implementation\n}\n```\n\nThis `Instrument`\n\nis exposed in the `Microsoft.Extensions.Diagnostics.ResourceMonitoring`\n\nmeter, and implemented in [the Microsoft.Extensions.Diagnostics.ResourceMonitoring NuGet package](https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.ResourceMonitoring).\n\n`Histogram<T>`\n\n`Histogram<T>`\n\nThe final instrument type is `Histogram<T>`\n\n, which is used to report arbitrary values, that you will typically want to aggregate using statistics.\n\nFor example, [the http.server.request.duration metric](https://github.com/dotnet/aspnetcore/blob/9a93048bd0afd7c2d09bdb5ce47ef7d78827c647/src/Hosting/Hosting/src/Internal/HostingMetrics.cs#L29) is a\n\n`Histogram<double>`\n\nwhich records the `\"Duration of HTTP server requests.\"`\n\n. Durations and latencies are a classic example of where you might want to use a histogram, so that you can calculate the p50, p90, p99 etc latencies, or to record *all*the values and plot them as a graph.\n\n```\n_requestDuration = _meter.CreateHistogram<double>(\n    \"http.server.request.duration\",\n    unit: \"s\",\n    description: \"Duration of HTTP server requests.\",\n    advice: new InstrumentAdvice<double> { HistogramBucketBoundaries = MetricsConstants.ShortSecondsBucketBoundaries });\n```\n\nThe example above also shows our first example of\n\n`InstrumentAdvice<T>`\n\n. This type provides suggested configuration settings for consumers, indicating the best settings to use when processing`Instrument`\n\nvalues. In this case, the advice provides a suggested set of histogram bucket boundaries:`[0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10]`\n\n, which can be useful for consumers to know how best to plot the metric values.\n\nThe `_requestDuration`\n\nhistogram instrument is called [whenever an ASP.NET Core request ends](https://github.com/dotnet/aspnetcore/blob/9ea8e2c28c695650c619a89b1edf2d6d6a75da67/src/Hosting/Hosting/src/Internal/HostingMetrics.cs#L45), recording the duration of the request, and a large associated number of tags. I've reproduced all the code below for completeness (expanding tag constants for clarity) but it's basically just building up a collection of tags which are recorded along with the duration of the request.\n\n```\npublic void RequestEnd(string protocol, string scheme, string method, string? route, int statusCode, bool unhandledRequest, Exception? exception, List<KeyValuePair<string, object?>>? customTags, long startTimestamp, long currentTimestamp, bool disableHttpRequestDurationMetric)\n{\n    var tags = new TagList();\n    InitializeRequestTags(ref tags, scheme, method);\n\n    if (!disableHttpRequestDurationMetric && _requestDuration.Enabled)\n    {\n        if (HostingTelemetryHelpers.TryGetHttpVersion(protocol, out var httpVersion))\n        {\n            tags.Add(\"network.protocol.version\", httpVersion);\n        }\n        if (unhandledRequest)\n        {\n            tags.Add(\"aspnetcore.request.is_unhandled\", true);\n        }\n\n        // Add information gathered during request.\n        tags.Add(\"http.response.status_code\", HostingTelemetryHelpers.GetBoxedStatusCode(statusCode));\n        if (route != null)\n        {\n            tags.Add(\"http.route\", RouteDiagnosticsHelpers.ResolveHttpRoute(route));\n        }\n\n        // Add before some built in tags so custom tags are prioritized when dealing with duplicates.\n        if (customTags != null)\n        {\n            for (var i = 0; i < customTags.Count; i++)\n            {\n                tags.Add(customTags[i]);\n            }\n        }\n\n        // This exception is only present if there is an unhandled exception.\n        // An exception caught by ExceptionHandlerMiddleware and DeveloperExceptionMiddleware isn't thrown to here. Instead, those middleware add error.type to custom tags.\n        if (exception != null)\n        {\n            // Exception tag could have been added by middleware. If an exception is later thrown in request pipeline\n            // then we don't want to add a duplicate tag here because that breaks some metrics systems.\n            tags.TryAddTag(\"error.type\", exception.GetType().FullName);\n        }\n        else if (HostingTelemetryHelpers.IsErrorStatusCode(statusCode))\n        {\n            // Add error.type for 5xx status codes when there's no exception.\n            tags.TryAddTag(\"error.type\", statusCode.ToString(CultureInfo.InvariantCulture));\n        }\n\n        var duration = Stopwatch.GetElapsedTime(startTimestamp, currentTimestamp);\n        _requestDuration.Record(duration.TotalSeconds, tags);\n    }\n}\n```\n\nIt's an interesting point to note that while the histogram is strictly about request durations, the presence of the many tags could enable you to derive various other metrics. For example, you could determine the number of \"successful\" requests, the number of requests to a particular route, or with a given status code.\n\nAnd that's it, we've covered all of the `Insturment`\n\ntypes currently available in .NET 10. Note that there's no `ObservableHistogram<T>`\n\ntype, as that generally wouldn't be practical to implement.\n\nWe now know how to create all the different types of `Instrument`\n\n, and in the [first post of this series](/creating-and-consuming-metrics-with-system-diagnostics-metrics-apis/) I showed how to record the metrics using `dotnet-counters`\n\n. In the following post in this series, we'll look at how to record these values in-process instead.\n\n[Summary](#summary)\n\nIn this post, I described each of the different `Instrument<T>`\n\ntypes exposed by the *System.Diagnostics.Metrics* APIs. For each type I described when you would use it and provided an example of both how to create the `Instrument<T>`\n\n, and how to record values, using examples from the .NET base class libraries and ASP.NET Core. In the next post we'll look at how to record values produced by `Instrument<T>`\n\ntypes in-process.", "url": "https://wpnews.pro/news/creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3", "canonical_source": "https://andrewlock.net/creating-standard-and-observable-instruments/", "published_at": "2026-02-17 10:00:00+00:00", "updated_at": "2026-05-23 21:38:29.288090+00:00", "lang": "en", "topics": ["developer-tools"], "entities": [".NET", "System.Diagnostics.Metrics", "ASP.NET Core", "NuGet"], "alternates": {"html": "https://wpnews.pro/news/creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3", "markdown": "https://wpnews.pro/news/creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3.md", "text": "https://wpnews.pro/news/creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3.txt", "jsonld": "https://wpnews.pro/news/creating-standard-and-observable-instruments-system-diagnostics-metrics-apis-3.jsonld"}}