Best practices

Follow these best practices to get the most out of OpenTelemetry .NET for metrics.

Package version

Use the System.Diagnostics.Metrics APIs from the latest stable version of System.Diagnostics.DiagnosticSource package, regardless of the .NET runtime version being used:

• If you are using the latest stable version of OpenTelemetry .NET SDK, you do not have to worry about the version of System.Diagnostics.DiagnosticSource package because it is already taken care of for you via package dependency.
• The .NET runtime team is holding a high bar for backward compatibility on System.Diagnostics.DiagnosticSource even during major version bumps, so compatibility is not a concern here.
• Refer to the .NET official document for more information about System.Diagnostics.Metrics.

Metrics API

Meter

Avoid creating System.Diagnostics.Metrics.Meter too frequently. Meter is fairly expensive and meant to be reused throughout the application. For most applications, it can be modeled as static readonly field or singleton through dependency injection.

Use dot-separated UpperCamelCase as the Meter.Name. In many cases, using the fully qualified class name might be a good option. For example:

static readonly Meter MyMeter = new("MyCompany.MyProduct.MyLibrary", "1.0");

Instruments

Understand and pick the right instrument type.

Avoid creating instruments (for example, Counter<T>) too frequently. Instruments are fairly expensive and meant to be reused throughout the application. For most applications, instruments can be modeled as static readonly fields or singleton through dependency injection.

Avoid invalid instrument names.

Avoid changing the order of tags while reporting measurements. For example:

counter.Add(2, new("name", "apple"), new("color", "red"));
counter.Add(3, new("name", "lime"), new("color", "green"));
counter.Add(5, new("name", "lemon"), new("color", "yellow"));
counter.Add(8, new("color", "yellow"), new("name", "lemon")); // bad perf

Use TagList properly to achieve the best performance. There are two different ways of passing tags to an instrument API:

• Pass the tags directly to the instrument API:

  counter.Add(100, new("Key1", "Value1"), new("Key2", "Value2"));
  

• Use TagList:

  var tags = new TagList
  {
      { "DimName1", "DimValue1" },
      { "DimName2", "DimValue2" },
      { "DimName3", "DimValue3" },
      { "DimName4", "DimValue4" },
  };
  
  counter.Add(100, tags);
  

As a general rule:

• When reporting measurements with 3 tags or less, pass the tags directly to the instrument API.
• When reporting measurements with 4 to 8 tags (inclusive), use TagList to avoid heap allocation if avoiding GC pressure is a primary performance goal. For high performance code which consider reducing CPU utilization more important (e.g. to reduce latency, to save battery, etc.) than optimizing memory allocations, use profiler and stress test to determine which approach is better.
• When reporting measurements with more than 8 tags, the two approaches share very similar CPU performance and heap allocation. TagList is recommended due to its better readability and maintainability.

MeterProvider management

Avoid creating MeterProvider instances too frequently. MeterProvider is fairly expensive and meant to be reused throughout the application. For most applications, one MeterProvider instance per process would be sufficient. For example:

graph LR

subgraph Meter A
  InstrumentX
end

subgraph Meter B
  InstrumentY
  InstrumentZ
end

subgraph Meter Provider 2
  MetricReader2
  MetricExporter2
  MetricReader3
  MetricExporter3
end

subgraph Meter Provider 1
  MetricReader1
  MetricExporter1
end

InstrumentX --> | Measurements | MetricReader1
InstrumentY --> | Measurements | MetricReader1 --> MetricExporter1
InstrumentZ --> | Measurements | MetricReader2 --> MetricExporter2
InstrumentZ --> | Measurements | MetricReader3 --> MetricExporter3

Manage the lifecycle of MeterProvider instances if they are created by you.

As a general rule:

• If you are building an application with dependency injection (DI) (e.g. ASP.NET Core and .NET Worker), in most cases you should create the MeterProvider instance and let DI manage its lifecycle. Refer to the Getting Started with OpenTelemetry .NET Metrics in 5 Minutes - ASP.NET Core Application tutorial to learn more.
• If you are building an application without DI, create a MeterProvider instance and manage the lifecycle explicitly. Refer to the Getting Started with OpenTelemetry .NET Metrics in 5 Minutes - Console Application tutorial to learn more.
• If you forget to dispose the MeterProvider instance before the application ends, metrics might get dropped due to the lack of proper flush.
• If you dispose the MeterProvider instance too early, any subsequent measurements will not be collected.

Memory management

In OpenTelemetry, measurements are reported via the metrics API. The SDK aggregates metrics using certain algorithms and memory management strategies to achieve good performance and efficiency. Here are the rules which OpenTelemetry .NET follows while implementing the metrics aggregation logic:

1. Pre-Aggregation: aggregation occurs within the SDK.
2. Cardinality Limits: the aggregation logic respects cardinality limits, so the SDK does not use indefinite amount of memory when there is cardinality explosion.
3. Memory Preallocation: the memory used by aggregation logic is allocated during the SDK initialization, so the SDK does not have to allocate memory on-the-fly. This is to avoid garbage collection being triggered on the hot code path.

Example

Let us take the following example:

• During the time range (T0, T1]:
  • value = 1, name = apple, color = red
  • value = 2, name = lemon, color = yellow
• During the time range (T1, T2]:
  • no fruit has been received
• During the time range (T2, T3]:
  • value = 5, name = apple, color = red
  • value = 2, name = apple, color = green
  • value = 4, name = lemon, color = yellow
  • value = 2, name = lemon, color = yellow
  • value = 1, name = lemon, color = yellow
  • value = 3, name = lemon, color = yellow

If we aggregate and export the metrics using Cumulative Aggregation Temporality:

• (T0, T1]
  • attributes: {name = apple, color = red}, count: 1
  • attributes: {verb = lemon, color = yellow}, count: 2
• (T0, T2]
  • attributes: {name = apple, color = red}, count: 1
  • attributes: {verb = lemon, color = yellow}, count: 2
• (T0, T3]
  • attributes: {name = apple, color = red}, count: 6
  • attributes: {name = apple, color = green}, count: 2
  • attributes: {verb = lemon, color = yellow}, count: 12

If we aggregate and export the metrics using Delta Aggregation Temporality:

• (T0, T1]
  • attributes: {name = apple, color = red}, count: 1
  • attributes: {verb = lemon, color = yellow}, count: 2
• (T1, T2]
  • nothing since we do not have any measurement received
• (T2, T3]
  • attributes: {name = apple, color = red}, count: 5
  • attributes: {name = apple, color = green}, count: 2
  • attributes: {verb = lemon, color = yellow}, count: 10

Pre-aggregation

Taking the fruit example, there are 6 measurements reported during (T2, T3]. Instead of exporting every individual measurement event, the SDK aggregates them and only exports the summarized results. This approach, as illustrated in the following diagram, is called pre-aggregation:

graph LR

subgraph SDK
  Instrument --> | Measurements | Pre-Aggregation[Pre-Aggregation]
end

subgraph Collector
  Aggregation
end

Pre-Aggregation --> | Metrics | Aggregation

Pre-aggregation brings several benefits:

1. Although the amount of calculation remains the same, the amount of data transmitted can be significantly reduced using pre-aggregation, thus improving the overall efficiency.
2. Pre-aggregation makes it possible to apply cardinality limits during SDK initialization, combined with memory preallocation, they make the metrics data collection behavior more predictable (e.g. a server under denial-of-service attack would still produce a constant volume of metrics data, rather than flooding the observability system with large volume of measurement events).

There are cases where users might want to export raw measurement events instead of using pre-aggregation, as illustrated in the following diagram. OpenTelemetry does not support this scenario at the moment, if you are interested, please join the discussion by replying to this feature ask.

graph LR

subgraph SDK
  Instrument
end

subgraph Collector
  Aggregation
end

Instrument --> | Measurements | Aggregation

Cardinality limits

The number of unique combinations of attributes is called cardinality. Taking the fruit example, if we know that we can only have apple/lemon as the name, red/yellow/green as the color, then we can say the cardinality is 6. No matter how many apples and lemons we have, we can always use the following table to summarize the total number of fruits based on the name and color.

In other words, we know how much storage and network are needed to collect and transmit these metrics, regardless of the traffic pattern.

In real world applications, the cardinality can be extremely high. Imagine if we have a long running service and we collect metrics with 7 attributes and each attribute can have 30 different values. We might eventually end up having to remember the complete set of all 21,870,000,000 combinations! This cardinality explosion is a well-known challenge in the metrics space. For example, it can cause surprisingly high costs in the observability system, or even be leveraged by hackers to launch a denial-of-service attack.

Cardinality limit is a throttling mechanism which allows the metrics collection system to have a predictable and reliable behavior when excessive cardinality happens, whether it was due to a malicious attack or developer making mistakes while writing code.

OpenTelemetry has a default cardinality limit of 2000 per metric. This limit can be configured at the individual metric level using the View API and the MetricStreamConfiguration.CardinalityLimit setting.

As of 1.10.0 once a metric has reached the cardinality limit, any new measurement that could not be independently aggregated will be automatically aggregated using the overflow attribute.

As of 1.10.0 when Delta Aggregation Temporality is used, it is possible to choose a smaller cardinality limit because the SDK will reclaim unused metric points.

Memory preallocation

OpenTelemetry .NET SDK aims to avoid memory allocation on the hot code path. When this is combined with proper use of Metrics API, heap allocation can be avoided on the hot code path.

You should measure memory allocation on hot code path, and ideally avoid any heap allocation while using the metrics API and SDK, especially when you use metrics to measure the performance of your application (for example, you do not want to spend 2 seconds doing garbage collection while measuring an operation which normally takes 10 milliseconds).

Metrics correlation

In OpenTelemetry, metrics can be correlated to traces via exemplars. Check the Exemplars tutorial to learn more.

Metrics enrichment

When metrics are being collected, they normally get stored in a time series database. From storage and consumption perspective, metrics can be multi-dimensional. Taking the fruit example, there are two dimensions - “name” and “color”. For basic scenarios, all the dimensions can be reported during the Metrics API invocation, however, for less trivial scenarios, the dimensions can come from different sources:

• Measurements reported via the Metrics API.
• Additional tags provided at instrument creation time. For example, the Meter.CreateCounter<T>(name, unit, description, tags) overload.
• Additional tags provided at meter creation time. For example, the Meter(name, version, tags, scope) overload.
• Resources configured at the MeterProvider level.
• Additional attributes provided by the exporter or collector. For example, jobs and instances in Prometheus.

As a general rule:

• If the dimension is static throughout the process lifetime (e.g. the name of the machine, data center):
  • If the dimension applies to all metrics, model it as Resource, or even better, let the collector add these dimensions if feasible (e.g. a collector running in the same data center should know the name of the data center, rather than relying on / trusting each service instance to report the data center name).
  • If the dimension applies to a subset of metrics (e.g. the version of a client library), model it as meter level tags.
• If the dimension value is dynamic, report it via the Metrics API.

Common issues that lead to missing metrics

• The Meter used to create the instruments is not added to the MeterProvider. Use AddMeter method to enable the processing for the required metrics.