Dual-exporting .NET metrics with OTLP and Prometheus

Many applications export their metrics directly to Prometheus. If you’re unfamiliar with Prometheus, in a nutshell it’s a time-series database for storing metrics, like counters and histograms. Applications that store their metrics in Prometheus typically use a popular Prometheus client as part of the integration.

Now that OpenTelemetry is a graduated CNCF project, many companies are now increasingly looking to move to OpenTelemetry to add more signals beyond metrics to their observability architecture. Logs and traces are popular additions for getting further insight into how applications behave. Profiles are also starting to become a popular fourth telemetry signal for even deeper understanding.

This can create a migration hurdle - how can we migrate our applications from one system to another for metrics without having a single cut-over event? To de-risk any migration an incremental approach would be preferred, where metrics are exported to both systems for a period of time so that “before” and “after” states can be compared and checked to ensure there is no loss of production visibility in either system for observing metrics or driving alerting.

Using the OpenTelemetry Prometheus exporter for .NET

The latest release of the OpenTelemetry Prometheus exporter for .NET allows you to take this exact approach with your production metrics. You can use the .NET Meter class from your application and framework code to collect metrics and export them to both Prometheus and another exporter, such as the OTLP exporter, provided by the OpenTelemetry.Exporter.OpenTelemetryProtocol NuGet package.

The Prometheus exporter is effectively a Prometheus client library written on top of the OpenTelemetry SDK, exposing an HTTP scrape endpoint in your application to allow a Prometheus server to collect metrics from your application at regular intervals. The exporter implements the OpenTelemetry Prometheus specification (matrix) and implements all of the documented text exposition formats (except OpenMetrics 2.0, which is still experimental) to allow for compatibility with OpenMetrics.

flowchart LR
    subgraph APP["Application"]
        AC["Application code"]
        SDK["OpenTelemetry SDK"]
        PE["Prometheus exporter"]
        OE["OTLP exporter (Client)"]
        EP["GET /metrics HTTP endpoint (Server)"]

        AC -->|"Generates metrics"| SDK
        SDK -->|"Feeds metrics"| PE
        PE -->|"Serves metrics as text/plain"| EP
        SDK -->|"Feeds metrics"| OE
    end

    P["Prometheus (Client)"]
    OTB["OpenTelemetry Backend (Server)"]

    P -->|"HTTP GET /metrics (scrape request)"| EP
    EP -->|"Metrics response (text format)"| P

    OE -->|"OTLP export request"| OTB
    OTB -->|"OTLP response/ack"| OE

By using only the Meter class alongside the Counter<T>, Gauge<T> and Histogram<T> instruments in your .NET application code metrics can be collected without needing to use both the .NET OpenTelemetry SDK and a dedicated Prometheus client.

public class BlogPostComments
{
    private readonly Meter _meter;
    private readonly Counter<long> _likes;

    public BlogPostComments(IMeterFactory meterFactory)
    {
        _meter = meterFactory.Create("OpenTelemetry.Blog");
        _likes = _meter.CreateCounter<long>("blog_post_likes");
    }

    public void BlogPostLiked(long id) =>
        _likes.Add(1, new KeyValuePair<string, object?>("post_id", id));
}

It’s then a small amount of code to configure the OpenTelemetry SDK to export your metrics to both Prometheus and over OTLP to a backend that supports OpenTelemetry by adding the OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package to your project.

using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;

using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .SetResourceBuilder(CreateResourceBuilder())
    .AddMeter("OpenTelemetry.Blog")
    .AddOtlpExporter()
    .AddPrometheusExporter()
    .Build();

Your application will also need to expose the HTTP scrape endpoint that Prometheus will use to collect metrics from your application. This can be done by adding the UseOpenTelemetryPrometheusScrapingEndpoint extension method to your IApplicationBuilder in the Configure method of your Startup class.

For example:

var builder = WebApplication.CreateBuilder(args);

// Configure services here

var app = builder.Build();

// Configure other middleware here

app.MapPrometheusScrapingEndpoint();

app.Run();

Using the Meter APIs to export metrics makes your application code more portable and uncoupled from Prometheus specific APIs. This allows you to remove any Prometheus client library dependencies from your application code. As well as making your code ready for use with the OpenTelemetry ecosystem, it also opens up the ability for you to use other .NET ecosystem tooling such as the dotnet-counters tool to view metrics.

If your application only uses a native Prometheus client such as prometheus-net today then you will need to gradually migrate to using the Meter APIs first. How long this migration will take will depend on the complexity of your existing Prometheus instrumentation and the resources available to you to make the appropriate changes.

Some challenges you may encounter during this migration may include the following Prometheus features which do not have direct equivalents in the Meter APIs, and are therefore not supported:

• the Prometheus summary data type;
• native histograms.

Pushing metrics to Prometheus using OTLP

Alternatively if you only have a Prometheus server and no OTLP compatible backend and only want to export metrics, Prometheus itself has opt-in support for ingesting metrics pushed to it over OTLP.

First ensure that you run Prometheus with the --web.enable-otlp-receiver command line flag.

Then configure the OTLP exporter similarly to the code snippet above, but in this case you wouldn’t need to use the Prometheus exporter as well. Also note that the OTLP exporter specifies a base path for the metrics OTLP endpoint and uses HTTP/protobuf as the protocol for the OTLP exporter.

using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;

using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .SetResourceBuilder(CreateResourceBuilder())
    .AddMeter("OpenTelemetry.Blog")
    .AddOtlpExporter((options, _) =>
    {
        options.Endpoint = new Uri("http://prometheus:9090/api/v1/otlp/v1/metrics");
        options.Protocol = OtlpExportProtocol.HttpProtobuf;
    })
    .Build();

This approach allows you to push metrics to Prometheus with the OpenTelemetry .NET SDK over OTLP without depending on a Prometheus client library in your application code.

You can find a complete example for this approach in the Getting Started with Prometheus and Grafana sample in the OpenTelemetry .NET repository.

Summary

With minimal runtime overhead, the application can both push OTLP metrics and have Prometheus metrics pulled, allowing for both systems to be used in parallel until such time that you decide to go all-in with an OpenTelemetry-compatible backend for your metrics.

• ←Previous
• Next→