Instrumentation

Instrumentation for OpenTelemetry .NET

Instrumentation is the act of adding observability code to an app yourself.

If you’re instrumenting an app, you need to use the OpenTelemetry SDK for your language. You’ll then use the SDK to initialize OpenTelemetry and the API to instrument your code. This will emit telemetry from your app, and any library you installed that also comes with instrumentation.

If you’re instrumenting a library, only install the OpenTelemetry API package for your language. Your library will not emit telemetry on its own. It will only emit telemetry when it is part of an app that uses the OpenTelemetry SDK. For more on instrumenting libraries, see Libraries.

For more information about the OpenTelemetry API and SDK, see the specification.

A note on terminology

.NET is different from other languages/runtimes that support OpenTelemetry. The Tracing API is implemented by the System.Diagnostics API, repurposing existing constructs like ActivitySource and Activity to be OpenTelemetry-compliant under the covers.

However, there are parts of the OpenTelemetry API and terminology that .NET developers must still know to be able to instrument their applications, which are covered here as well as the System.Diagnostics API.

If you prefer to use OpenTelemetry APIs instead of System.Diagnostics APIs, you can refer to the OpenTelemetry API Shim docs for tracing.

Example app preparation

This page uses a modified version of the example app from Getting Started to help you learn about manual instrumentation.

You don’t have to use the example app: if you want to instrument your own app or library, follow the instructions here to adapt the process to your own code.

Prerequisites

Create and launch an HTTP Server

To begin, set up an environment in a new directory called dotnet-otel-example. Within that directory, execute following command:

dotnet new web

To highlight the difference between instrumenting a library and a standalone app, split out the dice rolling into a library file, which then will be imported as a dependency by the app file.

Create the library file named Dice.cs and add the following code to it:

/*Dice.cs*/

public class Dice
{
    private int min;
    private int max;

    public Dice(int min, int max)
    {
        this.min = min;
        this.max = max;
    }

    public List<int> rollTheDice(int rolls)
    {
        List<int> results = new List<int>();

        for (int i = 0; i < rolls; i++)
        {
            results.Add(rollOnce());
        }

        return results;
    }

    private int rollOnce()
    {
        return Random.Shared.Next(min, max + 1);
    }
}

Create the app file DiceController.cs and add the following code to it:

/*DiceController.cs*/

using Microsoft.AspNetCore.Mvc;
using System.Net;


public class DiceController : ControllerBase
{
    private ILogger<DiceController> logger;

    public DiceController(ILogger<DiceController> logger)
    {
        this.logger = logger;
    }

    [HttpGet("/rolldice")]
    public List<int> RollDice(string player, int? rolls)
    {
        if(!rolls.HasValue)
        {
            logger.LogError("Missing rolls parameter");
            throw new HttpRequestException("Missing rolls parameter", null, HttpStatusCode.BadRequest);
        }

        var result = new Dice(1, 6).rollTheDice(rolls.Value);

        if (string.IsNullOrEmpty(player))
        {
            logger.LogInformation("Anonymous player is rolling the dice: {result}", result);
        }
        else
        {
            logger.LogInformation("{player} is rolling the dice: {result}", player, result);
        }

        return result;
    }
}

Replace the content of the Program.cs file with the following code:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();

var app = builder.Build();

app.MapControllers();

app.Run();

In the Properties subdirectory, replace the content of launchSettings.json with the following:

{
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "profiles": {
    "http": {
      "commandName": "Project",
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "applicationUrl": "http://localhost:8080",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    }
  }
}

To ensure that it is working, run the application with the following command and open http://localhost:8080/rolldice?rolls=12 in your web browser:

dotnet run

You should get a list of 12 numbers in your browser window, for example:

[5,6,5,3,6,1,2,5,4,4,2,4]

Manual instrumentation setup

Dependencies

Install the following OpenTelemetry NuGet packages:

OpenTelemetry.Exporter.Console

OpenTelemetry.Extensions.Hosting

dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Extensions.Hosting

For ASP.NET Core-based applications, also install the AspNetCore instrumentation package

OpenTelemetry.Instrumentation.AspNetCore

dotnet add package OpenTelemetry.Instrumentation.AspNetCore

Initialize the SDK

It is important to configure an instance of the OpenTelemetry SDK as early as possible in your application.

To initialize the OpenTelemetry SDK for an ASP.NET Core app like in the case of the example app, update the content of the Program.cs file with the following code:

using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

// Ideally, you will want this name to come from a config file, constants file, etc.
var serviceName = "dice-server";
var serviceVersion = "1.0.0";

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(
        serviceName: serviceName,
        serviceVersion: serviceVersion))
    .WithTracing(tracing => tracing
        .AddSource(serviceName)
        .AddAspNetCoreInstrumentation()
        .AddConsoleExporter())
    .WithMetrics(metrics => metrics
        .AddMeter(serviceName)
        .AddConsoleExporter());

builder.Logging.AddOpenTelemetry(options => options
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(
        serviceName: serviceName,
        serviceVersion: serviceVersion))
    .AddConsoleExporter());

builder.Services.AddControllers();

var app = builder.Build();

app.MapControllers();

app.Run();

If initializing the OpenTelemetry SDK for a console app, add the following code at the beginning of your program, during any important startup operations.

using OpenTelemetry.Logs;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

//...

var serviceName = "MyServiceName";
var serviceVersion = "1.0.0";

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource(serviceName)
    .ConfigureResource(resource =>
        resource.AddService(
          serviceName: serviceName,
          serviceVersion: serviceVersion))
    .AddConsoleExporter()
    .Build();

var meterProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter(serviceName)
    .AddConsoleExporter()
    .Build();

var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.AddOpenTelemetry(logging =>
    {
        logging.AddConsoleExporter();
    });
});

//...

tracerProvider.Dispose();
meterProvider.Dispose();
loggerFactory.Dispose();

For debugging and local development purposes, the example exports telemetry to the console. After you have finished setting up manual instrumentation, you need to configure an appropriate exporter to export the app’s telemetry data to one or more telemetry backends.

The example also sets up the mandatory SDK default attribute service.name, which holds the logical name of the service, and the optional, but highly encouraged, attribute service.version, which holds the version of the service API or implementation. Alternative methods exist for setting up resource attributes. For more information, see Resources.

To verify your code, build and run the app:

dotnet build
dotnet run

Traces

Initialize Tracing

To enable tracing in your app, you’ll need to have an initialized TracerProvider that will let you create a Tracer.

If a TracerProvider is not created, the OpenTelemetry APIs for tracing will use a no-op implementation and fail to generate data.

If you followed the instructions to initialize the SDK above, you have a TracerProvider setup for you already. You can continue with setting up an ActivitySource.

Setting up an ActivitySource

Anywhere in your application where you write manual tracing code should configure an ActivitySource, which will be how you trace operations with Activity elements.

It’s generally recommended to define ActivitySource once per app/service that is been instrumented, but you can instantiate several ActivitySources if that suits your scenario.

In the case of the example app, we will create a new file Instrumentation.cs as a custom type to hold reference for the ActivitySource.

using System.Diagnostics;

/// <summary>
/// It is recommended to use a custom type to hold references for ActivitySource.
/// This avoids possible type collisions with other components in the DI container.
/// </summary>
public class Instrumentation : IDisposable
{
    internal const string ActivitySourceName = "dice-server";
    internal const string ActivitySourceVersion = "1.0.0";

    public Instrumentation()
    {
        this.ActivitySource = new ActivitySource(ActivitySourceName, ActivitySourceVersion);
    }

    public ActivitySource ActivitySource { get; }

    public void Dispose()
    {
        this.ActivitySource.Dispose();
    }
}

Then we will update the Program.cs to add the Instrument object as a dependency injection:

//...

// Register the Instrumentation class as a singleton in the DI container.
builder.Services.AddSingleton<Instrumentation>();

builder.Services.AddControllers();

var app = builder.Build();

app.MapControllers();

app.Run();

In the application file DiceController.cs we will reference that activitySource instance and the same activitySource instance will also be passed to the library file Dice.cs

/*DiceController.cs*/

using Microsoft.AspNetCore.Mvc;
using System.Diagnostics;
using System.Net;

public class DiceController : ControllerBase
{
    private ILogger<DiceController> logger;

    private ActivitySource activitySource;

    public DiceController(ILogger<DiceController> logger, Instrumentation instrumentation)
    {
        this.logger = logger;
        this.activitySource = instrumentation.ActivitySource;
    }

    [HttpGet("/rolldice")]
    public List<int> RollDice(string player, int? rolls)
    {
        List<int> result = new List<int>();

        if (!rolls.HasValue)
        {
            logger.LogError("Missing rolls parameter");
            throw new HttpRequestException("Missing rolls parameter", null, HttpStatusCode.BadRequest);
        }

        result = new Dice(1, 6, activitySource).rollTheDice(rolls.Value);

        if (string.IsNullOrEmpty(player))
        {
            logger.LogInformation("Anonymous player is rolling the dice: {result}", result);
        }
        else
        {
            logger.LogInformation("{player} is rolling the dice: {result}", player, result);
        }

        return result;
    }
}

/*Dice.cs*/

using System.Diagnostics;

public class Dice
{
    public ActivitySource activitySource;
    private int min;
    private int max;

    public Dice(int min, int max, ActivitySource activitySource)
    {
        this.min = min;
        this.max = max;
        this.activitySource = activitySource;
    }

    //...
}

Create Activities

Now that you have activitySources initialized, you can create activities.

The code below illustrates how to create an activity.

public List<int> rollTheDice(int rolls)
{
    List<int> results = new List<int>();

    // It is recommended to create activities, only when doing operations that are worth measuring independently.
    // Too many activities makes it harder to visualize in tools like Jaeger.
    using (var myActivity = activitySource.StartActivity("rollTheDice"))
    {
        for (int i = 0; i < rolls; i++)
        {
            results.Add(rollOnce());
        }

        return results;
    }
}

If you followed the instructions using the example app up to this point, you can copy the code above in your library file Dice.cs. You should now be able to see activities/spans emitted from your app.

Start your app as follows, and then send it requests by visiting http://localhost:8080/rolldice?rolls=12 with your browser or curl.

dotnet run

After a while, you should see the spans printed in the console by the ConsoleExporter, something like this:

Activity.TraceId:            841d70616c883db82b4ae4e11c728636
Activity.SpanId:             9edfe4d69b0d6d8b
Activity.TraceFlags:         Recorded
Activity.ParentSpanId:       39fcd105cf958377
Activity.ActivitySourceName: dice-server
Activity.DisplayName:        rollTheDice
Activity.Kind:               Internal
Activity.StartTime:          2024-04-10T15:24:00.3620354Z
Activity.Duration:           00:00:00.0144329
Resource associated with Activity:
    service.name: dice-server
    service.version: 1.0.0
    service.instance.id: 7a7a134f-3178-4ac6-9625-96df77cff8b4
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.7.0

Create nested Activities

Nested spans let you track work that’s nested in nature. For example, the rollOnce() function below represents a nested operation. The following sample creates a nested span that tracks rollOnce():

private int rollOnce()
{
    using (var childActivity = activitySource.StartActivity("rollOnce"))
    {
      int result;

      result = Random.Shared.Next(min, max + 1);

      return result;
    }
}

When you view the spans in a trace visualization tool, rollOnce childActivity will be tracked as a nested operation under rollTheDice activity.

Get the current Activity

Sometimes it’s helpful to do something with the current/active Activity/Span at a particular point in program execution.

var activity = Activity.Current;

Activity Tags

Tags (the equivalent of Attributes) let you attach key/value pairs to an Activity so it carries more information about the current operation that it’s tracking.

private int rollOnce()
{
  using (var childActivity = activitySource.StartActivity("rollOnce"))
    {
      int result;

      result = Random.Shared.Next(min, max + 1);
      childActivity?.SetTag("dicelib.rolled", result);

      return result;
    }
}

Add Events to Activities

Spans can be annotated with named events (called Span Events) that can carry zero or more Span Attributes, each of which itself is a key:value map paired automatically with a timestamp.

myActivity?.AddEvent(new("Init"));
...
myActivity?.AddEvent(new("End"));

var eventTags = new ActivityTagsCollection
{
    { "operation", "calculate-pi" },
    { "result", 3.14159 }
};

activity?.AddEvent(new("End Computation", DateTimeOffset.Now, eventTags));

A Span may be linked to zero or more other Spans that are causally related via a Span Link. Links can be used to represent batched operations where a Span was initiated by multiple initiating Spans, each representing a single incoming item being processed in the batch.

var links = new List<ActivityLink>
{
    new ActivityLink(activityContext1),
    new ActivityLink(activityContext2),
    new ActivityLink(activityContext3)
};

var activity = MyActivitySource.StartActivity(
    ActivityKind.Internal,
    name: "activity-with-links",
    links: links);

Set Activity status

A Status can be set on a Span, typically used to specify that a Span has not completed successfully - Error. By default, all spans are Unset, which means a span completed without error. The Ok status is reserved for when you need to explicitly mark a span as successful rather than stick with the default of Unset (i.e., “without error”).

The status can be set at any time before the span is finished.

A status can be set on a span, typically used to specify that a span has not completed successfully - SpanStatus.Error.

By default, all spans are Unset, which means a span completed without error. The Ok status is reserved for when you need to explicitly mark a span as successful rather than stick with the default of Unset (i.e., “without error”).

The status can be set at any time before the span is finished.

It can be a good idea to record exceptions when they happen. It’s recommended to do this in conjunction with setting span status.

private int rollOnce()
{
    using (var childActivity = activitySource.StartActivity("rollOnce"))
    {
        int result;

        try
        {
            result = Random.Shared.Next(min, max + 1);
            childActivity?.SetTag("dicelib.rolled", result);
        }
        catch (Exception ex)
        {
            childActivity?.SetStatus(ActivityStatusCode.Error, "Something bad happened!");
            childActivity?.RecordException(ex);
            throw;
        }

        return result;
    }
}

Metrics

The documentation for the metrics API & SDK is missing, you can help make it available by editing this page.

Logs

The documentation for the logs API and SDK is missing. You can help make it available by editing this page.

Next steps

After you’ve set up manual instrumentation, you may want to use instrumentation libraries. As the name suggests, they will instrument relevant libraries you’re using and generate spans (activities) for things like inbound and outbound HTTP requests and more.

You’ll also want to configure an appropriate exporter to export your telemetry data to one or more telemetry backends.

You can also check the automatic instrumentation for .NET, which is currently in beta.

Last modified May 16, 2024: Update .NET manual instrumentation docs with example app (#3834) (744ef19d)