Instrumentation ecosystem

Instrumentation records telemetry using the API. The SDK is the built-in reference implementation of the API, and is configured to process and export the telemetry produced by instrumentation API calls. This page discusses the OpenTelemetry ecosystem in OpenTelemetry Java, including resources for end users and cross-cutting instrumentation topics:

• Instrumentation categories: There are a variety of categories of instrumentation addressing different use cases and installation patterns.
• Context propagation: Context propagation provides correlation between traces, metrics, and logs, allowing the signals to complement each other.
• Semantic conventions: The semantic conventions define how to produce telemetry for standard operations.
• Log instrumentation: The semantic conventions define how to produce telemetry for standard operations.

Instrumentation categories

There are several categories of instrumentation:

• Zero-code: Java agent is a form of zero-code instrumentation [1] that dynamically manipulates application bytecode.
• Zero-code: Spring Boot starter is a form of zero-code instrumentation [1] that leverages spring autoconfigure to install library instrumentation.
• Library instrumentation wraps or uses extension points to instrument a library, requiring users to install and/or adapt library usage.
• Native instrumentation is built directly into libraries and frameworks.
• Manual instrumentation is written by application authors, and typically specific to the application domain.

[1]: Zero-code instrumentation is installed automatically based on detected libraries / frameworks.

The opentelemetry-java-instrumentation project contains the source code for Java agent, Spring Boot starter, and Library instrumentation.

Zero-code: Java agent

The Java agent is a form of zero-code automatic instrumentation that dynamically manipulates application bytecode.

For a list of libraries instrumented by the Java agent, see the “Auto-instrumented versions” column on supported libraries.

See Java agent for more details.

Zero-code: Spring Boot starter

The Spring Boot starter is a form of zero-code automatic instrumentation that leverages spring autoconfigure to install library instrumentation.

See Spring Boot starter for details.

Library instrumentation

Library instrumentation wraps or uses extension points to instrument a library, requiring users to install and/or adapt library usage.

For a list of instrumentation libraries, see the “Standalone Library Instrumentation” column on supported libraries.

Native instrumentation

Native instrumentation is built directly into libraries or frameworks. OpenTelemetry encourages library authors to add native instrumentation using the API. In the long term, we hope the native instrumentation becomes the norm, and view the instrumentation maintained by OpenTelemetry in opentelemetry-java-instrumentation as a temporary means of filling the gap.

• Elasticsearch Java API Client
• Micrometer Tracing
• Quarkus

Manual instrumentation

Manual instrumentation is written by application authors, and typically specific to the application domain.

Shims

A shim is instrumentation that bridges data from one observability library to another, typically from some library into OpenTelemetry.

Shims maintained in the OpenTelemetry Java ecosystem:

Context propagation

The OpenTelemetry APIs are designed to be complementary, with the whole greater than the sum of the parts. Each signal has its own strengths, and collectively stitch together a compelling observability story.

Importantly, the data from the various signals are linked together through trace context:

• Spans are related to other spans through span parent and links, which each record trace context of related spans.
• Metrics are related to spans through exemplars, which record trace context of a particular measurement.
• Logs are related to spans by recording trace context on log records.

For this correlation to work, trace context must be propagated throughout an application (across function calls and threads), and across application boundaries. The context API facilitates this. Instrumentation needs to be written in a manner which is context aware:

• Libraries that represent the entry point to an application (i.e. HTTP servers, message consumers, etc.) should extract context from incoming messages.
• Libraries that represent an exit point from an application (i.e. HTTP clients, message producers, etc.) should inject context into outgoing messages.
• Libraries should implicitly or explicitly pass Context through the callstack and across any threads.

Semantic conventions

The semantic conventions define how to produce telemetry for standard operations. Among other things, the semantic conventions specify span names, span kinds, metric instruments, metric units, metric types, and attribute key, value, and requirement levels.

When writing instrumentation, consult the semantic conventions and conform to any which are applicable to the domain.

OpenTelemetry Java publishes artifacts to assist in conforming to the semantic conventions, including generated constants for attribute keys and values.

TODO: discuss instrumentation API and how it helps conform to semantic conventions

Log instrumentation

While the LoggerProvider / Logger APIs are structurally similar to the equivalent trace and metric APIs, they serve a different use case. As of now, LoggerProvider / Logger and associated classes represent the Log Bridge API, which exists to write log appenders to bridge logs recorded through other log APIs / frameworks into OpenTelemetry. They are not intended for end user use as a replacement for Log4j / SLF4J / Logback / etc.

There are two typical workflows for consuming log instrumentation in OpenTelemetry catering to different application requirements:

Direct to collector

In the direct to collector workflow, logs are emitted directly from an application to a collector using a network protocol (e.g. OTLP). This workflow is simple to set up as it doesn’t require any additional log forwarding components, and allows an application to easily emit structured logs that conform to the log data model. However, the overhead required for applications to queue and export logs to a network location may not be suitable for all applications.

To use this workflow:

• Install appropriate log appender. [1]
• Configure the OpenTelemetry Log SDK to export log records to desired target destination (the collector or other).

[1]: Log appenders are a type of shim which bridges logs from a log framework into the OpenTelemetry log SDK. See “Bridge Log4j into OpenTelemetry”, “Bridge Logback into OpenTelemetry” entries. See Log Appender example for demonstration of a variety of scenarios.

Via file or stdout

In the file or stdout workflow, logs are written to files or standout output. Another component (e.g. FluentBit) is responsible for reading / tailing the logs, parsing them to more structured format, and forwarding them a target, such as the collector. This workflow may be preferable in situations where application requirements do not permit additional overhead from direct to collector. However, it requires that all log fields required down stream are encoded into the logs, and that the component reading the logs parse the data into the log data model. The installation and configuration of log forwarding components is outside the scope of this document.

Log correlation with traces is available by installing a shim to bridge OpenTelemetry context into the log framework. See “Bridge OpenTelemetry context into Log4j”, “Bridge OpenTelemetry context into Logback” entries.