Agent Configuration

SDK Autoconfiguration

The SDK’s autoconfiguration module is used for basic configuration of the agent. Read the docs to find settings such as configuring export or sampling.

Here are some quick links into those docs for the configuration options for specific portions of the SDK & agent:

Enable Resource Providers that are disabled by default

In addition to the resource configuration from the SDK autoconfiguration, you can enable additional resource providers that are disabled by default:

System property: otel.resource.providers.aws.enabled Environment variable: OTEL_RESOURCE_PROVIDERS_AWS_ENABLED

Default: false
Description: Enables the AWS Resource Provider.

System property: otel.resource.providers.gcp.enabled Environment variable: OTEL_RESOURCE_PROVIDERS_GCP_ENABLED

Default: false
Description: Enables the GCP Resource Provider.

Configuring the agent

The agent can consume configuration from one or more of the following sources (ordered from highest to lowest priority):

Configuring with Environment Variables

In some environments, configuring via Environment Variables is more preferred. Any setting configurable with a System Property can also be configured with an Environment Variable. Many settings below include both options, but where they don’t apply the following steps to determine the correct name mapping of the desired System Property:

  • Convert the System Property to uppercase.
  • Replace all . and - characters with _.

For example otel.instrumentation.common.default-enabled would convert to OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED.

Configuration file

You can provide a path to agent configuration file by setting the following property:

System property: otel.javaagent.configuration-file Environment variable: OTEL_JAVAAGENT_CONFIGURATION_FILE

Description: Path to valid Java properties file which contains the agent configuration.

Extensions

You can enable extensions by setting the following property:

System property: otel.javaagent.extensions Environment variable: OTEL_JAVAAGENT_EXTENSIONS

Description: Path to an extension jar file or folder, containing jar files. If pointing to a folder, every jar file in that folder will be treated as separate, independent extension.

Java agent logging output

The agent’s logging output can be configured by setting the following property:

System property: otel.javaagent.logging Environment variable: OTEL_JAVAAGENT_LOGGING

Description: The Java agent logging mode. The following 3 modes are supported:

  • simple: The agent will print out its logs using the standard error stream. Only INFO or higher logs will be printed. This is the default Java agent logging mode.
  • none: The agent will not log anything - not even its own version.
  • application: The agent will attempt to redirect its own logs to the instrumented application's slf4j logger. This works the best for simple one-jar applications that do not use multiple classloaders; Spring Boot apps are supported as well. The Java agent output logs can be further configured using the instrumented application's logging configuration (e.g. logback.xml or log4j2.xml). Make sure to test that this mode works for your application before running it in a production environment.

Common instrumentation configuration

Common settings that apply to multiple instrumentations at once.

Peer service name

The peer service name is the name of a remote service to which a connection is made. It corresponds to service.name in the resource for the local service.

System property: otel.instrumentation.common.peer-service-mapping Environment variable: OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING

Description: Used to specify a mapping from host names or IP addresses to peer services, as a comma-separated list of <host_or_ip>=<user_assigned_name> pairs. The peer service is added as an attribute to a span whose host or IP address match the mapping.

For example, if set to the following:

1.2.3.4=cats-service,dogs-abcdef123.serverlessapis.com=dogs-api

Then, requests to 1.2.3.4 will have a peer.service attribute of cats-service and requests to dogs-abcdef123.serverlessapis.com will have an attribute of dogs-api.

Since Java agent version 1.31.0, it is possible to provide a port and a path to define a peer.service.

For example, if set to the following:

1.2.3.4:443=cats-service,dogs-abcdef123.serverlessapis.com:80/api=dogs-api

Then, requests to 1.2.3.4 will have no override for peer.service attribute, while 1.2.3.4:443 will have have peer.service of cats-service and requests to dogs-abcdef123.serverlessapis.com:80/api/v1 will have an attribute of dogs-api.

DB statement sanitization

The agent sanitizes all database queries/statements before setting the db.statement semantic attribute. All values (strings, numbers) in the query string are replaced with a question mark (?).

Note: JDBC bind parameters are not captured in db.statement. See the corresponding issue if you are looking to capture bind parameters.

Examples:

  • SQL query SELECT a from b where password="secret" will appear as SELECT a from b where password=? in the exported span;
  • Redis command HSET map password "secret" will appear as HSET map password ? in the exported span.

This behavior is turned on by default for all database instrumentations. Use the following property to disable it:

System property: otel.instrumentation.common.db-statement-sanitizer.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED

Default: true
Description: Enables the DB statement sanitization.

Capturing HTTP request and response headers

You can configure the agent to capture predefined HTTP headers as span attributes, according to the semantic convention. Use the following properties to define which HTTP headers you want to capture:

System property: otel.instrumentation.http.client.capture-request-headers Environment variable: OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_REQUEST_HEADERS

Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP request header values for all configured header names.

System property: otel.instrumentation.http.client.capture-response-headers Environment variable: OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS

Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP response header values for all configured header names.

System property: otel.instrumentation.http.server.capture-request-headers Environment variable: OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS

Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP request header values for all configured header names.

System property: otel.instrumentation.http.server.capture-response-headers Environment variable: OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_RESPONSE_HEADERS

Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP response header values for all configured header names.

These configuration options are supported by all HTTP client and server instrumentations.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Capturing servlet request parameters

You can configure the agent to capture predefined HTTP request parameter as span attributes for requests that are handled by Servlet API. Use the following property to define which servlet request parameters you want to capture:

System property: otel.instrumentation.servlet.experimental.capture-request-parameters Environment variable: OTEL_INSTRUMENTATION_SERVLET_EXPERIMENTAL_CAPTURE_REQUEST_PARAMETERS

Description: A comma-separated list of request parameter names.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Capturing consumer message receive telemetry in messaging instrumentations

You can configure the agent to capture the consumer message receive telemetry in messaging instrumentation. Use the following property to enable it:

System property: otel.instrumentation.messaging.experimental.receive-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_MESSAGING_EXPERIMENTAL_RECEIVE_TELEMETRY_ENABLED

Default: false
Description: Enables the consumer message receive telemetry.

Note that this will cause the consumer side to start a new trace, with only a span link connecting it to the producer trace.

Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.

Capturing enduser attributes

You can configure the agent to capture general identity attributes (enduser.id, enduser.role, enduser.scope) from instrumentation libraries like JavaEE/JakartaEE Servlet and Spring Security.

Note: Given the sensitive nature of the data involved, this feature is turned off by default while allowing selective activation for particular attributes. You must carefully evaluate each attribute’s privacy implications before enabling the collection of the data.

System property: otel.instrumentation.common.enduser.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_ENDUSER_ENABLED

Default: false
Description: Common flag for enabling/disabling enduser attributes.

System property: otel.instrumentation.common.enduser.id.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_ENDUSER_ID_ENABLED

Default: false
Description: Determines whether to capture enduser.id semantic attribute.

System property: otel.instrumentation.common.enduser.role.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_ENDUSER_ROLE_ENABLED

Default: false
Description: Determines whether to capture enduser.role semantic attribute.

System property: otel.instrumentation.common.enduser.scope.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_ENDUSER_SCOPE_ENABLED

Default: false
Description: Determines whether to capture enduser.scope semantic attribute.

Spring Security

For users of Spring Security who use custom granted authority prefixes, you can use the following properties to strip those prefixes from the enduser.* attribute values to better represent the actual role and scope names:

System property: otel.instrumentation.spring-security.enduser.role.granted-authority-prefix Environment variable: OTEL_INSTRUMENTATION_SPRING_SECURITY_ENDUSER_ROLE_GRANTED_AUTHORITY_PREFIX

Default: ROLE_
Description: Prefix of granted authorities identifying roles to capture in the enduser.role semantic attribute.

System property: otel.instrumentation.spring-security.enduser.scope.granted-authority-prefix Environment variable: OTEL_INSTRUMENTATION_SPRING_SECURITY_ENDUSER_SCOPE_GRANTED_AUTHORITY_PREFIX

Default: SCOPE_
Description: Prefix of granted authorities identifying scopes to capture in the enduser.scopes semantic attribute.

Suppressing specific auto-instrumentation

Disabling the agent entirely

System property: otel.javaagent.enabled Environment variable: OTEL_JAVAAGENT_ENABLED

Description: Set the value to false to disable the agent entirely.

Enable only specific instrumentation

You can disable all default auto instrumentation and selectively re-enable individual instrumentation. This may be desirable to reduce startup overhead or to have more control of which instrumentation is applied.

System property: otel.instrumentation.common.default-enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED

Description: Set to false to disable all instrumentation in the agent.

System property: otel.instrumentation.[name].enabled Environment variable: OTEL_INSTRUMENTATION_[NAME]_ENABLED

Description: Set to true to enable each desired instrumentation individually.

Note: Some instrumentation relies on other instrumentation to function properly. When selectively enabling instrumentation, be sure to enable the transitive dependencies too. Determining this dependency relationship is left as an exercise to the user.

Enable manual instrumentation only

You can suppress all auto instrumentations but have support for manual instrumentation with @WithSpan and normal API interactions by using -Dotel.instrumentation.common.default-enabled=false -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true

Suppressing specific agent instrumentation

You can suppress agent instrumentation of specific libraries.

System property: otel.instrumentation.[name].enabled Environment variable: OTEL_INSTRUMENTATION_[NAME]_ENABLED

Description: Set to false to suppress agent instrumentation of specific libraries, where [name] is the corresponding instrumentation name:

Library/Framework Instrumentation name
Additional methods tracing methods
Additional tracing annotations external-annotations
Akka Actor akka-actor
Akka HTTP akka-http
Apache Axis2 axis2
Apache Camel camel
Apache Cassandra cassandra
Apache CXF cxf
Apache DBCP apache-dbcp
Apache Dubbo apache-dubbo
Apache Geode geode
Apache HttpAsyncClient apache-httpasyncclient
Apache HttpClient apache-httpclient
Apache Kafka kafka
Apache MyFaces jsf-myfaces
Apache Pekko Actor pekko-actor
Apache Pekko HTTP pekko-http
Apache Pulsar pulsar
Apache RocketMQ rocketmq-client
Apache Struts 2 struts
Apache Tapestry tapestry
Apache Tomcat tomcat
Apache Wicket wicket
Armeria armeria
AsyncHttpClient (AHC) async-http-client
AWS Lambda aws-lambda
AWS SDK aws-sdk
Azure SDK azure-core
Couchbase couchbase
C3P0 c3p0
Dropwizard Views dropwizard-views
Dropwizard Metrics dropwizard-metrics
Eclipse Grizzly grizzly
Eclipse Jersey jersey
Eclipse Jetty jetty
Eclipse Jetty HTTP Client jetty-httpclient
Eclipse Metro metro
Eclipse Mojarra jsf-mojarra
Eclipse Vert.x HttpClient vertx-http-client
Eclipse Vert.x Kafka Client vertx-kafka-client
Eclipse Vert.x RxJava vertx-rx-java
Eclipse Vert.x Web vertx-web
Elasticsearch client elasticsearch-transport
Elasticsearch REST client elasticsearch-rest
Google Guava guava
Google HTTP client google-http-client
Google Web Toolkit gwt
Grails grails
GraphQL Java graphql-java
GRPC grpc
Hibernate hibernate
HikariCP hikaricp
Java HTTP Client java-http-client
Java HttpURLConnection http-url-connection
Java JDBC jdbc
Java JDBC DataSource jdbc-datasource
Java RMI rmi
Java Runtime runtime-telemetry
Java Servlet servlet
java.util.concurrent executors
java.util.logging java-util-logging
JAX-RS (Client) jaxrs-client
JAX-RS (Server) jaxrs
JAX-WS jaxws
JBoss Logging Appender jboss-logmanager-appender
JBoss Logging MDC jboss-logmanager-mdc
JMS jms
Jodd HTTP jodd-http
JSP jsp
K8s Client kubernetes-client
kotlinx.coroutines kotlinx-coroutines
Log4j Appender log4j-appender
Log4j MDC (1.x) log4j-mdc
Log4j Context Data (2.x) log4j-context-data
Logback Appender logback-appender
Logback MDC logback-mdc
Micrometer micrometer
MongoDB mongo
Netflix Hystrix hystrix
Netty netty
OkHttp okhttp
OpenLiberty liberty
OpenTelemetry Extension Annotations opentelemetry-extension-annotations
OpenTelemetry Instrumentation Annotations opentelemetry-instrumentation-annotations
OpenTelemetry API opentelemetry-api
Oracle UCP oracle-ucp
OSHI (Operating System and Hardware Information) oshi
Play Framework play
Play WS HTTP Client play-ws
Quartz quartz
R2DBC r2dbc
RabbitMQ Client rabbitmq
Ratpack ratpack
ReactiveX RxJava rxjava
Reactor reactor
Reactor Netty reactor-netty
Redis Jedis jedis
Redis Lettuce lettuce
Rediscala rediscala
Redisson redisson
Restlet restlet
Scala ForkJoinPool scala-fork-join
Spark Web Framework spark
Spring Batch spring-batch
Spring Core spring-core
Spring Data spring-data
Spring JMS spring-jms
Spring Integration spring-integration
Spring Kafka spring-kafka
Spring RabbitMQ spring-rabbit
Spring RMI spring-rmi
Spring Scheduling spring-scheduling
Spring Web spring-web
Spring WebFlux spring-webflux
Spring Web MVC spring-webmvc
Spring Web Services spring-ws
Spymemcached spymemcached
Tomcat JDBC tomcat-jdbc
Twilio SDK twilio
Twitter Finatra finatra
Undertow undertow
Vaadin vaadin
Vibur DBCP vibur-dbcp
ZIO zio

Note: When using environment variables, dashes (-) should be converted to underscores (_). For example, to suppress traces from akka-actor library, set OTEL_INSTRUMENTATION_AKKA_ACTOR_ENABLED to false.

Suppressing controller and/or view spans

Some instrumentations (e.g. Spring Web MVC instrumentation) produce SpanKind.Internal spans to capture the controller and/or view execution. These spans can be suppressed using the configuration settings below, without suppressing the entire instrumentation which would also disable the instrumentation’s capturing of http.route and associated span name on the parent SpanKind.Server span.

System property: otel.instrumentation.common.experimental.controller-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED

Default: false
Description: Set to true to enable controller telemetry.

System property: otel.instrumentation.common.experimental.view-telemetry.enabled Environment variable: OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED

Default: false
Description: Set to true to enable view telemetry.

Instrumentation span suppression behavior

Some libraries that this agent instruments in turn use lower-level libraries, that are also instrumented. This would normally result in nested spans containing duplicate telemetry data. For example:

  • Spans produced by the Reactor Netty HTTP client instrumentation would have duplicate HTTP client spans produced by the Netty instrumentation;
  • Dynamo DB spans produced by the AWS SDK instrumentation would have children HTTP client spans produced by its internal HTTP client library (which is also instrumented);
  • Spans produced by the Tomcat instrumentation would have duplicate HTTP server spans produced by the generic Servlet API instrumentation.

The Java agent prevents these situations by detecting and suppressing nested spans that duplicate telemetry data. The suppression behavior can be configured using the following configuration option:

System property: otel.instrumentation.experimental.span-suppression-strategy Environment variable: OTEL_INSTRUMENTATION_EXPERIMENTAL_SPAN_SUPPRESSION_STRATEGY

Description: The Java agent span suppression strategy. The following 3 strategies are supported:

  • semconv: The agent will suppress duplicate semantic conventions. This is the default behavior of the Java agent.
  • span-kind: The agent will suppress spans with the same kind (except INTERNAL).
  • none: The agent will not suppress anything at all. We do not recommend using this option for anything other than debug purposes, as it generates lots of duplicate telemetry data.

For example, suppose we instrument a database client which internally uses the Reactor Netty HTTP client; which in turn uses Netty.

Using the default semconv suppression strategy would result in 2 nested CLIENT spans:

  • CLIENT span with database client semantic attributes emitted by the database client instrumentation;
  • CLIENT span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation.

The Netty instrumentation would be suppressed, as it duplicates the Reactor Netty HTTP client instrumentation.

Using the suppression strategy span-kind would result in just one span:

  • CLIENT span with database client semantic attributes emitted by the database client instrumentation.

Both Reactor Netty and Netty instrumentations would be suppressed, as they also emit CLIENT spans.

Finally, using the suppression strategy none would result in 3 spans:

  • CLIENT span with database client semantic attributes emitted by the database client instrumentation;
  • CLIENT span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation;
  • CLIENT span with HTTP client semantic attributes emitted by the Netty instrumentation.