Configuration SDK

Status: Development

Overview

The configuration SDK is part of the declarative configuration interface.

The SDK is an implementation of Instrumentation Config API and other user facing declarative configuration capabilities. It consists of the following main components:

• In-Memory configuration model is an in-memory representation of the configuration model.
• ConfigProvider defines the SDK implementation of the ConfigProvider API.
• SDK extension components defines how users and libraries extend file configuration with custom SDK extension plugin interfaces (exporters, processors, etc).
• SDK operations defines user APIs to parse configuration files and produce SDK components from their contents.

In-Memory configuration model

SDKs SHOULD provide an in-memory representation of the configuration model. Whereas ConfigProperties is a schemaless representation of any mapping node, the in-memory configuration model SHOULD reflect the schema of the configuration model.

SDKs are encouraged to provide this in-memory representation in a manner that is idiomatic for their language. If an SDK needs to expose a class or interface, the name Configuration is RECOMMENDED.

ConfigProvider

The SDK implementation of ConfigProvider MUST be created using a ConfigProperties representing the .instrumentation mapping node of the configuration model.

SDK extension components

The SDK supports a variety of extension plugin interfaces, allowing users and libraries to customize behaviors including the sampling, processing, and exporting of data. In general, the configuration data model defines specific types for built-in implementations of these plugin interfaces. For example, the BatchSpanProcessor type refers to the built-in Batching span processor. The schema SHOULD also support the ability to specify custom implementations of plugin interfaces defined by libraries or users.

For example, a custom span exporter might be configured as follows:

tracer_provider:
  processors:
    - batch:
        exporter:
          my-exporter:
            config-parameter: value

Here we specify that the tracer provider has a batch span processor paired with a custom span exporter named my-exporter, which is configured with config-parameter: value. For this configuration to succeed, a ComponentProvider must be registered with type: SpanExporter, and name: my-exporter. When parse is called, the implementation will encounter my-exporter and translate the corresponding configuration to an equivalent ConfigProperties representation ( i.e. properties: {config-parameter: value}). When create is called, the implementation will encounter my-exporter and invoke create plugin on the registered ComponentProviderwith the ConfigProperties determined during parse.

Given the inherent differences across languages, the details of extension component mechanisms are likely to vary to a greater degree than is the case with other APIs defined by OpenTelemetry. This is to be expected and is acceptable so long as the implementation results in the defined behaviors.

ComponentProvider

A ComponentProvider is responsible for interpreting configuration and returning an implementation of a particular type of SDK extension plugin interface.

ComponentProviders are registered with an SDK implementation of configuration via register. This MAY be done automatically or require manual intervention by the user based on what is possible and idiomatic in the language ecosystem. For example in Java, ComponentProviders might be registered automatically using the service provider interface (SPI) mechanism.

See create, which details ComponentProvider usage in configuration model interpretation.

Supported SDK extension plugins

The configuration data model SHOULD support configuration of all SDK extension plugin interfaces. SDKs SHOULD support registration of custom implementations of SDK extension plugin interfaces via the ComponentProvider mechanism.

The following table lists each SDK extension plugin interface and its corresponding type in the configuration data model:

ComponentsProvider operations

The ComponentsProvider MUST provide the following functions:

• Create Plugin

Create Plugin

Interpret configuration to create a instance of a SDK extension plugin interface.

Parameters:

• properties - The ConfigProperties representing the configuration specified for the component in the configuration model.

Returns: A configured SDK extension plugin interface implementation.

The plugin interface MAY have properties which are optional or required, and have specific requirements around type or format. The set of properties a ComponentProvider accepts, along with their requirement level and expected type, comprise a configuration schema. A ComponentProvider SHOULD document its configuration schema and include examples.

When Create Plugin is invoked, the ComponentProvider interprets properties and attempts to extract data according to its configuration schema. If this fails (e.g. a required property is not present, a type is mismatches, etc.), Create Plugin SHOULD return an error.

SDK operations

SDK implementations of configuration MUST provide the following operations.

Note: Because these operations are stateless pure functions, they are not defined as part of any type, class, interface, etc. SDKs may organize them in whatever manner is idiomatic for the language.

TODO: Add operation to update SDK components with new configuration for usage with OpAmp

Parse

Parse and validate a configuration file.

Parameters:

• file: The configuration file to parse. This MAY be a file path, or language specific file data structure, or a stream of a file’s content.
• file_format: The file format of the file (e.g. yaml). Implementations MAY accept a file_format parameter, or infer it from the file extension, or include file format specific overloads of parse, e.g. parseYaml(file). If parse accepts file_format, the API SHOULD be structured so a user is obligated to provide it.

Returns: configuration model

Parse MUST perform environment variable substitution.

Parse MUST differentiate between properties that are missing and properties that are present but null. For example, consider the following snippet, noting .meter_provider.views[0].stream.drop is present but null:

meter_provider:
  views:
    - selector:
        name: some.metric.name
      stream:
        aggregation:
          drop:

As a result, the view stream should be configured with the drop aggregation. Note that some aggregations have additional arguments, but drop does not. The user MUST not be required to specify an empty object (i.e. drop: {}) in these cases.

When encountering a reference to a SDK extension component which is not built in to the SDK, Parse MUST resolve corresponding configuration to a generic ConfigProperties representation as described in Create Plugin.

Parse SHOULD return an error if:

• The file doesn’t exist or is invalid
• The parsed file content does not conform to the configuration model schema. Note that this includes enforcing all constraints encoded into the schema (e.g. required properties are present, that properties adhere to specified types, etc.).

Create

Interpret configuration model and return SDK components.

Parameters:

• configuration - An in-memory configuration model.

Returns: Top level SDK components:

• TracerProvider
• MeterProvider
• LoggerProvider
• Propagators
• ConfigProvider

The multiple responses MAY be returned using a tuple, or some other data structure encapsulating the components.

Create requirements around default and null behavior are described below. Note that defaultBehavior and nullBehavior are defined in the configuration data model.

• If a property is present and the value is null, Create MUST use the nullBehavior, or defaultBehavior if nullBehavior is not set.
• If a property is required, and not present, Create MUST return an error.

A few examples to illustrate:

• If configuring BatchSpanProcessor and schedule_delay is not present or present but null, the component is configured according to the defaultBehavior of 5000.
• If configuring SpanExporter and console is present and null, the component is configured with a console exporter with default configuration since console is nullable.

The configuration model uses the JSON schema description annotation to capture property semantics which cannot be encoded using standard JSON schema keywords. Create SHOULD return an error if it encounters a value which is invalid according to the property description. For example, if configuring HttpTls and ca_file is not an absolute file path as defined in the property description, return an error.

When encountering a reference to a SDK extension component which is not built in to the SDK, Create MUST resolve the component using Create Plugin of the ComponentProvider of the corresponding type and name used to register, including the configuration properties as an argument. If no ComponentProvider is registered with the type and name, Create SHOULD return an error. If Create Plugin returns an error, Create SHOULD propagate the error.

This SHOULD return an error if it encounters an error in configuration (i.e. fail fast) in accordance with initialization error handling principles.

SDK implementations MAY provide options to allow programmatic customization of the components initialized by Create. This allows configuration of concepts which are not yet or may never be representable in the configuration model. For example, java OTLP exporters allow configuration of the ExecutorService, a niche but important option for applications which need strict control of thread pools. This programmatic customization might take the form of passing an optional callback to Create, invoked with each SDK subcomponent (or a subset of SDK component types) as they are initialized. For example, consider the following snippet:

file_format: 1.0
tracer_provider:
  processors:
    - batch:
        exporter:
          otlp_http:

The callback would be invoked with the SDK representation of an OTLP HTTP exporter, a Batch SpanProcessor, and a Tracer Provider. This pattern provides the opportunity to programmatically configure lower-level without needing to walk to a particular component from the resolved top level SDK components.

TODO: define behavior if some portion of configuration model is not supported

Register ComponentProvider

The SDK MUST provide a mechanism to register ComponentProvider. The mechanism MAY be language-specific and automatic. For example, a java implementation might use the service provider interface mechanism to register implementations of a particular interface as ComponentProviders.

Parameters:

• component_provider - The ComponentProvider.
• type - The type of plugin interface it provides.
• name - The name used to identify the type of component. This is used in configuration model to specify that the corresponding component_provider is to provide the component.

The type and name comprise a unique key. Register MUST return an error if it is called multiple times with the same type and name combination.

SDKs SHOULD represent type in a manner that is idiomatic for their language. For example, a class literal, an enumeration, or similar. See supported SDK extension plugins for the set of supported type values.

Examples

Via configuration API

The configuration Parse and Create operations along with the Configuration Model can be combined in a variety of ways to achieve simple or complex configuration goals.

For example, a simple case would consist of calling Parse with a configuration file, and passing the result to Create to obtain configured SDK components:

OpenTelemetry openTelemetry = OpenTelemetry.noop();
try {
    // Parse configuration file to configuration model
    OpenTelemetryConfiguration configurationModel = parse(new File("/app/sdk-config.yaml"));
    // Create SDK components from configuration model
    openTelemetry = create(configurationModel);
} catch (Throwable e) {
    log.error("Error initializing SDK from configuration file", e);
}

// Access SDK components and install instrumentation
TracerProvider tracerProvider = openTelemetry.getTracerProvider();
MeterProvider meterProvider = openTelemetry.getMeterProvider();
LoggerProvider loggerProvider = openTelemetry.getLogsBridge();
ContextPropagators propagators = openTelemetry.getPropagators();
ConfigProvider configProvider = openTelemetry.getConfigProvider();

A more complex case might consist of parsing multiple configuration files from different sources, merging them using custom logic, and creating SDK components from the merged configuration model:

OpenTelemetry openTelemetry = OpenTelemetry.noop();
try {
    // Parse local and remote configuration files to configuration models
    OpenTelemetryConfiguration localConfigurationModel = parse(new File("/app/sdk-config.yaml"));
    OpenTelemetryConfiguration remoteConfigurationModel = parse(getRemoteConfiguration("http://example-host/config/my-application"));

    // Merge the configuration models using custom logic
    OpenTelemetryConfiguration resolvedConfigurationModel = merge(localConfigurationModel, remoteConfigurationModel);

    // Create SDK components from resolved configuration model
    openTelemetry = create(resolvedConfigurationModel);
} catch (Throwable e) {
    log.error("Error initializing SDK from configuration file", e);
}

// Access SDK components and install instrumentation
TracerProvider tracerProvider = openTelemetry.getTracerProvider();
MeterProvider meterProvider = openTelemetry.getMeterProvider();
LoggerProvider loggerProvider = openTelemetry.getLogsBridge();
ContextPropagators propagators = openTelemetry.getPropagators();
ConfigProvider configProvider = openTelemetry.getConfigProvider();

Via OTEL_EXPERIMENTAL_CONFIG_FILE

Setting the OTEL_EXPERIMENTAL_CONFIG_FILE environment variable (for languages that support it) provides users a convenient way to initialize OpenTelemetry components without needing to learn language-specific configuration details or use a large number of environment variables. The pattern for accessing the configured components and installing into instrumentation will vary by language. For example, the usage in Java might resemble:

# Set the required env var to the location of the configuration file
export OTEL_EXPERIMENTAL_CONFIG_FILE="/app/sdk-config.yaml"

// Initialize SDK using autoconfigure model, which recognizes that OTEL_EXPERIMENTAL_CONFIG_FILE is set and configures the SDK accordingly
OpenTelemetry openTelemetry = AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();

// Access SDK components and install instrumentation
TracerProvider tracerProvider = openTelemetry.getTracerProvider();
MeterProvider meterProvider = openTelemetry.getMeterProvider();
LoggerProvider loggerProvider = openTelemetry.getLogsBridge();
ContextPropagators propagators = openTelemetry.getPropagators();
ConfigProvider configProvider = openTelemetry.getConfigProvider();

If using auto-instrumentation, this initialization flow might occur automatically.

References

• Configuration proposal (OTEP #225)