Internal telemetry

You can monitor the health of any OpenTelemetry Collector instance by checking its own internal telemetry. Read on to learn about this telemetry and how to configure it to help you troubleshoot Collector issues.

Activate internal telemetry in the Collector

By default, the Collector exposes its own telemetry in two ways:

• Internal metrics are exposed using a Prometheus interface which defaults to port 8888.
• Logs are emitted to stderr by default.

Configure internal metrics

You can configure how internal metrics are generated and exposed by the Collector. By default, the Collector generates basic metrics about itself and exposes them for scraping at http://127.0.0.1:8888/metrics. You can expose the endpoint to one specific or all network interfaces when needed. For containerized environments, you might want to expose this port on a public interface.

Set the address in the config service::telemetry::metrics:

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888

You can adjust the verbosity of the Collector metrics output by setting the level field to one of the following values:

• none: no telemetry is collected.
• basic: essential service telemetry.
• normal: the default level, adds standard indicators on top of basic.
• detailed: the most verbose level, includes dimensions and views.

Each verbosity level represents a threshold at which certain metrics are emitted. For the complete list of metrics, with a breakdown by level, see Lists of internal metrics.

The default level for metrics output is normal. To use another level, set service::telemetry::metrics::level:

service:
  telemetry:
    metrics:
      level: detailed

The Collector can also be configured to scrape its own metrics and send them through configured pipelines. For example:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otelcol'
          scrape_interval: 10s
          static_configs:
            - targets: ['0.0.0.0:8888']
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: '.*grpc_io.*'
              action: drop
exporters:
  debug:
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [debug]

Configure internal logs

You can find log output in stderr. The verbosity level for logs defaults to INFO, but you can adjust it in the config service::telemetry::logs:

service:
  telemetry:
    logs:
      level: 'debug'

You can also see logs for the Collector on a Linux systemd system using journalctl:

• All logs
• Errors only

journalctl | grep otelcol

journalctl | grep otelcol | grep Error

Types of internal observability

The OpenTelemetry Collector aims to be a model of observable service by clearly exposing its own operational metrics. Additionally, it collects host resource metrics that can help you understand if problems are caused by a different process on the same host. Specific components of the Collector can also emit their own custom telemetry. In this section, you will learn about the different types of observability emitted by the Collector itself.

Values observable with internal metrics

The Collector emits internal metrics for the following current values:

• Resource consumption, including CPU, memory, and I/O.
• Data reception rate, broken down by receiver.
• Data export rate, broken down by exporters.
• Data drop rate due to throttling, broken down by data type.
• Data drop rate due to invalid data received, broken down by data type.
• Throttling state, including Not Throttled, Throttled by Downstream, and Internally Saturated.
• Incoming connection count, broken down by receiver.
• Incoming connection rate showing new connections per second, broken down by receiver.
• In-memory queue size in bytes and in units.
• Persistent queue size.
• End-to-end latency from receiver input to exporter output.
• Latency broken down by pipeline elements, including exporter network roundtrip latency for request/response protocols.

Rate values are averages over 10 second periods, measured in bytes/sec or units/sec (for example, spans/sec).

The Collector also emits internal metrics for these cumulative values:

• Total received data, broken down by receivers.
• Total exported data, broken down by exporters.
• Total dropped data due to throttling, broken down by data type.
• Total dropped data due to invalid data received, broken down by data type.
• Total incoming connection count, broken down by receiver.
• Uptime since start.

Lists of internal metrics

The following tables group each internal metric by level of verbosity: basic, normal, and detailed. Each metric is identified by name and description and categorized by instrumentation type.

basic-level metrics

Additional normal-level metrics

Additional detailed-level metrics

Events observable with internal logs

The Collector logs the following internal events:

• A Collector instance starts or stops.
• Data dropping begins due to throttling for a specified reason, such as local saturation, downstream saturation, downstream unavailable, etc.
• Data dropping due to throttling stops.
• Data dropping begins due to invalid data. A sample of the invalid data is included.
• Data dropping due to invalid data stops.
• A crash is detected, differentiated from a clean stop. Crash data is included if available.