Internal telemetry
You can monitor the health of any OpenTelemetry Collector instance by checking its own internal telemetry. Read on to learn how to configure this telemetry to help you troubleshoot Collector issues.
Activate internal telemetry in the Collector
By default, the Collector exposes its own telemetry in two ways:
- Internal metrics are exposed using a Prometheus
interface which defaults to port
8888
. - Logs are emitted to
stderr
by default.
Configure internal metrics
You can configure how internal metrics are generated and exposed by the
Collector. By default, the Collector generates basic metrics about itself and
exposes them for scraping at http://127.0.0.1:8888/metrics
. You can expose the
endpoint to one specific or all network interfaces when needed. For
containerized environments, you might want to expose this port on a public
interface.
Set the address in the config service::telemetry::metrics
:
service:
telemetry:
metrics:
address: '0.0.0.0:8888'
You can enhance the metrics telemetry level using the level
field. The
following is a list of all possible values and their explanations.
none
indicates that no telemetry data should be collected.basic
is the recommended value and covers the basics of the service telemetry.normal
adds other indicators on top of basic.detailed
adds dimensions and views to the previous levels.
For example:
service:
telemetry:
metrics:
level: detailed
address: ':8888'
The Collector can also be configured to scrape its own metrics and send them through configured pipelines. For example:
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otelcol'
scrape_interval: 10s
static_configs:
- targets: ['0.0.0.0:8888']
metric_relabel_configs:
- source_labels: [__name__]
regex: '.*grpc_io.*'
action: drop
exporters:
debug:
service:
pipelines:
metrics:
receivers: [prometheus]
exporters: [debug]
Caution
Self-monitoring is a risky practice. If an issue arises, the source of the problem is unclear and the telemetry is unreliable.Configure internal logs
You can find log output in stderr
. The verbosity level for logs defaults to
INFO
, but you can adjust it in the config service::telemetry::logs
:
service:
telemetry:
logs:
level: 'debug'
You can also see logs for the Collector on a Linux systemd system using
journalctl
:
journalctl | grep otelcol
journalctl | grep otelcol | grep Error