This page will walk you through the fastest way to get started monitoring your Kubernetes cluster using OpenTelemetry. It will focus on collecting metrics and logs for Kubernetes clusters, nodes, pods, and containers, as well as enabling the cluster to support services emitting OTLP data.
If you’re looking to see OpenTelemetry in action with Kubernetes, the best place to start is the OpenTelemetry Demo. The demo is intended to illustrate the implementation of OpenTelemetry, but it is not intended to be an example of how to monitor Kubernetes itself. Once you finish with this walkthrough, it can be a fun experiment to install the demo and see how all the monitoring responds to an active workload.
If you’re looking to start migrating from Prometheus to OpenTelemetry, or if you’re interested in using the OpenTelemetry Collector to collect Prometheus metrics, see Prometheus Receiver.
Kubernetes exposes a lot of important telemetry in many different ways. It has logs, events, metrics for many different objects, and the data generated by its workloads.
To collect all of this data we’ll use the OpenTelemetry Collector. The collector has many different tools at its disposal which allow it to efficiently collect all this data and enhance it in meaningful ways.
To collect all the data, we’ll need two installations of the collector, one as a Daemonset and one as a Deployment. The Daemonset installation of the collector will be used to collect telemetry emitted by services, logs, and metrics for nodes, pods, and containers. The deployment installation of the collector will be used to collect metrics for the cluster and events.
To install the collector, we’ll use the OpenTelemetry Collector Helm chart, which comes with a few configuration options that will make configure the collector easier. If you’re unfamiliar with Helm, check out the Helm project site. If you’re interested in using a Kubernetes operator, see OpenTelemetry Operator, but this guide will focus on the Helm chart.
This guide will assume the use of a Kind cluster, but you’re free to use any Kubernetes cluster that you deem fit.
Assuming you already have Kind installed, create a new kind cluster:
kind create cluster
Assuming you already have Helm installed, add the OpenTelemetry Collector Helm chart so it can be installed later.
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
The first step to collecting Kubernetes telemetry is to deploy a daemonset instance of the OpenTelemetry Collector to gather telemetry related to nodes and workloads running on those node. A daemonset is used to guarantee that this instance of the collector is installed on all nodes. Each instance of the collector in the daemonset will collect data only from the node on which it is running.
This instance of the collector will use the following components:
- OTLP Receiver: to collect application traces, metrics and logs.
- Kubernetes Attributes Processor: to add Kubernetes metadata to incoming application telemetry.
- Kubeletstats Receiver: to pull node, pod, and container metrics from the API server on a kubelet.
- Filelog Receiver: to collect Kubernetes logs and application logs written to stdout/stderr.
Let’s break these down.
The OTLP Receiver is the best solution for collecting traces, metrics, and logs in the OTLP format. If you are emitting application telemetry in another format, there is a good chance that the Collector has a receiver for it, but for this tutorial we’ll assume the telemetry is formatted in OTLP.
Although not a requirement, it is a common practice for applications running on
a node to emit their traces, metrics, and logs to a collector running on the
same node. This keeps network interactions simple and allows easy correlation of
Kubernetes metadata using the
Kubernetes Attributes Processor
The Kubernetes Attributes Processor is a highly recommended component in any collector receive telemetry from Kubernetes pods. This processor automatically discovers Kubernetes pods, extracts their metadata such as pod name or node name, and adds the extracted metadata to spans, metrics, and logs as resource attributes. Because it adds Kubernetes context to your telemetry, the Kubernetes Attributes Processor lets you correlate your application’s traces, metrics, and logs signals with your Kubernetes telemetry, such as pod metrics and traces.
The Kubeletstats Receiver is the receiver that gathers metrics about the node. It will gather metrics like container memory usage, pod cpu usage, and node network errors. All of the telemetry includes Kubernetes metadata like pod name or node name. Since we’re using the Kubernetes Attributes Processor, we’ll be able to correlate our application traces, metrics, and logs with the metrics produced by the Kubeletstats Receiver.
The Filelog Receiver will collect
logs written to stdout/stderr by tailing the logs Kubernetes writes to
/var/log/pods/*/*/*.log. Like most log tailers, the filelog receiver provides
a robust set of actions that allow you to parse the file however you need.
Someday you may need to configure a Filelog Receiver on your own, but for this walkthrough the OpenTelemetry Helm Chart will handle all the complex configuration for you. In addition, it will extract useful Kubernetes metadata based on the file name. Since we’re using the Kubernetes Attributes Processor, we’ll be able to correlate the application traces, metrics, and logs with the logs produced by the Filelog Receiver.
The OpenTelemetry Collector Helm chart makes configuring all of these components in a daemonset installation of the collector easy. It will also take care of all of the Kubernetes-specific details, such as RBAC, mounts and host ports.
One caveat - the chart doesn’t send the data to any backend by default. If you want to actually use your data in your favorite backend you’ll need to configure an exporter yourself.
values.yaml is what we’ll use
mode: daemonset presets: # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines kubernetesAttributes: enabled: true # enables the kubeletstatsreceiver and adds it to the metrics pipelines kubeletMetrics: enabled: true # Enables the filelogreceiver and adds it to the logs pipelines logsCollection: enabled: true ## The chart only includes the loggingexporter by default ## If you want to send your data somewhere you need to ## configure an exporter, such as the otlpexporter # config: # exporters: # otlp: # endpoint: "<SOME BACKEND>" # service: # pipelines: # traces: # exporters: [ otlp ] # metrics: # exporters: [ otlp ] # logs: # exporters: [ otlp ]
To use this
values.yaml with the chart, save it to your preferred file
location and then run the following command to install the chart
helm install otel-collector open-telemetry/opentelemetry-collector --values <path where you saved the chart>
You should now have a daemonset installation of the OpenTelemetry Collector running in your cluster collecting telemetry from each node!
The next step to collecting Kubernetes telemetry is to deploy a deployment instance of the Collector to gather telemetry related to the cluster as a whole. A deployment with exactly one replica ensures that we don’t produce duplicate data.
This instance of the Collector will use the following components:
- Kubernetes Cluster Receiver: to collect cluster-level metrics and entity events.
- Kubernetes Objects Receiver: to collect objects, such as events, from the Kubernetes API server.
Let’s break these down.
Kubernetes Cluster Receiver
The Kubernetes Cluster Receiver is the Collector’s solution for collecting metrics about the state of the cluster as a whole. This receiver can gather metrics about node conditions, pod phases, container restarts, available and desired deployments, and more.
Kubernetes Objects Receiver
The Kubernetes Objects Receiver is the Collector’s solution for collecting Kubernetes objects as logs. Although any object can be collected, a common and important use case is to collect Kubernetes events.
The OpenTelemetry Collector Helm chart streamlines the configuration for all of these components in a deployment installation of the Collector. It will also take care of all of the Kubernetes-specific details, such as RBAC and mounts.
One caveat - the chart doesn’t send the data to any backend by default. If you want to actually use your data in your preferred backend, you’ll need to configure an exporter yourself.
values.yaml is what we’ll use:
mode: deployment # We only want one of these collectors - any more and we'd produce duplicate data replicaCount: 1 presets: # enables the k8sclusterreceiver and adds it to the metrics pipelines clusterMetrics: enabled: true # enables the k8sobjectsreceiver to collect events only and adds it to the logs pipelines kubernetesEvents: enabled: true ## The chart only includes the loggingexporter by default ## If you want to send your data somewhere you need to ## configure an exporter, such as the otlpexporter # config: # exporters: # otlp: # endpoint: "<SOME BACKEND>" # service: # pipelines: # traces: # exporters: [ otlp ] # metrics: # exporters: [ otlp ] # logs: # exporters: [ otlp ]
To use this
values.yaml with the chart, save it to your preferred file
location and then run the following command to install the chart:
helm install otel-collector-cluster open-telemetry/opentelemetry-collector --values <path where you saved the chart>
You should now have a deployment installation of the collector running in your cluster that collects cluster metrics and events!