Configure OBI service discovery

The OTEL_EBPF_AUTO_TARGET_EXE and OTEL_EBPF_OPEN_PORT are environment variables that make it easier to configure OBI to instrument a single service or a group of related services.

In some scenarios, OBI instruments many services. For example, as a Kubernetes DaemonSet that instruments all the services in a node. The discovery YAML section lets you specify more granular selection criteria for the services OBI can instrument.

Discovery services

You can override the service name, namespace, and other configurations per service type.

Open ports

Selects the process to instrument by the port it has open (listens to). This property accepts a comma-separated list of ports, for example 80, and port ranges, for example 8000-8999. If the executable matches only one of the ports in the list, OBI considers it a match.

For example, specifying the following property:

discovery:
  instrument:
    - open_ports: 80,443,8000-8999

OBI selects any executable that opens port 80, 443, or any of the ports between 8000 and 8999 included.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

If an executable opens multiple ports, you only need to specify one of those ports for OBI to instrument all the HTTP/S and gRPC requests on all application ports. Currently, you can’t restrict the instrumentation only to the methods exposed through a specific port.

Executable path

Selects the processes to instrument by their executable name path. This property accepts a glob to match against the full executable command line, including the directory where the executable resides on the file system.

OBI tries to instrument all the processes with an executable path matching this property. For example, setting exe_path: * makes OBI try to instrument all the executables in the host.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

Containers only

Selects processes to instrument which are running in an OCI container. To perform this check, OBI inspects the process network namespace and matches it against its own network namespace. If OBI doesn’t have enough permissions to perform the network namespace inspection, it ignores this option.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s namespace

This selector property limits the instrumentation to the applications running in the Kubernetes Namespaces with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s Pod name

This selector property limits the instrumentation to the applications running in the Kubernetes Pods with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s deployment name

This selector property limits the instrumentation to the applications running in the Kubernetes Deployments with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s replicaset name

This selector property limits the instrumentation to the applications running in the Kubernetes ReplicaSets with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s statefulset name

This selector property limits the instrumentation to the applications running in the Kubernetes StatefulSets with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s daemonset name

This selector property limits the instrumentation to the applications running in the Kubernetes DaemonSet with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s owner name

This selector property limits the instrumentation to the applications running in the Pods owned by a Deployment, ReplicaSet, DaemonSet, or StatefulSet with a name matching the provided glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

K8s Pod labels

This selector property limits the instrumentation to the applications running in the Pods with labels matching the provided value as glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

For example:

discovery:
  instrument:
    - k8s_namespace: frontend
      k8s_pod_labels:
        instrument: obi

The preceding example discovers all Pods in the frontend namespace that have a label instrument with a value that matches the glob obi.

K8s Pod annotations

This selector property limits the instrumentation to the applications running in the Pods with annotations matching the provided value as glob.

If you specify other selectors in the same instrument entry, the processes must match all the selector properties.

For example:

discovery:
  instrument:
    - k8s_namespace: backend
      k8s_pod_annotations:
        obi.instrument: 'true'

The preceding example discovers all Pods in the backend namespace that have an annotation obi.instrument with a value that matches the glob true.

Exclude services from instrumentation

The exclude_instrument section lets you specify selection criteria for excluding services from being instrumented. It follows the same definition format as described in the discovery services section of this document.

This option helps you avoid instrumenting services typically found in observability environments. For example, use this option to exclude instrumenting Prometheus.

Example: Exclude specific namespaces

discovery:
  instrument:
    - k8s_namespace: '*' # Instrument all namespaces
  exclude_instrument:
    - k8s_namespace: development # Except development namespace
    - k8s_namespace: staging # And staging namespace

Example: Exclude services by labels

discovery:
  rules:
    - match:
        k8s_namespace: production
      exclude:
        k8s_pod_labels:
          skip-instrumentation: 'true'

In this example, skip-instrumentation is a user-defined Kubernetes pod label. You can use any custom label key and value for matching or exclusion based on your organization’s labeling conventions.

Example: Exclude specific executables

discovery:
  rules:
    - match:
        open_ports: 80,443,8080
      exclude:
        exe_path: '*prometheus*'
        exe_path: '*grafana*'

Default exclude services from instrumentation

The default_exclude_instrument section disables instrumentation of OBI itself (self-instrumentation), the OpenTelemetry Collector and other observability components. It also disables instrumentation of various Kubernetes system namespaces to reduce the overall cost of metric generation. The following section contains all excluded components:

• Excluded services by exe_path: *beyla, *alloy, *obi, *otelcol, *otelcol-contrib, *otelcol-contrib[!/]*.
• Excluded services by k8s_namespace: kube-system, kube-node-lease, local-path-storage, grafana-alloy, cert-manager, monitoring, gke-connect, gke-gmp-system, gke-managed-cim, gke-managed-filestorecsi, gke-managed-metrics-server, gke-managed-system, gke-system, gke-managed-volumepopulator, gatekeeper-system.

Change this option to allow OBI to instrument itself or some of the other excluded components.

Note: to enable such self-instrumentation, you still need to include them in the instrument section, or these components need to be a part of a encompassing inclusion criteria.

Example: Enable instrumentation for a default-excluded namespace

To enable instrumentation for a namespace that is excluded by default (such as monitoring), you must both remove it from the default exclusions and add it to the instrument section.

The following example enables instrumentation for the monitoring namespace:

discovery:
  # Include the monitoring namespace in instrumentation
  instrument:
    - k8s_namespace: monitoring

  # Override default exclusions to remove monitoring namespace
  # This list keeps other default exclusions but removes monitoring
  default_exclude_instrument:
    - exe_path: '{*beyla,*alloy,*obi,*otelcol,*otelcol-contrib,*otelcol-contrib[!/]*}'
    - k8s_namespace: kube-system
    - k8s_namespace: kube-node-lease
    - k8s_namespace: local-path-storage
    - k8s_namespace: grafana-alloy
    - k8s_namespace: cert-manager
    # monitoring namespace removed from this list
    - k8s_namespace: gke-connect
    - k8s_namespace: gke-gmp-system
    - k8s_namespace: gke-managed-cim
    - k8s_namespace: gke-managed-filestorecsi
    - k8s_namespace: gke-managed-metrics-server
    - k8s_namespace: gke-managed-system
    - k8s_namespace: gke-system
    - k8s_namespace: gke-managed-volumepopulator
    - k8s_namespace: gatekeeper-system

Example: Disable all default exclusions

To disable all default exclusions and allow OBI to instrument any matched service (including itself and other observability components), set the default_exclude_instrument to an empty list:

discovery:
  instrument:
    - k8s_namespace: '*' # or specific namespaces/selectors

  # Empty list disables all default exclusions
  default_exclude_instrument: []

Skip go specific tracers

The skip_go_specific_tracers option disables the detection of Go specifics when the eBPF tracer inspects executables to be instrumented. The tracer falls back to using generic instrumentation, which is generally less efficient.

Exclude OTel instrumented services

The exclude_otel_instrumented_services option disables OBI instrumentation of services already instrumented with OpenTelemetry. Since OBI is often deployed to monitor all services in a Kubernetes cluster, monitoring already instrumented services can lead to duplicate telemetry data, unless you carefully craft the instrumentation selection (or exclusion) criteria. To avoid unnecessary configuration overhead, OBI monitors for OpenTelemetry SDK calls to publish metrics and traces, and automatically turns off instrumentation of services that publish their own telemetry data. Turn this option off if your application-generated telemetry data doesn’t conflict with the OBI generated metrics and traces.

Override service name and namespace

If you export instrumentation data via OpenTelemetry or Prometheus, OBI follows the service name conventions from the OpenTelemetry operator to improve interoperability with other instrumentation solutions.

OBI uses the following criteria in this order to automatically set the service name and namespace:

1. Resource attributes set via OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables of the instrumented process or container.
2. In Kubernetes, resource attributes set via the following Pod annotations:
   • resource.opentelemetry.io/service.name
   • resource.opentelemetry.io/service.namespace
3. In Kubernetes, resource attributes set via the following Pod labels:
   • app.kubernetes.io/name sets the service name
   • app.kubernetes.io/part-of sets the service namespace
4. In Kubernetes, resource attributes calculated from the Pod owner’s metadata, in the following order (according to their availability):
   • k8s.deployment.name
   • k8s.replicaset.name
   • k8s.statefulset.name
   • k8s.daemonset.name
   • k8s.cronjob.name
   • k8s.job.name
   • k8s.pod.name
   • k8s.container.name
5. The executable name of the instrumented process.

You can override the Kubernetes labels from the previous bullet 3 via configuration.

In YAML:

attributes:
  kubernetes:
    resource_labels:
      service.name:
        # gets service name from the first existing Pod label
        - override-svc-name
        - app.kubernetes.io/name
      service.namespace:
        # gets service namespace from the first existing Pod label
        - override-svc-ns
        - app.kubernetes.io/part-of

They accept a comma-separated list of annotation and label names.

Enhanced service name lookup (v0.5.0+)

Starting with v0.5.0, OBI includes enhanced service name resolution that provides more accurate service identification, especially in dynamic and distributed environments.

Improvements:

1. DNS-based resolution: OBI can resolve service names using DNS queries, providing better alignment with actual service discovery mechanisms used by applications

2. Metadata enrichment: Enhanced lookup from multiple metadata sources including container runtimes and orchestration platforms

3. Connection tracking: Better tracking of service-to-service communication by resolving both client and server identities

How it works:

When OBI detects network communication between services, it attempts to resolve service names using the available configured sources, in this order of priority:

1. Kubernetes metadata lookup: Check Pod and owner metadata from the Kubernetes API (enabled by default)
2. Reverse DNS from eBPF: Use DNS responses captured at the kernel level (optional, requires OTEL_EBPF_NAME_RESOLVER_SOURCES=rdns)
3. Standard DNS reverse lookup: Perform reverse DNS queries for IP addresses (optional, requires OTEL_EBPF_NAME_RESOLVER_SOURCES=dns)

Local service names follow this priority hierarchy:

1. OTEL_SERVICE_NAME environment variable (highest priority)
2. OTEL_RESOURCE_ATTRIBUTES environment variable (service.name key)
3. Service name annotations (resource.opentelemetry.io/service.name)
4. Service name labels (e.g., app.kubernetes.io/name)
5. Kubernetes Pod owner name (e.g., Deployment name) (lowest priority)

This enhancement is particularly valuable for:

• Service mesh environments: Where DNS is used for service routing
• Kubernetes clusters: Improved correlation with Service resources
• Microservices architectures: Better service graph visualization with accurate service names

Configuration:

Kubernetes metadata-based service lookup is enabled by default. To enable DNS-based resolution methods, configure the name resolver sources:

# Enable both Kubernetes metadata and standard DNS reverse lookups
export OTEL_EBPF_NAME_RESOLVER_SOURCES=k8s,dns

# Or enable eBPF-captured DNS lookups (requires DNS event capture)
export OTEL_EBPF_NAME_RESOLVER_SOURCES=k8s,rdns

# Enable all resolver sources
export OTEL_EBPF_NAME_RESOLVER_SOURCES=k8s,dns,rdns

# Optional: Adjust cache size (default: 1024)
export OTEL_EBPF_NAME_RESOLVER_CACHE_LEN=2048

# Optional: Adjust cache time-to-live (default: 5 minutes)
export OTEL_EBPF_NAME_RESOLVER_CACHE_TTL=10m

In Kubernetes environments, ensure:

• Network policies allow DNS queries (if using DNS-based resolution)
• CoreDNS or equivalent DNS service is running
• If using RDNS, eBPF programs can capture DNS events

Benefits:

• More accurate service graphs: Service-to-service communication shows real service names instead of IPs
• Better trace correlation: Traces show service names that match your service catalog
• Easier troubleshooting: Identify which services are communicating without manual IP lookups

Example:

Without enhanced lookup, you might see:

service.name: "10.0.1.42"
peer.service: "10.0.2.15"

With enhanced lookup enabled:

service.name: "frontend"
peer.service: "backend-api"