# Managed Telemetry Platforms for Kubernetes Workloads LLMS index: [llms.txt](/llms.txt) --- ## Summary This blueprint provides strategic guidance for organizations aiming to follow Platform Engineering practices to ease adoption of OpenTelemetry tooling and standards across their engineering teams. This includes usage of SDKs, instrumentation libraries, configuration patterns, and Collector architectures to provide centrally managed telemetry platforms paired with self-serve tooling designed to be consumed โ€œas-a-serviceโ€. It is aimed at organizations operating in cloud and Kubernetes environments, wishing to provide a consistent, scalable, and governed telemetry platform across workloads owned by highly autonomous product teams, achieving the following outcomes: - Consistent SDK and instrumentation configuration, improving time-to-value by facilitating adoption of organization-specific standards across all workloads, reducing cognitive load for product teams. - Cohesive semantic conventions that allow for telemetry correlation between signals, applications, and domains, from client-side to infrastructure, providing high-quality telemetry to be utilized by manual or automatic analysis. - Elimination of Collector configuration sprawl, reducing operational toil through consolidation of telemetry pipelines. - Resilient, scalable, and reliable ingest pipelines for all telemetry signals avoiding single points of failure. - Centralized telemetry governance and data optimization to reduce operational costs and carbon emissions by minimizing storage, network transfer, and compute requirements of telemetry processing. - Future-proof telemetry pipelines that shield product teams from changes in the underlying observability backend, enabling data migrations or multi-vendor strategies with minimal changes to application instrumentation or collection infrastructure. ## Background As organizations increase the rate of adoption of cloud native standards and modern software delivery practices, they often adopt federated models where teams, or business units, operate with high autonomy and are made responsible for the full Software Development Lifecycle (SDLC) of their systems, from designing to operating software in production. This โ€œyou build, you run itโ€ model is designed to empower product delivery, however it can inadvertently create fragmented service management practices and cluttered observability landscapes that fail to reap the benefits of OpenTelemetry and modern observability tooling. Product teams prioritize feature delivery over Non-Functional Requirements (NFRs), like telemetry instrumentation, and see these tasks as a burden on their delivery goals. To address this, organizations are widely adopting cloud native Platform Engineering models to reduce cognitive load and abstract complexity. By treating observability as a curated internal [platform product][1], organizations can offer a paved road, or a golden path, that ensures high-quality, contextual observability with minimal friction, while allowing teams to remain focused on instrumenting domain-specific concepts impossible to capture in out-of-the-box telemetry. ## Common challenges Organizations operating in these federated, distributed environments typically face a distinct set of challenges that hinder effective observability and cloud native maturity. ### 1. Inconsistent configuration and low adoption of organization standards {#challenge-1} In environments where product teams operate with autonomy, distinct ways of configuring individual applications and services for observability may coexist, while still operating under a shared compute layer. This includes setting up OpenTelemetry SDKs for applications, configuring instrumentation packages and libraries, or deciding how to propagate observability context from/to their dependencies. Organizations may have a set of documented engineering standards they wish all engineers to follow, but they often rely on manual implementation of these standards by each individual team, including configuration and code-level changes. This is often treated by teams as an afterthought, not part of the software design process, and focused on a particular application without considering the overall distributed system in a holistic way. ```mermaid --- title: "Figure 1: Silos due to lack of consistent semantic conventions and context propagation." config: flowchart: curve: basis --- flowchart LR subgraph K8sNode["Kubernetes Node"] direction TB AppA["๐Ÿ“ฆ App A"]:::node AppB["๐Ÿ“ฆ App B"]:::node Collector["๐Ÿ”€ Collector"]:::node end subgraph TracesDB["๐Ÿงต๏ธTraces Backend"] direction LR TraceX[("๐Ÿงต Trace X")]:::node TraceY[("๐Ÿงต Trace Y")]:::node end subgraph MetricsDB["๐Ÿ“ˆ Metrics Backend"] Metrics[("๐Ÿ“ˆ Container Metrics")]:::node end User["๐Ÿ‘ค User"]:::node User L_User_AppA@-- Inbound request --> AppA AppA L_AppA_AppB@-. "Dependency
(broken trace context)" .-x AppB TracesDB L_TracesDB_MetricsDB@x-. "Broken correlation
(missing k8s.* attributes)" .-x MetricsDB AppA L_AppA_TraceX@== Spans ==> TraceX AppB L_AppB_TraceY@== Spans ==> TraceY Collector L_Collector_MetricsDB@== "Metrics
(k8s.pod.name=app-...)" ==> MetricsDB classDef node fill:#ffffff, stroke:#818cf8, stroke-width:2px, color:#6b7280 style K8sNode fill:#eef2ff, stroke:#818cf8, stroke-width:2px, color:#818cf8 style TracesDB fill:#eef2ff, stroke:#818cf8, stroke-width:2px, color:#818cf8 style MetricsDB fill:#eef2ff, stroke:#818cf8, stroke-width:2px, color:#818cf8 linkStyle 0 stroke:#7dd3fc, fill:none, stroke-width:3px linkStyle 1 stroke:#fca5a5, fill:none, stroke-width:3px linkStyle 2 stroke:#fca5a5, fill:none, stroke-width:3px linkStyle 3,4,5 stroke:#a3e635, fill:none, stroke-width:3px L_User_AppA@{ animation: slow } L_AppA_AppB@{ animation: slow } L_TracesDB_MetricsDB@{ animation: slow } L_AppA_TraceX@{ animation: fast } L_AppB_TraceY@{ animation: fast } L_Collector_MetricsDB@{ animation: fast } ``` This leads to: - **Inconsistent [Semantic Conventions][2]:** Telemetry lacks common [Resource][3] attributes (e.g., `service.version`, `k8s.cluster.name`, `example.cost.center`), breaking correlation across different signals, applications, and system layers, and limiting the usefulness of observability data for automatic analysis. - **Context silos:** Without consistent [context propagation][4] (e.g. W3C Trace Context) baked into every SDK, distributed traces break at service boundaries, making it impossible to tie backend performance regressions to customer-facing business impact. - **SDK version fragmentation**: Widely different versions of OpenTelemetry SDKs running in production, introducing maintenance and security concerns. - **High cognitive load:** Developers must manually configure SDKs and instrumentation packages for every new service, increasing toil and the risk of misconfiguration. - **Reduced velocity:** Any change in engineering standards related to telemetry instrumentation, or any change in the underlying observability backend such as data or protocol migrations, creates friction and reduces overall velocity in the organization as technology adoption is ultimately hampered by manual implementation. ### 2. Collector configuration sprawl across clusters {#challenge-2} As OpenTelemetry adoption scales, and organizations deploy across tens or hundreds of Kubernetes clusters, managing individual OpenTelemetry Collector configurations manually across these environments creates a maintenance burden. This is especially challenging in organizations where different Collector deployments are handled by different teams. This leads to: - **Configuration drift:** Different clusters end up with varying parsing rules, filtering logic, and endpoint configurations, causing unpredictable telemetry behavior. - **Lack of separation of concerns:** There is no clear distinction between the different types of telemetry processing done at different layers of Collectors (e.g. where to transform, where to sample) which can lead to inconsistent or incomplete data. - **Manual toil:** Platform teams spend an excessive amount of time on repetitive configuration tasks and manual updates, rather than building scalable solutions. - **Unreliable rollouts:** Without version-controlled, auditable deployments, applying a fix or a new configuration across the fleet becomes highly risky and error-prone. ### 3. Data pipelines not optimized for observability data requirements {#challenge-3} In some legacy instrumentation models, applications or instrumentation agents often export telemetry directly to telemetry backends. This model lacks a way to process and transform telemetry between the application and the backend, reducing data sovereignty. It can also add extra complexity if the backend is a third-party vendor, or any endpoint requiring public traffic or authentication. Managing credentials across thousands of applications can be challenging, and sporadic network connectivity issues between a single exporter and a public endpoint can create service interruptions. Conversely, in environments where data pipelines are centralized, data requirements for telemetry data are often mixed with those for other types of data. This can lead to solutions that are optimized for completeness (e.g. audit logging, financial data reporting) rather than context-aware transformations and low-latency processing. This increases the time between data emission and actionable insights, necessary to maintain reliable operations. This leads to: - **Single points of failure:** Direct egress from hundreds of individual applications to the internet strips the organization of central network governance and load balanced exports. - **Latency and operational value:** Ultimately, stale observability data is almost as good as no observability data. Overly complex logging pipelines can introduce significant lag, rendering real-time operational alerts useless during a major incident. - **Lack of central control:** Platform teams cannot easily reroute data, change vendors, or apply global network policies when configurations are deeply embedded within individual applications. > [!NOTE] Help wanted > > The scope of this blueprint is defined by common challenges faced by platform > teams to provide pipelines optimized for low-latency and efficient resource > usage. In certain scenarios, like those requiring audit logging or business > reporting, balancing completeness or durability guarantees is critical. These > challenges are out of scope for this blueprint and may be targeted in a > separate blueprint. See our [guidance][5] if you are interested in > contributing. ### 4. Lack of telemetry governance and low ROI {#challenge-4} Without centralized governance and measurable adoption of observability standards, autonomous teams may generate vast amounts of low-value data, reducing the signal-to-noise ratio. OpenTelemetry signals are often not used for their intended purpose, ultimately making their production harder to maintain for platform teams (e.g. having to ensure fast and accurate querying over days or weeks of individual logs simply to compute the number of requests for a given service). As traffic grows and telemetry volume increases, teams in charge of observability have no scalable way of ensuring data quality across their landscape. This leads to: - **Unattributed data quality issues**: As consistent semantic conventions are not enforced, platform teams cannot associate telemetry spend or data quality with specific business units or engineering teams. - **Inefficient data types:** Organizations incur heavy storage and indexing costs for raw logs or other signals when not used for their intended purpose, while reducing the overall quality of the insights extracted from observability data. - **Unnecessary costs**: Increasing costs associated with data storage, network egress, or ingest into a particular backend, incurred from data that does not always improve the insights one may require to operate systems reliably. - **Carbon emissions**: Processing of low-value data can be detrimental to achieving green software targets, including scope 3 emissions from embedded carbon present in the devices necessary for fast retrieval of observability data, e.g. SSDs. - **High cognitive load**: Large data volumes not only result in unnecessary costs, they may also increase noise, forcing users and agents to filter through low-quality data to find relevant telemetry. > [!NOTE] Help wanted > > Multi-tenant environments often deal with strict compliance requirements > (GDPR, HIPAA, PCI) and security concerns such as authentication and encryption > between pipeline layers. These challenges are out of scope for this blueprint > and may be targeted in a separate blueprint. See our [guidance][5] if you are > interested in contributing. ### 5. Low observability and operational efficiency of SDKs and data pipelines {#challenge-5} One of the challenges of operating OpenTelemetry SDKs and Collectors in production is identifying if, and when, the default configuration applied for aspects regarding queuing, retrying, or batching of telemetry data is not optimal for a particular environment. OpenTelemetryโ€™s sensible defaults may not be suitable either to implement a leaner approach on resource utilization, or higher reliability guarantees. This may depend on architectural patterns in use, e.g. exporting to a local cluster endpoint may require less buffering than a public internet endpoint. This leads to: - **Silent data drops and export failures:** Data exports suffer from failures to export to backends, or Collectors, ultimately dropping data, without those errors being observed or alerted on. - **Unnecessary resource utilization:** Operators overprovision resources on SDKs and Collectors, increasing resource utilization, potentially affecting performance overhead and cost. ## General Guidelines ### 1. Centralize default, extensible configuration for SDKs and instrumentation packages {#guideline-1} **Challenges addressed**: [1](#challenge-1), [4](#challenge-4) | **Implementation actions**: [1](#action-1), [2](#action-2) We recommend teams in charge of observability tooling maintain a set of resources (see [Action 1](#action-1)) to provide basic, out-of-the-box configuration for [SDKs][6] and [instrumentation libraries][7]. The aim is for applications deployed in a Kubernetes cluster to emit a basic level of telemetry, and to propagate context from and to dependencies, with minimal input required from application owners, e.g. at most adding an annotation, or calling a shared internal library. Platform teams should ensure that this base configuration remains extensible, allowing application owners to control different aspects of the SDK (e.g. buffer sizes, exporter retries) and instrumentation libraries, to meet the requirements specific to their applications. By implementing this guideline, organizations can expect to achieve: - **Cohesive organization standards:** Specific organization standards (e.g. resource attributes, exporter endpoint, etc) are applied automatically across the stack. - **Consistent context propagation:** Trace Context is propagated between services using compatible propagator configurations. - **Lower cognitive load:** Application owners can abstract themselves from lower-level configuration, such as that related to setting up the OpenTelemetry SDK. - **Easier maintenance:** Effort to adopt engineering standards and best practices in observability is minimized, as new standards can be rolled out via version bumps of internal tooling. ### 2. Establish shared ownership for telemetry production {#guideline-2} **Challenges addressed**: [4](#challenge-4), [5](#challenge-5) | **Implementation actions**: [1](#action-1), [2](#action-2), [5](#action-5) To balance governance and autonomy, platform teams operating in the environments described in this blueprint should aim to โ€œshift leftโ€ on instrumentation, ensuring that application owners have full control and ownership of the telemetry emitted by their applications. Default configurations mentioned in [Guideline 1](#guideline-1) should ensure that provenance of data is guaranteed, including technical attributes (e.g. cluster, deployment, pod) and organizational information (e.g. team, business domain), with the aim of making it trivial to identify the source of telemetry, and the owning team. OpenTelemetry [client design principles][8] establish a clear separation between the API, being a no-op implementation by default, and the SDK, which provides an implementation for that API when registered. This provides a clear separation of responsibilities, and allows application owners to rely solely on the OpenTelemetry API, focusing their efforts on enriching telemetry with domain-specific context (e.g., business transactions, user IDs) that is impossible to capture generically, while relying on the provided default configuration to produce telemetry out of the box. ```mermaid --- title: "Figure 2: Shared ownership model between platform teams and application owners." config: flowchart: curve: basis --- flowchart TD subgraph User["Application Ownership"] Application["Application"]:::node Config["โš™๏ธ
Config"]:::node end subgraph Platform["Platform Ownership"] Collector[("๐Ÿ”€๏ธ
Collector
Pipelines")]:::node BaseConfig["โš™๏ธ
Base Config"]:::node end subgraph Application["Application"] AppCode["๐Ÿ’ผ
Biz Logic"]:::node ThirdParty["๐Ÿ‘ฝ
3rd-Party Libs"]:::node subgraph OTel["OpenTelemetry"] InstLibs["๐Ÿ“ฆ
Instrumentation"]:::node OTelSDK["๐Ÿ“ฆ
OTel SDK"]:::node OTelAPI["๐Ÿ“ฆ
OTel API"]:::node end end Sink[("๐Ÿ—„๏ธ Observability Backend")]:::node AppCode L_AppCode_API@-- Uses --> OTelAPI ThirdParty L_ThirdParty_API@-- Uses --> OTelAPI InstLibs L_InstLibs_API@-- Uses --> OTelAPI OTelAPI L_SDK_API@-. Implemented By .-> OTelSDK Config L_Config_InstLibs@-.-> InstLibs Config L_Config_SDK@-.-> OTelSDK OTelSDK L_SDK_Collector@-- Exports --> Collector Collector L_Collector_Sink@--> Sink BaseConfig L_BaseConfig_Config@-- Extended By --> Config classDef node fill:#ffffff, stroke:#818cf8, stroke-width:2px, color:#6b7280 style User fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style Platform fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style Application fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style OTel fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 linkStyle 0,1,2 stroke:#7dd3fc, fill:none, stroke-width:3px linkStyle 3,4,5,8 stroke:#fde68a, fill:none, stroke-width:3px linkStyle 6,7 stroke:#a3e635, fill:none, stroke-width:3px L_AppCode_API@{ animation: fast } L_ThirdParty_API@{ animation: fast } L_InstLibs_API@{ animation: fast } L_SDK_API@{ animation: slow } L_Config_InstLibs@{ animation: slow } L_Config_SDK@{ animation: slow } L_SDK_Collector@{ animation: fast } L_Collector_Sink@{ animation: fast } L_BaseConfig_Config@{ animation: slow } ``` This model relies on OpenTelemetryโ€™s API design to abstract implementation details. We recommend considering direct usage of the different signal APIs and avoiding building further abstractions around them, unless these provide more value than simply hiding implementation details. When required, SDK features (e.g. [Metric Views][9], or [Span Processors][10]) can be utilized to transform telemetry at the application level (see [Guideline 4](#guideline-4)). > [!NOTE] Help wanted > > [Weaver][11] can help teams to manage organization-specific semantic > convention registries, and to measure and validate adherence to those, > ensuring instrumentation quality by design. Learn more about Weaver in [this > blogpost][12]. Semantic conventions governance is out of scope for this > blueprint and may be targeted in a future blueprint. See our [guidance][5] if > you are interested in contributing. Ultimately, application owners should remain owners of the telemetry data emitted by their applications (both manually and automatically instrumented), and be accountable for its quality and resiliency. This includes monitoring and alerting on [SDK telemetry][13], automatically configured by the platform team in languages that support it, and optimize their configuration according to their specific application needs. This involves tuning SDK components like the `BatchSpanProcessor` or the `PeriodicMetricReader` to change buffer sizes, retry queues, cardinality limits, or timeouts, as required by their telemetry volumes. By implementing this guideline, organizations can expect to achieve: - **Correlation to business outcomes:** Telemetry emitted by applications contains the necessary domain and business logic context to correlate user experience to technical components and infrastructure. - **Clear ownership and responsibilities:** Provenance of data is ensured, allowing teams to measure telemetry quality and ensure standards are adopted at scale. - **Improved usage of telemetry signals:** As application owners become more familiar with OpenTelemetry signals, guided by organization standards, their optimal usage of OpenTelemetry APIs will improve. - **Reliable telemetry production:** Monitoring internal SDK metrics provides application or platform owners with the necessary information to optimize aspects regarding queuing, retrying, or batching of telemetry data. ### 3. Maintain a set of centrally managed Collector Gateways {#guideline-3} **Challenges addressed**: [2](#challenge-2), [3](#challenge-3), [4](#challenge-4) | **Implementation actions**: [1](#action-1), [3](#action-3), [5](#action-5) We recommend that telemetry in this type of Kubernetes environment is automatically ingested into a centralized layer deployed as an OpenTelemetry [Collector Gateway.][14] The base configuration provided as part of [Guideline 1](#guideline-1) should ensure that telemetry is exported to this layer using OTLP. ```mermaid --- title: "Figure 3: General behaviour of an OpenTelemetry Collector Gateway." config: flowchart: curve: basis --- flowchart LR subgraph App["Application"] SDK["๐Ÿ“ฆ OTel SDK"]:::node end LB["โš–๏ธLoad Balancer"]:::node subgraph OTelCol["Collector Gateway"] direction TB C1["๐Ÿ”€ Collector 1"]:::node C2["๐Ÿ”€ Collector 2"]:::node C3["๐Ÿ”€ Collector 3"]:::node end Backend[("๐Ÿ—„๏ธ Backend")]:::node SDK L_SDK_LB@-- "OTLP" --> LB LB L_LB_C1@--> C1 LB L_LB_C2@--> C2 LB L_LB_C3@--> C3 C1 L_C1_Backend@--> Backend C2 L_C2_Backend@--> Backend C3 L_C3_Backend@--> Backend classDef node fill:#ffffff, stroke:#818cf8, stroke-width:2px, color:#6b7280 style App fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style OTelCol fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 linkStyle 0,1,2,3,4,5,6 stroke:#a3e635, fill:none, stroke-width:3px L_SDK_LB@{ animation: fast } L_LB_C1@{ animation: fast } L_LB_C2@{ animation: fast } L_LB_C3@{ animation: fast } L_C1_Backend@{ animation: fast } L_C2_Backend@{ animation: fast } L_C3_Backend@{ animation: fast } ``` In multi-tenant environments, multiple Collector Gateways may need to be chained to accommodate for different scenarios. For instance, multi-cluster setups with local Gateways per cluster and a global Gateway for tail-sampling (see [Guideline 4](#guideline-4)), or namespace-scoped Gateways managed by independent teams, feeding into a cluster-wide Gateway in heavily federated environments. Ideally, base SDK configuration should automatically select the most optimal Collector endpoint and any necessary credentials according to information available in the application environment (e.g. locality-based traffic routing, conditionally changing server address depending on environment name, etc). Finally, depending on organization-specific conditions, different OpenTelemetry signals may be given different non-functional requirements. For instance, due to their stable telemetry volumes and their use in critical alerts, metrics may be assigned higher reliability requirements than spans, favoring dropping data on the latter before affecting the former. To accommodate for these conditions platform teams may consider different options, including: - **Isolated Gateways per signal:** Deploy separate Gateways for logs, metrics, and spans. Isolated deployments simplify compute resource allocation and capacity planning per signal, but shared processor configuration must be duplicated across Gateways. This can be managed through external templating tools, e.g. Kapitan, Kustomize, or using multiple config [locations][61] to override each other. However, it may increase maintenance toil. - **Multiple memory limiters on a single Gateway:** Defining separate [memory_limiter][15] configurations per signal, with different thresholds. This relies on the OTLP receiver in front of a `memory_limiter` returning a retryable error code to OTLP clients (e.g. SDKs or other Collectors) when telemetry is refused, applying backpressure as required. Pipelines with lower priority can then be configured with lower memory limiter thresholds in order to apply backpressure earlier, leaving memory headroom for higher priority pipelines. Platform engineers should make use of [internal Collector telemetry][16] to ensure the reliability of the data ingested, processed, and exported by their pipelines, and optimize their configuration accordingly. This includes configuring components like the `memory_limiter`, or OTLP options like `sending_queue` or `retry_on_failure`. These metrics should be used to avoid default CPU-based autoscaling of Collector Gateways, scaling fleets based on pipeline queue depth or memory consumption, to handle sudden telemetry spikes. By implementing this guideline, organizations can expect to achieve: - **Pipelines optimized for observability data requirements:** By combining OTLP exporter and receiver configurations with load-balanced, reliable Collector pipelines, teams are able to fulfil their reliability requirements on a per-signal basis. - **Efficient use of compute resources:** Horizontally scaled, centralized Gateways utilize compute resources more efficiently than per-node DaemonSets or per-pod Sidecars in heterogeneous, multi-tenant environments. DaemonSets normally have to be over provisioned to handle variable node sizes (i.e. a single node may serve 4 or 40 application pods) and variable per-pod telemetry volume that fluctuates over time. Keeping per-node footprints small matters as teams often struggle to schedule workloads on smaller nodes. A central Gateway tier scales independently, sized to total telemetry volume. - **Consolidated Collector configuration:** As described in [Action 3](#action-3), this model allows for a consolidated deployment of Collector configuration across multiple layers, minimizing maintenance toil and reducing risk of change failure. ### 4. Efficiently aggregate, process, and sample telemetry at different layers {#guideline-4} **Challenges addressed**: [3](#challenge-3), [4](#challenge-4) | **Implementation actions**: [2](#action-2), [4](#action-4) At an application level, OpenTelemetry client design decouples instrumentation APIs and their SDK implementations. This allows instrumentation authors (including application or library owners) to use the API to [record measurements][17], [create spans][18], or [emit log records][19], without having to define how those will be aggregated in memory, processed, and ultimately exported. This decision can be deferred to the moment when [meter][20], [tracer][21], and [logger][22] providers are created as part of the SDK setup. Configuration of these aspects should be shared, with platform teams providing a basic layer of configuration, and application owners extending that configuration for their particular use cases. At a distributed system level, different [trace sampling][23] techniques may be used to efficiently store the most valuable traces in a consistent manner. See [Appendix 1](#appendix-1) for an introduction to these techniques. When trace sampling is implemented, consistent use of semantic conventions becomes crucial. Metrics provide complete (yet aggregated) views of telemetry, using [Exemplars][27] to correlate to high granularity trace spans for a given operation, which can then link to logs and other telemetry signals (e.g. profiles). Using standard semantic conventions and consistent _Resource_ attributes also empowers correlation between these signals, allowing operators to โ€œzoom inโ€ from long-term, aggregated metric streams to highly-granular, contextual traces. The following diagram provides a summary of different layers where aggregation, processing, and sampling may be configured in a tail-sampling, multi-cluster scenario. ```mermaid --- title: "Figure 4: Multi-tenant architecture with Trace ID based global load balancing and tail sampling." config: flowchart: curve: basis --- flowchart LR subgraph LocalA["Local Gateway"] direction LR LA1["๐Ÿ”€ Collector"]:::node ~~~ LA2["๐Ÿ”€ Collector"]:::node end subgraph ClusterA["Cluster A"] direction TB AppA["๐Ÿ“ฆ OTel SDK"]:::node LocalA end subgraph LocalB["Local Gateway"] direction LR LB1["๐Ÿ”€ Collector"]:::node ~~~ LB2["๐Ÿ”€ Collector"]:::node end subgraph ClusterB["Cluster B"] direction TB AppB["๐Ÿ“ฆ OTel SDK"]:::node LocalB end subgraph LB_Layer["Load Balancing Layer"] direction TD GLB1["๐Ÿ”€ Collector"]:::node ~~~ GLB2["๐Ÿ”€ Collector"]:::node ~~~ GLB3["๐Ÿ”€ Collector"]:::node end subgraph SamplingLayer["Tail Sampling Layer"] direction TD TS1["๐Ÿ”€ Collector"]:::node ~~~ TS2["๐Ÿ”€ Collector"]:::node ~~~ TS3["๐Ÿ”€ Collector"]:::node end subgraph GlobalTier["Unified Global Gateway"] direction LR LB_Layer SamplingLayer end ObsBackend[("๐Ÿ—„๏ธ Observability Backend")]:::node AppA L_AppA_LocalA@-- OTLP --> LocalA AppB L_AppB_LocalB@-- OTLP --> LocalB LocalA L_LocalA_LBLayer@-- "OTLP (spans)" --> LB_Layer LocalB L_LocalB_LBLayer@-- "OTLP (spans)" --> LB_Layer LB_Layer L_LBLayer_Sampling@-- "Route by
Trace ID" --> SamplingLayer LocalA L_LocalA_Backend@-- "OTLP (metrics & logs)" --> ObsBackend LocalB L_LocalB_Backend@-- "OTLP (metrics & logs)" --> ObsBackend SamplingLayer L_Sampling_Backend@-- "OTLP (sampled spans)" --> ObsBackend AppB -.- n1["Head sampling, aggregation, limits, etc."]:::note LocalA -.- n2["Redaction, enrichment, OTTL, governance, etc."]:::note SamplingLayer -.- n4["Sample traces, post-processing"]:::note classDef node fill:#ffffff, stroke:#818cf8, stroke-width:2px, color:#6b7280 classDef note fill:#f9fafb, stroke:#c7d2fe, stroke-width:1px, color:#9ca3af style ClusterA fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style ClusterB fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style LocalA fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style LocalB fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style GlobalTier fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style LB_Layer fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 style SamplingLayer fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8 linkStyle 6,7 stroke:#7dd3fc, fill:none, stroke-width:3px linkStyle 8,9,10,13 stroke:#a3e635, fill:none, stroke-width:3px linkStyle 11,12 stroke:#fde68a, fill:none, stroke-width:3px linkStyle 14,15,16 stroke:#c7d2fe, fill:none, stroke-width:1px L_AppA_LocalA@{ animation: fast } L_AppB_LocalB@{ animation: fast } L_LocalA_LBLayer@{ animation: fast } L_LocalB_LBLayer@{ animation: fast } L_LBLayer_Sampling@{ animation: fast } L_LocalA_Backend@{ animation: fast } L_LocalB_Backend@{ animation: fast } L_Sampling_Backend@{ animation: fast } ``` Generally, processing of telemetry should be done as close as possible to the application layer, avoiding compute and transfer costs. However, deferring processing decisions to different Collector layers may be desirable in certain situations, such as facilitating maintenance, enforcing standards, performing advanced filtering/transformations with [OTTL][28], or securing pipelines with [redaction][31] rules to ensure sensitive information never reaches a particular backend. By combining intelligent sampling, metric aggregation at different layers, and central transform/filter processors to reduce noisy telemetry, this architecture can reduce transfer and compute costs while preserving operational visibility for engineering teams. By implementing this guideline, organizations can expect to achieve: - **Efficient telemetry volumes:** Optimal use of OpenTelemetry signals, sampling, and aggregation provide telemetry volumes that allow organizations to balance between high-granularity, cost, and observability requirements. - **Efficient use of compute resources:** Position data processing at different levels limits data transfer and compute resources associated with data that can be aggregated or filtered at early stages. - **Central governance and guardrails:** Platform teams have a central point to control data emissions, allowing them to filter, transform, redact, or completely block telemetry that does not follow organization standards or adhere to data volume limits, safeguarding the organization from emitting unwanted data to backends. ## Implementation ### 1. Use OpenTelemetry Operator, or internal shared packages, for application-level configuration {#action-1} **Guidelines implemented:** [1](#guideline-1) If the environment in scope is in the supported [Kubernetes versions][32] and [instrumented languages][33], we recommend prioritizing the use of the [OpenTelemetry Operator for Kubernetes][34] for [auto-instrumentation][35]. This involves: - Installing the OpenTelemetry Operator. - Creating the relevant `Instrumentation` CRs to configure SDKs and instrumentation. - Adding annotations to individual pods or namespaces (to instrument all pods in a namespace). If deploying the OpenTelemetry Operator is not possible/compatible, we recommend providing application owners with build-time resources to easily configure the OpenTelemetry SDK and instrumentation libraries. This can be implemented following two main models: - For languages supported by [zero-code instrumentation][36], we recommend providing base container images to download instrumentation agents/libraries, provide default configuration, and configure the base `CMD` on the resulting container image to utilize these settings. - For languages not supported by zero-code instrumentation, we recommend providing shared [language-specific libraries][37] that take care of configuring the OpenTelemetry SDK and instrumentation libraries programmatically, providing hooks for users of said libraries to extend this configuration as required. This non-operator model puts application owners in charge of using these base container images or shared libraries in their codebase. While it may initially require more effort than auto-attached instrumentation, it provides a mechanism for platform teams to manage phased upgrades or configuration changes with minor version bumps of their internal libraries, requiring no further code changes from application owners. When managing centralized configuration in base container images or internal libraries, and when supported by the language, we recommend standardizing on the use of [declarative configuration][38]. Although currently not fully supported by all languages, this YAML-based configuration model provides consistency in SDK and instrumentation configuration. ### 2. Include organization standards into default, extensible application-level configuration {#action-2} **Guidelines implemented:** [1](#guideline-1), [2](#guideline-2), [4](#guideline-4) Regardless of how the configuration is delivered as part of [Action 1](#action-1), we recommend the platform team to include the following minimum base configuration as part of their offering: - **Exporters:** OTLP HTTP/protobuf (default) or OTLP gRPC configured to export to the most optimal Collector (e.g. local Gateway in the same cluster). See [Appendix 2](#appendix-2) and [Action 3](#action-3) for more details on side effects of OTLP gRPC used with standard Kubernetes Services. - **Note:** Backend/SaaS endpoints or API keys should not be included in application-level configuration, as we recommend handling these at a Collector Gateway. - **Propagators:** W3C Trace Context (`tracecontext`) to ensure distributed traces do not break across service boundaries. If necessary, include legacy formats as secondary options (Propagators API will prioritize in the order they are configured). - **Resource detectors:** Auto-detectors for the underlying infrastructure (e.g., cloud provider, Kubernetes, OS, container) to achieve consistency with no manual input. - **Instrumentation libraries**: Ensure a minimal set of instrumentation libraries are configured out of the box. If auto-instrumentation is being used, platform teams should not default to enable all by instrumentation libraries by default, carefully selecting the ones most critical to their environment, prioritizing client and server instrumentation (e.g. gRPC, HTTP, messaging, database). - **Processors, readers and views**: Settings specific to the backend in use (e.g. aggregation temporality, export intervals, attribute limits) or organization-wide standards (e.g. span/metric attributes). - **Note:** Depending on language implementations, OTLP exporters may retry when receiving retryable errors such as HTTP `429`, `503`, or gRPC `UNAVAILABLE` with optional `RetryInfo`. However, these exporters do not have the same capabilities as Collectors in terms of sending queues, and will drop batches of data if unsuccessful. Platform teams should manage sensible defaults for these buffer sizes and prioritize export to local Collectors (e.g. cluster-local Gateway) to move telemetry out of the application process as fast and reliably as possible. Application owners should monitor [SDK telemetry][13], when available, and react accordingly. - **Organization-specific resource attributes:** Standard conventions critical for routing, billing, and ownership. At a minimum, we recommend: - `service.name` ideally extracted from existing environment variables or labels injected via CI/CD tooling. - `service.version` to identify telemetry source during blue/green deployments or progressive rollouts. - `service.namespace` or `service.owner` for resource ownership. - `deployment.environment.name` (e.g. `production`, `staging`). - Other attributes injected as environment variables via the Kubernetes [Downward API][39] (i.e. `valueFrom.fieldRef.fieldPath`) standardized across application deployment templates. Platform teams must provide ways for application owners to override and extend this default configuration. The mechanism to do so will depend on the methods established in [Action 1](#action-1) for providing OTel configuration. Possible options are documented in [Appendix 3](#appendix-3) ### 3. Use OpenTelemetry Operator or Helm Charts to deploy Collector Gateways {#action-3} **Guidelines implemented:** [3](#guideline-3) To deploy centralized Gateway tiers, platform teams should standardize on either the [OpenTelemetry Operator][34] or the official [OpenTelemetry Helm Charts][44]. Both support GitOps workflows, but they require specific architectural considerations for enterprise workloads: - **OpenTelemetry Operator**: Ideal if the Operator is already used for application auto-instrumentation ([Action 1](#action-1)). The Gateway can be deployed by creating an `OpenTelemetryCollector` CR and setting `mode: deployment` or `mode: statefulset` (depending on requirements). The Operator abstracts away much of the Kubernetes boilerplate. See the Operator [documentation][45] for further guidance on how to enable autoscaling. - **Official Helm Charts**: A better option if infrastructure teams prefer granular control over native Kubernetes manifests (e.g., specific `Ingress` configurations, `PodDisruptionBudgets`, or complex affinity rules) without relying on CRDs. Regardless of the deployment tool chosen, the Gateway tier is a critical point, and owners should ensure resiliency is configured from the start: - **Configure [memory_limiter][15]**: When configured as the first processor in every Collector pipeline, this prevents Out-of-Memory (OOM) crashes during massive telemetry spikes by forcing the Collector to drop data and/or apply backpressure when memory usage hits a configured threshold. As mentioned in [Guideline 3](#guideline-3), different `memory_limiter` processors per signal may be required. - **Configure [otlp][46] or [otlp_http][47] exporter:** Ensure queues and retries are aligned with expectations on reliability vs resource consumption, handling transient backend failures before dropping data. In particular, consider `sending_queue` options like `batch`, which allows for efficient network transfer and backpressure propagation, and `block_on_overflow`, which controls if the Collector should drop data or wait until space becomes available if the queue (persistent or in-memory) is full. - **Consider [file_storage][48] extension**: If dropping data on extended observability backend service interruptions is critical to the functioning of the business, consider configuring `sending_queue.storage` in your OTLP exporter with the [file_storage][48] extension. With this extension configured, if the backend is unavailable or rate-limits exports, the Collector will buffer data to disk and automatically retry, preventing data loss. See [Appendix 4](#appendix-4) for notes on deploying the `file_storage` extension. - **gRPC load balancing**: OTLP/gRPC can be very efficient, but standard Kubernetes service routing can make it inefficient. See [Appendix 2](#appendix-2) to implement gRPC load balancing, or consider OTLP/HTTP (the default for most SDKs). - **Scale on memory and internal telemetry:** Utilize the Kubernetes Horizontal Pod Autoscaler (HPA) combined with custom metrics (see [Action 5](#action-5)). Configure the cluster to scale Gateway replicas based on memory utilization, active connections, or pipeline queue depth. - **Configuration as code**: Store the Helm values or Operator CRs in a central Git repository and use tools like ArgoCD or Flux to deploy them. This provides an audit trail and allows for phased rollouts and instant rollbacks. ### 4. Configure Collector processors for efficient telemetry volumes {#action-4} **Guidelines implemented:** [4](#guideline-4) To enrich telemetry with infrastructure context, reduce transfer and ingestion costs from low-value telemetry data, and enforce compliance before data leaves the corporate network, the platform team should consider configuring pipelines in Collector Gateways to execute the following processing steps (in order): - [**k8s_attributes**][49] **processor:** While some Kubernetes resource details (like pod ID or namespace name) can be appended at the application level (see [Action 2](#action-2)), the platform team must ensure 100% compliance for unmanaged workloads. This includes fields not available via the Downward API. Configure this processor to extract and append attributes like `k8s.deployment.name`, `k8s.statefulset.name`, etc. based on the incoming connection's pod IP. See [Appendix 5](#appendix-5) for details to consider when using the `k8s_attributes` processor. - **Processors to filter and transform data:** As a fallback measure on applications that could not apply these settings at the SDK level before being sent to the Collector, use processors like [attributes][50], [filter][51], [redaction][31], [resource][52], or [transform][53] to define rules to: - Drop single-span traces and access logs for routine endpoints (`/health`, `/metrics`, `/ready`) or non-actionable debug logs (`level=DEBUG` or `level=TRACE`). - Remove specific noisy attributes (e.g. `process.command_line`) that may be less useful in Kubernetes environments where these attributes are present in CI/CD pipelines. - Any other processing to remove low-value, noisy telemetry. - [**tail_sampling**][25] **processor:** Define strict retention policies, for instance keeping 100% of traces containing errors or exceeding a latency threshold, and a small baseline (e.g., 5%) of successful, normal-duration requests. As documented in [Guideline 4](#guideline-4), this requires two layers of collectors, using the [load_balancing][26] exporter on the first layer to route traces to the second layer based on Trace ID. See more information about load balancing exporting in our [documentation][26]. This is not an exhaustive list, and OpenTelemetry Collectors have many [processors][54] and [connectors][55] that allow organizations to extract more value out of their telemetry data. ### 5. Monitor SDKs and Collectors to ensure reliability requirements {#action-5} **Guidelines implemented:** [2](#guideline-2), [3](#guideline-3) OpenTelemetry SDKs and Collectors export standard telemetry describing the internal state of their components in operation. Application owners and platform teams should ensure these are produced reliably, monitored, and actioned as required. To identify and monitor data loss occurring before telemetry even leaves the application process (e.g., if the SDK's internal queue fills up), we recommend: - Where supported by the language ecosystem (e.g. Java via the `opentelemetry-sdk` instrumentation library, or Go via the `sdk/metric` package), enable SDK self-metrics to expose internal queue capacities, dropped spans, and exporter latency. [OpenTelemetry SDK Semantic Conventions][56] define the telemetry to be produced by SDKs, but support varies depending on language. - Languages lacking native SDK metric support for internal telemetry may still support internal diagnostics in different ways (e.g. .NET's `EventSource`, Java's `java.util.logging`, or Node.js's `diag`). Users should refer to specific implementations to configure for their particular needs and verbosity. To monitor the health of the aggregation and processing tiers, the platform team must actively capture and alert on internal [Collector telemetry][16]. We recommend following these steps: - **Export internal telemetry via OTLP:** Configure the `service.telemetry` block to emit internal metrics via OTLP to the observability backend, following company-wide standards. In addition to monitoring, these metrics should also be used as the data source for autoscaling decisions (see [Action 3](#action-3)). Please note, this OTLP configuration is separate from the OTLP exporter configured on Collector pipelines. - **Monitor and troubleshoot:** Follow advice present in the [monitor][16] and [troubleshoot][57] sections of Collector documentation, and create the necessary high-priority alerts to detect resource exhaustion and failed receiving/exporting before data is dropped by individual replicas. ## Reference Implementations {#reference-implementations} - [Adobe: An OpenTelemetry pipeline designed for simplicity at scale][58] - [Mastodon: Running OpenTelemetry Collectors in production with a small team][59] - [Skyscanner: Managing OpenTelemetry Collectors across 24 production clusters][60] ## Appendix {#appendix} ### 1. Distributed trace sampling techniques {#appendix-1} At a very high level, sampling can be mainly configured at two distinct layers: - **SDK:** Head sampling configured at the SDK level provides an efficient use of compute resources as unsampled traces are never recorded or exported by a given application. However, sampling decisions need to be made at span creation, normally resulting in probabilistic sampling, which could miss critical traces (e.g. those containing errors). - **Collector**: Collectors empower two main sampling techniques: - [_Probabilistic_][24] _sampling:_ Can be configured at any Collector layer and does not require coordination between Collectors as long as the same algorithm and seed are in use for the same trace. - [_Tail_][25] _sampling:_ A single Collector replica must store all spans for a given trace in memory before making a decision. As single replica deployment is not recommended in production environments, this model normally requires one layer of Collectors to [load balance][26] spans according to trace ID and another to perform sampling. Tail sampling requires more resources to operate and maintain. However, it provides a richer way of defining sampling policies that allow organizations to efficiently store only the traces that are critical for their services operations. For instance, traces with durations longer than a particular threshold, or those containing errors across any span in a given trace. > [!NOTE] Help wanted > > Distributed trace sampling is a complex topic in and on itself, designing a > sampling architecture across all different layers. These challenges are out of > scope for this blueprint and may be targeted in a separate blueprint. See our > [guidance][5] if you are interested in contributing. ### 2. gRPC load balancing {#appendix-2} gRPC relies on HTTP/2, multiplexing many requests over a single, long-lived TCP connection. Standard Kubernetes Services operate at Layer 4 (TCP) using `kube-proxy`, so they balance _connections_, not individual _requests_. When an SDK or a local Collector or application connects to a Gateway via a standard Kubernetes Service, it establishes one TCP connection and holds it open indefinitely. As a result, 100% of the telemetry from that agent will stream to a single Gateway pod. In high-throughput environments, this creates hot spots on specific Gateway replicas as newly scaled pods receive no traffic, undermining horizontal pod autoscaling and risking resource exhaustion on long-lived pods. To distribute telemetry evenly, platform teams should consider one of the three following patterns: #### Client-side load balancing {#client-side-load-balancing} OTLP gRPC exporters can perform the load balancing on the client side, querying Kubernetes DNS to discover the IPs of all available Gateway pods and distribute requests in a round-robin fashion across them. To achieve this, the Gateway tier should be deployed with a Headless Service so that DNS queries return a list of pod IPs rather than a single virtual IP. - **OpenTelemetry Operator:** If you deploy an `OpenTelemetryCollector` CR in `statefulset` mode, the Operator automatically generates a headless service named `{collector-name}-collector-headless.{namespace}.svc.cluster.local`. If deployed as a `deployment`, you will have to manually create a headless Kubernetes Service with `ClusterIP: None`. - **Helm Chart:** Set `service.clusterIP: None` when deploying the Gateway. The sending OTLP exporter should be configured to use the DNS resolver and the round-robin balancer. When configuring the OTLP exporter on the Collector (e.g. from local Collector to a Gateway): - **`endpoint`** must start with `dns:///` to instruct the gRPC client to perform continuous DNS resolution. - **`balancer_name`** should be set to `round_robin` (default in the Collector since `v0.105.0`). Specific client SDKs may configure gRPC clients in different ways. Refer to individual client implementations to configure client-side gRPC load balancing. #### Layer 7 Proxy / Service Mesh {#layer-7-proxy-service-mesh} In this approach, an HTTP/2-aware Layer 7 proxy is placed between the OTLP gRPC exporter and the Gateway tier. Because the proxy operates at Layer 7, it understands the HTTP/2 frames. It accepts the single long-lived TCP connection, inspects the individual gRPC requests, and distributes them evenly across all backend Gateway pods. **Implementation Methods:** - **Service Mesh (e.g., Istio, Linkerd):** If the cluster already runs a service mesh, gRPC load balancing is handled automatically. The mesh sidecar (or equivalent) intercepts the egress traffic from the edge agent and balances it across the Gateway pods. - **Standalone Proxy (e.g., Envoy, NGINX):** Deploy an Envoy or NGINX proxy (configured for `grpc_pass`) directly in front of the Gateway tier. The edge agents point to the Proxy's Kubernetes Service, and the Proxy balances the traffic to the Gateways. - **Ingress Controllers:** If SDKs or local Collectors are sending telemetry from outside the cluster (or across clusters), ensure the Ingress Controller (e.g., NGINX Ingress, Traefik, AWS ALB) is explicitly configured to support gRPC and HTTP/2 backend routing. #### Server-side connection recycling {#server-side-connection-recycling} Lastly, the Gateway OTLP gRPC receiver can be configured to close long-lived connections after a set duration using `keepalive.server_parameters.max_connection_age`. When a connection reaches this age, the server sends a `GoAway` frame, forcing the client to reconnect. On reconnection, standard Kubernetes Service routing redistributes the client across available Gateway pods. This option requires zero client-side changes โ€” only the Gateway receiver configuration needs updating: ```yaml receivers: otlp: protocols: grpc: keepalive: server_parameters: max_connection_age: 60s max_connection_age_grace: 10s ``` This approach is less precise than per-request balancing (traffic is only redistributed on reconnection, not per-request), and newly scaled pods will not receive traffic until existing connections expire. However, it is the simplest option as it requires no headless services, DNS resolvers, or L7 proxies. #### Recommendation If the organization already runs a Service Mesh, the L7 proxy option requires zero configuration on the OpenTelemetry side. If there is no Service Mesh and traffic remains within the same cluster, client-side load balancing provides the most precise distribution. Server-side connection recycling is the simplest starting point when neither is available. Alternatively, including cases where operators don't have control over the receiving backend (e.g. connections routed via public internet), consider using OTLP/HTTP (see [Action 2](#action-2)) which operates over HTTP/1.1 or short-lived HTTP/2 connections and does not suffer from the same pinning behavior. ### 3. SDK config overrides {#appendix-3} Depending on the methods established in [Action 1](#action-1) for providing OTel configuration, the platform team must document exactly how developers inherit the baseline and how they can extend it: - **OpenTelemetry Operator**: The platform team provisions a central `Instrumentation` CR in the cluster. Application owners can opt in or out via pod/namespace annotations. - _Basic overrides:_ Application owners can override specific baseline properties by injecting standard [environment variables][40] directly into their Pod spec. The [compliance matrix][41] details support for different environment variables per language. Additionally, some language implementations (e.g. [Java][42]) support configuring instrumentation libraries via library-specific environment variables. - _Complex overrides:_ If teams need to modify the `Instrumentation` CR itself (e.g., to add custom samplers or specific auto-instrumentation libraries), the platform team should manage the CR via Helm or Kustomize. This allows the platform to maintain a base template while application owners provide local overrides or value files that are merged before deployment into the cluster. - **Base container images**: Similarly to above, teams may override specific aspects via environment variables overriding defaults set in the base image. - **Internal libraries:** Internal shared libraries should provide the necessary hooks for users to pass in standard configuration blocks as required. For instance, in JavaScript a wrapper library to set up a Node SDK should allow the user to provide standard [NodeSDKConfiguration][43] configurations like `resource` or `traceExporter`. - **Declarative configuration**: Platform teams may utilize the environment variable interpolation features of the file-based configuration and allow application owners to set local environment variables that the base YAML file reads, or, as the file-based configuration standard matures, use configuration merging to blend a developer-provided `custom-otel.yaml` with the platform's `base-otel.yaml`. ### 4. Deployment notes on `file_storage` extension {#appendix-4} While using the [file_storage][48] extension and OTLP exporter's `sending_queue.storage` provides added completeness guarantees, it moves away from stateless deployments, requiring the Gateway to be deployed as a `StatefulSet` with `PersistentVolumeClaims`. As with OTLP exporter settings, operators must consider the trade-offs between the criticality of dropping data (in this case potentially stale data) and the cost of maintenance toil and support (e.g., managing disk pressure, volume resizing, etc). Additionally, enabling persistent queues delays backpressure propagation to downstream clients. Data is buffered to disk before memory pressure builds, meaning the `memory_limiter` will not trigger early backpressure. Once the persistent queue is full, the pipeline will block and the receiver will return retryable errors (e.g. `429`) to clients, signaling backpressure. Operators must size the persistent queue and monitor disk usage accordingly. ### 5. Deployment notes on `k8s_attributes` processor {#appendix-5} If a proxy is placed between the pod emitting telemetry and the Collector processing it, ensure pass-through mode is enabled on the proxy so the Gateway sees the original Pod IP of the application, not the proxy's IP. Alternatively, inject fields available via Downward API (e.g. `k8s.pod.uid`) as Resource attributes when configuring OTel SDKs, and configure `k8s_attributes` pod association rules to match the incoming Resource attributes with a given Pod. When using the `k8s_attributes` processor, the `ServiceAccount` used by the Collector must be granted RBAC `get`, `watch`, and `list` permissions on the Kubernetes resources corresponding to the attributes being extracted, e.g. `deployments` for `k8s.deployment.name` and `k8s.deployment.uid`. Missing any of these will result in the Collector silently skipping enrichment for the affected attributes. Finally, running this processor in a Gateway will cause higher memory utilization on each of the Collectors, scaling according to cluster size. The `k8s_attributes` processor keeps metadata in memory related to Objects in the cluster, and the more Objects there are to cache, the more memory the collector will consume. When run as a Gateway, each pod in the collector Deployment or StatefulSet has to remember ALL the metadata about the entire cluster (as opposed to running as a DaemonSet, where the pod only has to know about the metadata for its own node). The component documentation contains more details on [deployment and scaling considerations][62]. [1]: https://platformengineering.org/talks-library/platform-as-a-product [2]: /docs/concepts/semantic-conventions/ [3]: /docs/concepts/resources/ [4]: /docs/concepts/context-propagation/ [5]: /docs/guidance/ [6]: /docs/languages/sdk-configuration/ [7]: /docs/concepts/instrumentation/libraries/ [8]: /docs/specs/otel/library-guidelines/#opentelemetry-client-generic-design [9]: /docs/specs/otel/metrics/sdk/#view [10]: /docs/specs/otel/trace/sdk/#span-processor [11]: https://github.com/open-telemetry/weaver [12]: /blog/2025/otel-weaver/ [13]: /docs/specs/semconv/otel/sdk-metrics/ [14]: /docs/collector/deploy/gateway/ [15]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor/README.md [16]: /docs/collector/internal-telemetry/ [17]: /docs/specs/otel/metrics/#programming-model [18]: /docs/specs/otel/trace/api/#span-creation [19]: /docs/specs/otel/logs/api/#emit-a-logrecord [20]: /docs/specs/otel/metrics/sdk/#meterprovider [21]: /docs/specs/otel/trace/sdk/#tracer-provider [22]: /docs/specs/otel/logs/sdk/#loggerprovider [23]: /docs/concepts/sampling/ [24]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor/README.md [25]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor/README.md [26]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter/README.md [27]: /docs/specs/otel/metrics/sdk/#exemplar [28]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md [31]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor/README.md [32]: https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/compatibility.md#compatibility-matrix [33]: https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection [34]: /docs/platforms/kubernetes/operator/ [35]: /docs/platforms/kubernetes/operator/automatic/ [36]: /docs/zero-code/ [37]: /docs/languages/ [38]: /docs/languages/sdk-configuration/declarative-configuration/ [39]: https://kubernetes.io/docs/concepts/workloads/pods/downward-api/ [40]: /docs/specs/otel/configuration/sdk-environment-variables/ [41]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#environment-variables [42]: /docs/zero-code/java/agent/configuration/ [43]: https://github.com/open-telemetry/opentelemetry-js/blob/main/experimental/packages/opentelemetry-sdk-node/src/types.ts [44]: /docs/platforms/kubernetes/helm/ [45]: /docs/platforms/kubernetes/operator/horizontal-pod-autoscaling/ [46]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter/README.md [47]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlphttpexporter/README.md [48]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage/README.md [49]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor/README.md [50]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor/README.md [51]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor/README.md [52]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor/README.md [53]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor/README.md [54]: /docs/collector/components/processor/ [55]: /docs/collector/components/connector/ [56]: /docs/specs/semconv/otel/ [57]: /docs/collector/troubleshooting/ [58]: /docs/guidance/reference-implementations/adobe/ [59]: /docs/guidance/reference-implementations/mastodon/ [60]: /docs/guidance/reference-implementations/skyscanner/ [61]: /docs/collector/configuration/#location [62]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md#production-deployment-guide