OTel component performance benchmarks
As more and more users are looking to use OpenTelemetry instrumentation in their production deployments, one important consideration is the impact that OpenTelemetry will have on their application performance. In this blog post I will discuss a few recent improvements in tooling around performance benchmarking.
Instrumentation is not free. It intercepts an application’s operations and collects (often) a large amount of data, which takes additional CPU and memory. This can have a direct effect on throughput and response time, which can affect the end-user experience with the application. It can also have an impact on operational cost, such as increasing the number of instances a service runs on.
Providing general guidance about performance overhead is inherently difficult. There are many factors that affect performance: the application throughput, hardware the application runs on, what exactly is instrumented, how the OpenTelemetry SDK is configured, sampling, etc. Ultimately, the best way to measure performance is in the context of the specific application by running a load test.
With that said a number of OpenTelemetry components include performance tests that help catch regressions and can be used to provide some idea of their performance characteristics.
- Tests run on community-owned bare metal machines, which has made test results more consistent.
- Test results are published automatically: for a subset of the load test results, see Collector Benchmarks. The complete test results are available as well.
A number of OpenTelemetry SDKs already include existing micro-benchmark tests, for example:
There is work in progress to make the same updates for Python and Go.
Performance optimization is often considered only as an afterthought, but it does not have to be. We are making improvements to automated tooling and documentation to provide project maintainers and the community with reliable performance testing during development. Ultimately our focus as a community is to give end users confidence when using our components, especially around the impact of OpenTelemetry’s instrumentation on their applications’ performance.