Troubleshooting Python automatic instrumentation issues

Installation issues

Python package installation failure

The Python package installs require gcc and gcc-c++, which you may need to install if you’re running a slim version of Linux, such as CentOS.

• CentOS
• Debian/Ubuntu
• Alpine

yum -y install python3-devel
yum -y install gcc-c++

apt install -y python3-dev
apt install -y build-essential

apk add python3-dev
apk add build-base

Bootstrap using uv

Running opentelemetry-bootstrap -a install when using the uv package manager may result in errored or unexpected dependency setups.

Instead, you can generate OpenTelemetry requirements dynamically and install them using uv.

First, install the appropriate packages (or add them to your project file and run uv sync):

uv pip install opentelemetry-distro opentelemetry-exporter-otlp

Now, you can install the auto instrumentation:

uv run opentelemetry-bootstrap -a requirements | uv pip install --requirement -

Finally, use uv run to start your application (see Configuring the agent):

uv run opentelemetry-instrument python myapp.py

Please note that you have to reinstall the auto instrumentation every time you run uv sync or update existing packages. It is therefore recommended to make the installation part of your build pipeline.

Instrumentation issues

Flask debug mode with reloader breaks instrumentation

The debug mode can be enabled in the Flask app like this:

if __name__ == "__main__":
    app.run(port=8082, debug=True)

The debug mode can break instrumentation from happening because it enables a reloader. To run instrumentation while the debug mode is enabled, set the use_reloader option to False:

if __name__ == "__main__":
    app.run(port=8082, debug=True, use_reloader=False)

Pre-fork server issues

A pre-fork server, such as Gunicorn with multiple workers, could be run like this:

gunicorn myapp.main:app --workers 4

However, specifying more than one --workers may break the generation of metrics when auto-instrumentation is applied. This is because forking, the creation of worker/child processes, creates inconsistencies between each child in the background threads and locks assumed by key OpenTelemetry SDK components. Specifically, the PeriodicExportingMetricReader spawns its own thread to periodically flush data to the exporter. See also issues #2767 and #3307. After forking, each child seeks a thread object in memory that is not actually run, and any original locks may not unlock for each child. See also forks and deadlocks described in Python issue 6721.

Workarounds

There are some workarounds for pre-fork servers with OpenTelemetry. The following table summarizes the current support of signal export by different auto-instrumented web server gateway stacks that have been pre-forked with multiple workers. See below for more details and options:

Deploy with Gunicorn and UvicornWorker

To auto-instrument a server with multiple workers, it is recommended to deploy using Gunicorn with uvicorn.workers.UvicornWorker if it is an Asynchronous Server Gateway Interface (ASGI) app (FastAPI, Starlette, etc). The UvicornWorker class is specifically designed to handle forks with preservation of background processes and threads. For example:

opentelemetry-instrument gunicorn \
  --workers 4 \
  --worker-class uvicorn.workers.UvicornWorker \
  --bind 0.0.0.0:8000 \
  myapp.main:app

Use programmatic auto-instrumentation

Initialize OpenTelemetry inside the worker process with programmatic auto-instrumentation after the server fork, instead of with opentelemetry-instrument. For example:

from opentelemetry.instrumentation.auto_instrumentation import initialize
initialize()

from your_app import app

If using FastAPI, note that initialize() must be called before importing FastAPI because of how instrumentation is patched. For example:

from opentelemetry.instrumentation.auto_instrumentation import initialize
initialize()

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

Then, run the server with:

uvicorn main:app --workers 2

Use Prometheus with direct OTLP

Consider using a recent version of Prometheus to receive OTLP metrics directly. Set up a PeriodicExportingMetricReader and one OTLP worker per process to push to Prometheus server. We recommend not using PrometheusMetricReader with forking – see issue #3747.

Use a single worker

Alternatively, use a single worker in pre-fork with zero-code instrumentation:

opentelemetry-instrument gunicorn your_app:app --workers 1

Connectivity issues

gRPC Connectivity

To debug Python gRPC connectivity issues, set the following gRPC debug environment variables:

export GRPC_VERBOSITY=debug
export GRPC_TRACE=http,call_error,connectivity_state
opentelemetry-instrument python YOUR_APP.py