# Observability primer > Core observability concepts. --- LLMS index: [llms.txt](/llms.txt) --- ## What is Observability? Observability lets you understand a system from the outside by letting you ask questions about that system without knowing its inner workings. Furthermore, it allows you to easily troubleshoot and handle novel problems, that is, "unknown unknowns". It also helps you answer the question "Why is this happening?" To ask those questions about your system, your application must be properly instrumented. That is, the application code must emit [signals](/docs/concepts/signals/) such as [traces](/docs/concepts/signals/traces/), [metrics](/docs/concepts/signals/metrics/), and [logs](/docs/concepts/signals/logs/). An application is properly instrumented when developers don't need to add more instrumentation to troubleshoot an issue, because they have all of the information they need. [OpenTelemetry](/docs/what-is-opentelemetry/) is the mechanism by which application code is instrumented to help make a system observable. ## Reliability and metrics **Telemetry** refers to data emitted from a system and its behavior. The data can come in the form of [traces](/docs/concepts/signals/traces/), [metrics](/docs/concepts/signals/metrics/), and [logs](/docs/concepts/signals/logs/). **Reliability** answers the question: "Is the service doing what users expect it to be doing?" A system could be up 100% of the time, but if, when a user clicks "Add to Cart" to add a black pair of shoes to their shopping cart, the system doesn't always add black shoes, then the system could be **un**reliable. **Metrics** are aggregations over a period of time of numeric data about your infrastructure or application. Examples include: system error rate, CPU utilization, and request rate for a given service. For more on metrics and how they relate to OpenTelemetry, see [Metrics](/docs/concepts/signals/metrics/). **SLI**, or Service Level Indicator, represents a measurement of a service's behavior. A good SLI measures your service from the perspective of your users. An example SLI can be the speed at which a web page loads. **SLO**, or Service Level Objective, represents the means by which reliability is communicated to an organization/other teams. This is accomplished by attaching one or more SLIs to business value. ## Understanding distributed tracing Distributed tracing lets you observe requests as they propagate through complex, distributed systems. Distributed tracing improves the visibility of your application or system's health and lets you debug behavior that is difficult to reproduce locally. It is essential for distributed systems, which commonly have nondeterministic problems or are too complicated to reproduce locally. To understand distributed tracing, you need to understand the role of each of its components: logs, spans, and traces. ### Logs A **log** is a timestamped message emitted by services or other components. Unlike [traces](#distributed-traces), they aren't necessarily associated with any particular user request or transaction. You can find logs almost everywhere in software. Logs have been heavily relied on in the past by both developers and operators to help them understand system behavior. Sample log: ```text I, [2021-02-23T13:26:23.505892 #22473] INFO -- : [6459ffe1-ea53-4044-aaa3-bf902868f730] Started GET "/" for ::1 at 2021-02-23 13:26:23 -0800 ``` Logs aren't enough for tracking code execution, as they usually lack contextual information, such as where they were called from. They become far more useful when they are included as part of a [span](#spans), or when they are correlated with a trace and a span. For more on logs and how they pertain to OpenTelemetry, see [Logs](/docs/concepts/signals/logs/). ### Spans A **span** represents a single unit of work or operation. Spans track specific operations that a request makes, painting a picture of what happened during the time in which that operation was executed. A span contains name, time-related data, [structured log messages](/docs/concepts/signals/traces/#span-events), and [other metadata (that is, Attributes)](/docs/concepts/signals/traces/#attributes) to provide information about the operation it tracks. #### Span attributes Span attributes are metadata attached to a span. The following table contains examples of span attributes: | Key | Value | | :-------------------------- | :--------------------------------------------------------------------------------- | | `http.request.method` | `"GET"` | | `network.protocol.version` | `"1.1"` | | `url.path` | `"/webshop/articles/4"` | | `url.query` | `"?s=1"` | | `server.address` | `"example.com"` | | `server.port` | `8080` | | `url.scheme` | `"https"` | | `http.route` | `"/webshop/articles/:article_id"` | | `http.response.status_code` | `200` | | `client.address` | `"192.0.2.4"` | | `client.socket.address` | `"192.0.2.5"` (the client goes through a proxy) | | `user_agent.original` | `"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0"` | For more on spans and how they relate to OpenTelemetry, see [Spans](/docs/concepts/signals/traces/#spans). ### Distributed traces A **distributed trace**, more commonly known as a **trace**, records the path taken by a single request (made by an application or end user) as it propagates through multiple services in an architecture, such as microservice or serverless applications. A trace is made of one or more spans. The first span represents the root span. Each root span represents a request from start to finish. The spans underneath the parent provide a more in-depth context of what occurs during a request (or what steps make up a request). For example, when a user loads a web page, the initial HTTP request may pass through an API gateway, a backend service, and a database. Each of these steps is represented by a span, and together they form a single trace that shows the end-to-end journey of the request. Without tracing, finding the root cause of performance problems in a distributed system can be challenging. Tracing makes debugging and understanding distributed systems less daunting by breaking down what happens within a request as it flows through a distributed system. Many Observability backends visualize traces as waterfall diagrams that look like this: ![Sample Trace](/img/waterfall-trace.svg 'Trace waterfall diagram') Waterfall diagrams show the parent-child relationship between a root span and its child spans. When a span encapsulates another span, this also represents a nested relationship. For more on traces and how they pertain to OpenTelemetry, see [Traces](/docs/concepts/signals/traces/).