Semantic conventions for RPC spans

NOTICE Semantic Conventions are moving to a new location.

No changes to this document are allowed.

Status: Experimental

This document defines how to describe remote procedure calls (also called “remote method invocations” / “RMI”) with spans.

Warning Existing RPC instrumentations that are using v1.20.0 of this document (or prior):

  • SHOULD NOT change the version of the networking attributes that they emit until the HTTP semantic conventions are marked stable (HTTP stabilization will include stabilization of a core set of networking attributes which are also used in RPC instrumentations).
  • SHOULD introduce an environment variable OTEL_SEMCONV_STABILITY_OPT_IN in the existing major version which supports the following values:
    • none - continue emitting whatever version of the old experimental networking attributes the instrumentation was emitting previously. This is the default value.
    • http - emit the new, stable networking attributes, and stop emitting the old experimental networking attributes that the instrumentation emitted previously.
    • http/dup - emit both the old and the stable networking attributes, allowing for a seamless transition.
  • SHOULD maintain (security patching at a minimum) the existing major version for at least six months after it starts emitting both sets of attributes.
  • SHOULD drop the environment variable in the next major version (stable next major version SHOULD NOT be released prior to October 1, 2023).

Common remote procedure call conventions

A remote procedure calls is described by two separate spans, one on the client-side and one on the server-side.

For outgoing requests, the SpanKind MUST be set to CLIENT and for incoming requests to SERVER.

Remote procedure calls can only be represented with these semantic conventions, when the names of the called service and method are known and available.

Span name

The span name MUST be the full RPC method name formatted as:

$package.$service/$method

(where $service MUST NOT contain dots and $method MUST NOT contain slashes)

If there is no package name or if it is unknown, the $package. part (including the period) is omitted.

Examples of span names:

  • grpc.test.EchoService/Echo
  • com.example.ExampleRmiService/exampleMethod
  • MyCalcService.Calculator/Add reported by the server and MyServiceReference.ICalculator/Add reported by the client for .NET WCF calls
  • MyServiceWithNoPackage/theMethod

Common attributes

AttributeTypeDescriptionExamplesRequirement Level
rpc.systemstringA string identifying the remoting system. See below for a list of well-known identifiers.grpcRequired
rpc.servicestringThe full (logical) name of the service being called, including its package name, if applicable. [1]myservice.EchoServiceRecommended
rpc.methodstringThe name of the (logical) method being called, must be equal to the $method part in the span name. [2]exampleMethodRecommended
network.transportstringOSI Transport Layer or Inter-process Communication method. The value SHOULD be normalized to lowercase.tcp; udpRecommended
network.typestringOSI Network Layer or non-OSI equivalent. The value SHOULD be normalized to lowercase.ipv4; ipv6Recommended
server.addressstringRPC server host name. [3]example.comRequired
server.portintLogical server port number80; 8080; 443Conditionally Required: See below
server.socket.addressstringPhysical server IP address or Unix socket address.10.5.3.2See below
server.socket.portintPhysical server port.16456Recommended: [4]

[1]: This is the logical name of the service from the RPC interface perspective, which can be different from the name of any implementing class. The code.namespace attribute may be used to store the latter (despite the attribute name, it may include a class name; e.g., class with method actually executing the call on the server side, RPC client stub class on the client side).

[2]: This is the logical name of the method from the RPC interface perspective, which can be different from the name of any implementing method/function. The code.function attribute may be used to store the latter (e.g., method actually executing the call on the server side, RPC client stub method on the client side).

[3]: May contain server IP address, DNS name, or local socket name. When host component is an IP address, instrumentations SHOULD NOT do a reverse proxy lookup to obtain DNS name and SHOULD set server.address to the IP address provided in the host component.

[4]: If different than server.port and if server.socket.address is set.

Additional attribute requirements: At least one of the following sets of attributes is required:

rpc.system has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
grpcgRPC
java_rmiJava RMI
dotnet_wcf.NET WCF
apache_dubboApache Dubbo
connect_rpcConnect RPC

For client-side spans server.port is required if the connection is IP-based and the port is available (it describes the server port they are connecting to). For server-side spans client.socket.port is optional (it describes the port the client is connecting from).

Service name

On the server process receiving and handling the remote procedure call, the service name provided in rpc.service does not necessarily have to match the service.name resource attribute. One process can expose multiple RPC endpoints and thus have multiple RPC service names. From a deployment perspective, as expressed by the service.* resource attributes, it will be treated as one deployed service with one service.name. Likewise, on clients sending RPC requests to a server, the service name provided in rpc.service does not have to match the peer.service span attribute.

As an example, given a process deployed as QuoteService, this would be the name that goes into the service.name resource attribute which applies to the entire process. This process could expose two RPC endpoints, one called CurrencyQuotes (= rpc.service) with a method called getMeanRate (= rpc.method) and the other endpoint called StockQuotes (= rpc.service) with two methods getCurrentBid and getLastClose (= rpc.method). In this example, spans representing client request should have their peer.service attribute set to QuoteService as well to match the server’s service.name resource attribute. Generally, a user SHOULD NOT set peer.service to a fully qualified RPC service name.

Client attributes

AttributeTypeDescriptionExamplesRequirement Level
server.socket.domainstringThe domain name of an immediate peer. [1]proxy.example.comRecommended: [2]

[1]: Typically observed from the client side, and represents a proxy or other intermediary domain name.

[2]: If different than server.address and if server.socket.address is set.

Server attributes

AttributeTypeDescriptionExamplesRequirement Level
client.addressstringClient address - unix domain socket name, IPv4 or IPv6 address. [1]/tmp/my.sock; 10.1.2.80Recommended
client.portintClient port number [2]65123Recommended
client.socket.addressstringImmediate client peer address - unix domain socket name, IPv4 or IPv6 address./tmp/my.sock; 127.0.0.1Recommended: If different than client.address.
client.socket.portintImmediate client peer port number35555Recommended: If different than client.port.
network.transportstringOSI Transport Layer or Inter-process Communication method. The value SHOULD be normalized to lowercase.tcp; udpRecommended
network.typestringOSI Network Layer or non-OSI equivalent. The value SHOULD be normalized to lowercase.ipv4; ipv6Recommended

[1]: When observed from the server side, and when communicating through an intermediary, client.address SHOULD represent client address behind any intermediaries (e.g. proxies) if it’s available.

[2]: When observed from the server side, and when communicating through an intermediary, client.port SHOULD represent client port behind any intermediaries (e.g. proxies) if it’s available.

Events

In the lifetime of an RPC stream, an event for each message sent/received on client and server spans SHOULD be created. In case of unary calls only one sent and one received message will be recorded for both client and server spans.

The event name MUST be message.

AttributeTypeDescriptionExamplesRequirement Level
message.typestringWhether this is a received or sent message.SENTRecommended
message.idintMUST be calculated as two different counters starting from 1 one for sent messages and one for received message. [1]Recommended
message.compressed_sizeintCompressed size of the message in bytes.Recommended
message.uncompressed_sizeintUncompressed size of the message in bytes.Recommended

[1]: This way we guarantee that the values will be consistent between different implementations.

message.type MUST be one of the following:

ValueDescription
SENTsent
RECEIVEDreceived

Distinction from HTTP spans

HTTP calls can generally be represented using just HTTP spans. If they address a particular remote service and method known to the caller, i.e., when it is a remote procedure call transported over HTTP, the rpc.* attributes might be added additionally on that span, or in a separate RPC span that is a parent of the transporting HTTP call. Note that method in this context is about the called remote procedure and not the HTTP verb (GET, POST, etc.).

gRPC

For remote procedure calls via gRPC, additional conventions are described in this section.

rpc.system MUST be set to "grpc".

gRPC Attributes

AttributeTypeDescriptionExamplesRequirement Level
rpc.grpc.status_codeintThe numeric status code of the gRPC request.0Required

rpc.grpc.status_code MUST be one of the following:

ValueDescription
0OK
1CANCELLED
2UNKNOWN
3INVALID_ARGUMENT
4DEADLINE_EXCEEDED
5NOT_FOUND
6ALREADY_EXISTS
7PERMISSION_DENIED
8RESOURCE_EXHAUSTED
9FAILED_PRECONDITION
10ABORTED
11OUT_OF_RANGE
12UNIMPLEMENTED
13INTERNAL
14UNAVAILABLE
15DATA_LOSS
16UNAUTHENTICATED

gRPC Status

The table below describes when the Span Status MUST be set to Error or remain unset depending on the gRPC status code and Span Kind.

gRPC Status CodeSpanKind.SERVER Span StatusSpanKind.CLIENT Span Status
OKunsetunset
CANCELLEDunsetError
UNKNOWNErrorError
INVALID_ARGUMENTunsetError
DEADLINE_EXCEEDEDErrorError
NOT_FOUNDunsetError
ALREADY_EXISTSunsetError
PERMISSION_DENIEDunsetError
RESOURCE_EXHAUSTEDunsetError
FAILED_PRECONDITIONunsetError
ABORTEDunsetError
OUT_OF_RANGEunsetError
UNIMPLEMENTEDErrorError
INTERNALErrorError
UNAVAILABLEErrorError
DATA_LOSSErrorError
UNAUTHENTICATEDunsetError

gRPC Request and Response Metadata

AttributeTypeDescriptionExamplesRequirement Level
rpc.grpc.request.metadata.<key>string[]gRPC request metadata, <key> being the normalized gRPC Metadata key (lowercase, with - characters replaced by _), the value being the metadata values. [1]rpc.grpc.request.metadata.my_custom_metadata_attribute=["1.2.3.4", "1.2.3.5"]Opt-In
rpc.grpc.response.metadata.<key>string[]gRPC response metadata, <key> being the normalized gRPC Metadata key (lowercase, with - characters replaced by _), the value being the metadata values. [1]rpc.grpc.response.metadata.my_custom_metadata_attribute=["attribute_value"]Opt-In

[1]: Instrumentations SHOULD require an explicit configuration of which metadata values are to be captured. Including all request/response metadata values can be a security risk - explicit configuration helps avoid leaking sensitive information.

Connect RPC conventions

For remote procedure calls via connect, additional conventions are described in this section.

rpc.system MUST be set to "connect_rpc".

Connect RPC Attributes

Below is a table of attributes that SHOULD be included on client and server RPC measurements when rpc.system is "connect_rpc".

AttributeTypeDescriptionExamplesRequirement Level
rpc.connect_rpc.error_codestringThe error codes of the Connect request. Error codes are always string values.cancelledConditionally Required: [1]

[1]: If response is not successful and if error code available.

rpc.connect_rpc.error_code MUST be one of the following:

ValueDescription
cancelledcancelled
unknownunknown
invalid_argumentinvalid_argument
deadline_exceededdeadline_exceeded
not_foundnot_found
already_existsalready_exists
permission_deniedpermission_denied
resource_exhaustedresource_exhausted
failed_preconditionfailed_precondition
abortedaborted
out_of_rangeout_of_range
unimplementedunimplemented
internalinternal
unavailableunavailable
data_lossdata_loss
unauthenticatedunauthenticated

Connect RPC Status

If rpc.connect_rpc.error_code is set, Span Status MUST be set to Error and left unset in all other cases.

Connect RPC Request and Response Metadata

AttributeTypeDescriptionExamplesRequirement Level
rpc.connect_rpc.request.metadata.<key>string[]Connect request metadata, <key> being the normalized Connect Metadata key (lowercase, with - characters replaced by _), the value being the metadata values. [1]rpc.request.metadata.my_custom_metadata_attribute=["1.2.3.4", "1.2.3.5"]Optional
rpc.connect_rpc.response.metadata.<key>string[]Connect response metadata, <key> being the normalized Connect Metadata key (lowercase, with - characters replaced by _), the value being the metadata values. [1]rpc.response.metadata.my_custom_metadata_attribute=["attribute_value"]Optional

[1]: Instrumentations SHOULD require an explicit configuration of which metadata values are to be captured. Including all request/response metadata values can be a security risk - explicit configuration helps avoid leaking sensitive information.

JSON RPC

Conventions specific to JSON RPC.

rpc.system MUST be set to "jsonrpc".

JSON RPC Attributes

AttributeTypeDescriptionExamplesRequirement Level
rpc.jsonrpc.versionstringProtocol version as in jsonrpc property of request/response. Since JSON-RPC 1.0 does not specify this, the value can be omitted.2.0; 1.0Conditionally Required: If other than the default version (1.0)
rpc.jsonrpc.request_idstringid property of request or response. Since protocol allows id to be int, string, null or missing (for notifications), value is expected to be cast to string for simplicity. Use empty string in case of null value. Omit entirely if this is a notification.10; request-7; ``Recommended
rpc.jsonrpc.error_codeinterror.code property of response if it is an error response.-32700; 100Conditionally Required: If response is not successful.
rpc.jsonrpc.error_messagestringerror.message property of response if it is an error response.Parse error; User already existsRecommended
rpc.methodstringThe name of the (logical) method being called, must be equal to the $method part in the span name. [1]exampleMethodRequired

[1]: This is always required for jsonrpc. See the note in the general RPC conventions for more information.