APM reference

This page describes what data is provided by APM and how it is used.

Span schema

Spans are ingested to the Tracing/Span dataset.

Observe APM is OpenTelemetry-native, meaning it is powered entirely by OpenTelemetry data and is fully compatible with the OpenTelemetry semantic conventions. In order to make the product more intuitive for users who may not be intimately familiar with OpenTelemetry, we transform raw OpenTelemetry data into the following schema:

Span dataset field	Underlying OpenTelemetry field from which it is derived
start_time	Span start_time_unix_nano
duration	Span start_time_unix_nano and send_time_unix_nano
response_status	Span status.code
service_name	service.name in the span’s resource attributes
span_name	Span name
status_code	derived from http.status_code, rpc.status_code, grpc.status_code, or rpc.grpc.status_code in the span’s attributes
error	True if the Span Status is set to Error, false otherwise
status_message	Span Status object’s status.message
span_type	Span kind (see below for the mapping between span_type and kind
service_namespace	service.namespace in the span’s resource attributes
service_instance_id	service.instance.id in the span’s resource attributes
environment	deployment.environment in the span’s resource attributes
service_version	service.version in the span’s resource attributes
attributes	The span’s attributes
resource_attributes	The span’s resource attributes
instrumentation_library	Instrumentation Scope object
span_id
parent_span_id
trace_id

The span_type field provides an intuitive categorization of OpenTelemetry span kinds, mapping 1:1 to the kind field in OpenTelemetry but using more user-friendly terms to describe the span’s role in the system:

Observe span_type value	OpenTelemetry Span kind value
Service entry point	SERVER
Remote call	CLIENT
Async consumer	CONSUMER
Async producer	PRODUCER
Internal operation	INTERNAL

Endpoint terminology

The “Endpoint” terminology in Observe has equivalents in other APM products:

Observe	Datadog	New Relic	AppDynamics / Dynatrace
Endpoint	Resource	Transaction	Business Transaction

Toggling between HTTP/RPC endpoints and message-based endpoints in Observe also has equivalents in other APM products:

Observe	Datadog	New Relic
Endpoint dropdown	Operation dropdown	Web/non-web transaction dropdown

Underlying OpenTelemetry data used to detect endpoints:

• Spans with kind SERVER are used to detect HTTP/RPC endpoints, which are known as “Service entry points” in Observe. The endpoint name in Observe is the span name.

• Spans with kind CONSUMER are used to detect message-based endpoints, which are known as “Async consumers” in Observe. The endpoint name in Observe is the span name.

The Service Inspector also allows you to toggle between synchronous (HTTP/RPC) and asynchronous (message-based) endpoints with the “operation type” dropdown.

Correlation tags on span data

The following correlation tags are added to the Span dataset and can be used to correlate other data with traces:

Correlation tag	Span dataset field on which it is defined
deployment.environment.name	resource_attributes."deployment.environment"
k8s.cluster.uid	resource_attributes."k8s.cluster.uid"
k8s.namespace.name	resource_attributes."k8s.namespace.name"
k8s.pod.name	resource_attributes."k8s.pod.name"
service.name	resource_attributes."service.name"
service.namespace	resource_attributes."service.namespace"
service.version	resource_attributes."service.version"
service.name	service_name
service.namespace	service_namespace
operation.name	span_name
service.version	service_version

Span event dataset schema

Span events are ingested to the Tracing/Span Event dataset.

Span Event dataset field	Underlying OTel field from which it is derived
span_id	the span_id of the span containing the event
trace_id	the trace_id of the span containing the event
event_name	The event’s name
attributes	The event’s attributes
resource_attributes	The resource attributes of the span containing the event

Span link dataset schema

Span links are ingested to the Tracing/Span Link dataset.

Span Link dataset field	Underlying OTel field from which it is derived
span_id	the span_id of the span containing the link
trace_id	the trace_id of the span containing the link
linked_span_id	the link’s span_id
linked_trace_id	the link’s trace_id
attributes	the link’s attributes

APM metrics

The following metrics are derived from spans and are available in the Metric Explorer, and can be used for dashboarding and monitors.

Facets

The following dimensions are tracked by all the APM metrics:

• service.name, peer.db.name, and peer.messaging.system: they are mutually exclusive and represent the system being instrumented – microservice, database, or messaging system.

• service.namespace

• service.version

• span.name: only the span names for spans that correspond to endpoints (i.e., spans of type Service entry point or Async consumer) are available in APM metrics

• span.type

• k8s.cluster.uid

• k8s.namespace.name

• k8s.pod.name

RED metrics

• apm_service_call_count

• apm_service_error_count

• apm_service_duration (histogram)

• The following metrics are the same as above but with extra dimensions, i.e., status.code and status.message:

  • apm_service_call_count_with_status

  • apm_service_error_count_with_status

  • apm_service_duration_with_status

• apm_service_duration_log: same as apm_service_duration but values are log base-2

• apm_service_duration_sum_with_status: total duration (i.e., sum of latency across all invocations)

• apm_service_exception_count: exception count per service

  • Unique labels/dimensions:

    • exception.type

    • exception.stacktrace

    • exception.message.pattern.hash

• apm_service_error_count_by_fingerprint: number of invocations per unique exceptions per service

Other metrics are also published beyond the basic RED metrics, all with the prefix apm_.