Skip to content

Kafka

Read before enabling trace propagation

Trace propagation is set to none (disabled) by default due to the chance that some versions of Kafka do not support record headers. If a producer supports headers but a consumer doesn't, enabling trace propagation can mean that the consumer is unable to consume records, potentially causing missing data.

Semantic Span Attributes

Attribute Key/Name Attribute Value Obfuscated/Unset when -Dfr.observability.trace.attributes.semantics.obfuscation.enabled=true
net.peer.name Remote hostname. True
net.peer.port Remote port. True
messaging.system The type of messaging system. The value will be kafka False
messaging.destination_kind The kind of messaging destination. The value will be topic False
messaging.destination The topic. False
messaging.message_payload_size_bytes The size, in bytes, of the received payload. False
messaging.operation The kind of messaging operation. Value will be receive for basicGet and process for handleDelivery. False
messaging.kafka.partition The partition the record is on. False
messaging.kafka.consumer_group The consumer group the consumer belongs to. False
messaging.kafka.offset The offset of the consumed record. False
messaging.kafka.client_id The client ID of the producer/consumer. False
messaging.kafka.message_key The key for the record. Is the String representation of the object. False
messaging.queue_time_ms How long the message was queued for in milliseconds. Might not be accurate as the timestamp might be created when the record is created not produced. False
messaging.kafka.header.{header_key} The header value corresponding to the specified header key. Will be forced to lowercase and have - characters replaced with _ False

Span Name

Span names for producer/publisher spans are formatted as {topic} send, whilst receiver and processor spans are formatted as {topic} {messaging operation}.

Things to be aware of

  • There are three modes of trace propagation for Kafka spans which can be set via observability.trace.propagation.messaging.type=(parent|link|none) in reactor.conf or -Dfr.observability.trace.propagation.messaging.type=(parent|link|none). They are:
    • None (Default) - sent messages won't propagate trace information to message receivers.
    • Link - sent messages will propagate trace information which the message receivers will use to start a new trace for the receiving transaction but will add a link to the span which sent the message.
    • Parent - sent messages will propagate trace information which the message receivers will use to append the receiving span to the producer span. Both the producing process and the receiving process will be on the same trace.
  • You can change the propagation type, during runtime, by finding the Tracing section within the FusionReactor settings.
  • Topic will be <unknown_topic> if not found.
  • Track up to 10 message headers by settings a comma-separated list of header keys via -Dfr.observability.trace.kafka.headers=.
  • The tracked header span attribute value will be added as a list of values which match the key. This doesn't mean the value is a list.
  • Callbacks from producing records will be appended to the producer span's parent, if one doesn't exist a new trace will be created.
  • When propagation is set to link/none, a span will be created for the consumer poll which will be the parent of all the consumer record spans.

Properties for Kafka

Check the attributes page for general properties that affect Kafka spans.

Property Key Default Description
fr.observability.trace.propagation.messaging.type none Determines the propagation relationship for messaging spans ( Kafka/RabbitMQ ). Options are none, link and parent. If none, headers aren't injected when producing and aren't extracted on consuming. If link, the consumer span will set the propagated trace context as a link. If parent, the consumer span will set the propagated trace context as a parent. Headers are injected for both link and parent.
fr.observability.trace.kafka.headers `` Comma-separated list of header names/keys to track for Kafka Producer and Consumer transactions. Maximum of 10 allowed.

Example Span

Example of a Kafka trace with propagation type set to parent. The producer span (ota_topic send) has been set as the parent of the consumer span (ota_topics process) on the same trace. The producer callback (HTTP GET) has been appended to the producer span's parent. The header has been added as a list even though the value wasn't.
The same span as before, but this time the propagation type has been set to link. The consumer/process span is on a different trace to the span which produced it, but it has a hyperlink to view said span.