链路追踪常见协议实现
mufiye 内核新手
  2024-09-20 23:21:19 2024-09-20 23:21      3.1k 字  16 分钟  

1. Skywalking

1.1 Segment

Segment是一批span的集合，它表示一个线程上下文，如果跨线程则需要重新记录为一个segment；跨进程可以表示发起了一个跨线程调用，更多地情况（也是链路追踪最大的意义）是表示了两个服务之间的调用。

  • traceId：一条调用链路的唯一id，用其串联起一条链路
  • traceSegmentId：一个segment的id
  • spans：该segment上下文中所有的span
  • service：表示该segment所在的服务
  • serviceInstance：表示该segment所在的服务实例（在kubernetes中对应一个pod，服务基本都是多实例部署的）
  • isSizeLimited：是否有一些span因为过大被丢弃了
1
2
3
4
5
6
7
8
9
10
11
12
13
message SegmentObject {
    string traceId = 1;

    string traceSegmentId = 2;

    repeated SpanObject spans = 3;

    string service = 4;

    string serviceInstance = 5;

    bool isSizeLimited = 6;
}

1.2 Span

span是segment下层的概念，它的含义是一个更细粒度的方法调用

  • spanId：该span的id，从0开始，在一个segment中保证唯一

  • parentSpanId：该span在这个segment中的父亲span的id，如果是-1表示没有父亲span，则该span是根span

  • startTime：开始时间

  • endTime：结束时间，开始时间和结束时间可以得到耗时信息

  • refs：如果发生了跨线程/进程，记录上个segment的信息；如果是MQ或者批量操作的情况下，会有多个ref

  • operationName：span的操作名称

  • peer：如果是exit span，记录要调用的接口或http请求

  • spanType：

    • entry span：一个segment的头部span
    • local span：一个segment本地操作的span
    • exit span：一个segment的尾部span，标志一个跨线程调用的发起

  • spanLayer：span所在的服务的类型，rpc，db，cache等

  • componentId：预定义的id，根据埋点的组件决定，比如spring等

  • isError：该span是否错误

  • tags：键值对，给span打标签

  • logs：记录该span中发生的事件，记录键值对和时间

  • skipAnalysis：是否跳过分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
message SpanObject {

    int32 spanId = 1;

    int32 parentSpanId = 2;

    int64 startTime = 3;

    int64 endTime = 4;

    repeated SegmentReference refs = 5;

    string operationName = 6;

    string peer = 7;

    SpanType spanType = 8;

    SpanLayer spanLayer = 9;

    int32 componentId = 10;

    bool isError = 11;

    repeated KeyStringValuePair tags = 12;

    repeated Log logs = 13;

    bool skipAnalysis = 14;
}

1.2.1 SpanType

1
2
3
4
5
6
7
8
9
// Map to the type of span
enum SpanType {
    // Server side of RPC. Consumer side of MQ.
    Entry = 0;
    // Client side of RPC. Producer side of MQ.
    Exit = 1;
    // A common local code execution.
    Local = 2;
}

1.2.2 SpanLayer

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// Map to the layer of span
enum SpanLayer {
    // Unknown layer. Could be anything.
    Unknown = 0;
    // A database layer, used in tracing the database client component.
    Database = 1;
    // A RPC layer, used in both client and server sides of RPC component.
    RPCFramework = 2;
    // HTTP is a more specific RPCFramework.
    Http = 3;
    // A MQ layer, used in both producer and consuer sides of the MQ component.
    MQ = 4;
    // A cache layer, used in tracing the cache client component.
    Cache = 5;
    // A FAAS layer, used in function-as-a-Service platform.
    FAAS = 6;
}

1.2.3 Log

1
2
3
4
5
6
7
message Log {
    // The timestamp in milliseconds of this event.,
    // measured between the current time and midnight, January 1, 1970 UTC.
    int64 time = 1;
    // String key, String value pair.
    repeated KeyStringValuePair data = 2;
}

1.3 Segment Reference

segment reference用于连接两个segment，把他们串起来。使用segment reference可以根据服务侧的entry span方便地生成拓扑，具体拓扑生成的方式可以看wusheng的这篇STAM分析方法。

  • refType：父亲segment到儿子segment的调用关系，是跨进程还是跨线程
  • traceId：父亲segment的trace id
  • parentTraceSegmentId：父亲segment的segment id
  • parentSpanId：父亲segment的exit span id
  • parentService：父亲segment的服务名称
  • parentServiceInstance：父亲segment的服务实例
  • parentEndpoint：父亲segment的enpoint信息，如果父亲服务是http服务，那么就是一个父亲服务的url；如果父亲服务是rpc服务，那么就是它的服务名+接口。这个也对应父亲segment的entry span的operation name。
  • networkAddressUsedAtPeer：客户端调用服务端，服务端的网络地址；这个等同于exit span的peer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
message SegmentReference {

    RefType refType = 1;

    string traceId = 2;

    string parentTraceSegmentId = 3;

    int32 parentSpanId = 4;

    string parentService = 5;

    string parentServiceInstance = 6;

    string parentEndpoint = 7;

    string networkAddressUsedAtPeer = 8;
}

2. OpenTelemetry

其实总的来说链路追踪的协议设计上都比较接近，但是opentelemetry没有直接的segment的概念

2.1 一批trace数据的集合

1
2
3
message TracesData {
  repeated ResourceSpans resource_spans = 1;
}

2.2 对应一条trace的概念

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// A collection of ScopeSpans from a Resource.
message ResourceSpans {
  reserved 1000;

  // The resource for the spans in this message.
  // If this field is not set then no resource info is known.
  opentelemetry.proto.resource.v1.Resource resource = 1;

  // A list of ScopeSpans that originate from a resource.
  repeated ScopeSpans scope_spans = 2;

  // The Schema URL, if known. This is the identifier of the Schema that the resource data
  // is recorded in. To learn more about Schema URL see
  // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url
  // This schema_url applies to the data in the "resource" field. It does not apply
  // to the data in the "scope_spans" field which have their own schema_url field.
  string schema_url = 3;
}

2.3 对应一个segment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// A collection of Spans produced by an InstrumentationScope.
message ScopeSpans {
  // The instrumentation scope information for the spans in this message.
  // Semantically when InstrumentationScope isn't set, it is equivalent with
  // an empty instrumentation scope name (unknown).
  opentelemetry.proto.common.v1.InstrumentationScope scope = 1;

  // A list of Spans that originate from an instrumentation scope.
  repeated Span spans = 2;

  // The Schema URL, if known. This is the identifier of the Schema that the span data
  // is recorded in. To learn more about Schema URL see
  // https://opentelemetry.io/docs/specs/otel/schemas/#schema-url
  // This schema_url applies to all spans and span events in the "spans" field.
  string schema_url = 3;
}

2.4 基本单元，Span

  • trace_id
  • span_id
  • trace_state：它用于在分布式跟踪的环境中传递有关当前 Trace 的数据，可以是键值对的形式组成string
  • parent_span_id
  • flags：标识位表示该span是否是远程调用
  • name
  • kind：span的类型，不过使用了server、client、internal、producer、consumer和unspecified
  • start_time_unix_nano
  • end_time_unix_nano
  • attributes：属性键值对，对当前span打了标签
  • dropped_attributes_count
  • events：更细粒度的事件
  • dropped_events_count
  • links：用于将span进行串联
  • dropped_links_count
  • status：span的状态
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
// A Span represents a single operation performed by a single component of the system.
//
// The next available field id is 17.
message Span {
  // A unique identifier for a trace. All spans from the same trace share
  // the same `trace_id`. The ID is a 16-byte array. An ID with all zeroes OR
  // of length other than 16 bytes is considered invalid (empty string in OTLP/JSON
  // is zero-length and thus is also invalid).
  //
  // This field is required.
  bytes trace_id = 1;

  // A unique identifier for a span within a trace, assigned when the span
  // is created. The ID is an 8-byte array. An ID with all zeroes OR of length
  // other than 8 bytes is considered invalid (empty string in OTLP/JSON
  // is zero-length and thus is also invalid).
  //
  // This field is required.
  bytes span_id = 2;

  // trace_state conveys information about request position in multiple distributed tracing graphs.
  // It is a trace_state in w3c-trace-context format: https://www.w3.org/TR/trace-context/#tracestate-header
  // See also https://github.com/w3c/distributed-tracing for more details about this field.
  string trace_state = 3;

  // The `span_id` of this span's parent span. If this is a root span, then this
  // field must be empty. The ID is an 8-byte array.
  bytes parent_span_id = 4;

  // Flags, a bit field.
  //
  // Bits 0-7 (8 least significant bits) are the trace flags as defined in W3C Trace
  // Context specification. To read the 8-bit W3C trace flag, use
  // `flags & SPAN_FLAGS_TRACE_FLAGS_MASK`.
  //
  // See https://www.w3.org/TR/trace-context-2/#trace-flags for the flag definitions.
  //
  // Bits 8 and 9 represent the 3 states of whether a span's parent
  // is remote. The states are (unknown, is not remote, is remote).
  // To read whether the value is known, use `(flags & SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK) != 0`.
  // To read whether the span is remote, use `(flags & SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK) != 0`.
  //
  // When creating span messages, if the message is logically forwarded from another source
  // with an equivalent flags fields (i.e., usually another OTLP span message), the field SHOULD
  // be copied as-is. If creating from a source that does not have an equivalent flags field
  // (such as a runtime representation of an OpenTelemetry span), the high 22 bits MUST
  // be set to zero.
  // Readers MUST NOT assume that bits 10-31 (22 most significant bits) will be zero.
  //
  // [Optional].
  fixed32 flags = 16;

  // A description of the span's operation.
  //
  // For example, the name can be a qualified method name or a file name
  // and a line number where the operation is called. A best practice is to use
  // the same display name at the same call point in an application.
  // This makes it easier to correlate spans in different traces.
  //
  // This field is semantically required to be set to non-empty string.
  // Empty value is equivalent to an unknown span name.
  //
  // This field is required.
  string name = 5;

  // Distinguishes between spans generated in a particular context. For example,
  // two spans with the same name may be distinguished using `CLIENT` (caller)
  // and `SERVER` (callee) to identify queueing latency associated with the span.
  SpanKind kind = 6;

  // start_time_unix_nano is the start time of the span. On the client side, this is the time
  // kept by the local machine where the span execution starts. On the server side, this
  // is the time when the server's application handler starts running.
  // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970.
  //
  // This field is semantically required and it is expected that end_time >= start_time.
  fixed64 start_time_unix_nano = 7;

  // end_time_unix_nano is the end time of the span. On the client side, this is the time
  // kept by the local machine where the span execution ends. On the server side, this
  // is the time when the server application handler stops running.
  // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January 1970.
  //
  // This field is semantically required and it is expected that end_time >= start_time.
  fixed64 end_time_unix_nano = 8;

  // attributes is a collection of key/value pairs. Note, global attributes
  // like server name can be set using the resource API. Examples of attributes:
  //
  //     "/http/user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
  //     "/http/server_latency": 300
  //     "example.com/myattribute": true
  //     "example.com/score": 10.239
  //
  // The OpenTelemetry API specification further restricts the allowed value types:
  // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/README.md#attribute
  // Attribute keys MUST be unique (it is not allowed to have more than one
  // attribute with the same key).
  repeated opentelemetry.proto.common.v1.KeyValue attributes = 9;

  // dropped_attributes_count is the number of attributes that were discarded. Attributes
  // can be discarded because their keys are too long or because there are too many
  // attributes. If this value is 0, then no attributes were dropped.
  uint32 dropped_attributes_count = 10;
  
  // events is a collection of Event items.
  repeated Event events = 11;

  // dropped_events_count is the number of dropped events. If the value is 0, then no
  // events were dropped.
  uint32 dropped_events_count = 12;
    
  // links is a collection of Links, which are references from this span to a span
  // in the same or different trace.
  repeated Link links = 13;

  // dropped_links_count is the number of dropped links after the maximum size was
  // enforced. If this value is 0, then no links were dropped.
  uint32 dropped_links_count = 14;

  // An optional final status for this span. Semantically when Status isn't set, it means
  // span's status code is unset, i.e. assume STATUS_CODE_UNSET (code = 0).
  Status status = 15;
}

2.4.1 SpanKind

表示span的类型，不同于skywalking中的entry span、local span和exit span，其使用internal对应了local span，使用server、client表示了调用关系的entry span和exit span，但是其特别标识了不明确类型的span，以及区分出了消费者和生产者的span，对应于MQ的场景。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// SpanKind is the type of span. Can be used to specify additional relationships between spans
// in addition to a parent/child relationship.
enum SpanKind {
  // Unspecified. Do NOT use as default.
  // Implementations MAY assume SpanKind to be INTERNAL when receiving UNSPECIFIED.
  SPAN_KIND_UNSPECIFIED = 0;

  // Indicates that the span represents an internal operation within an application,
  // as opposed to an operation happening at the boundaries. Default value.
  SPAN_KIND_INTERNAL = 1;

  // Indicates that the span covers server-side handling of an RPC or other
  // remote network request.
  SPAN_KIND_SERVER = 2;

  // Indicates that the span describes a request to some remote service.
  SPAN_KIND_CLIENT = 3;

  // Indicates that the span describes a producer sending a message to a broker.
  // Unlike CLIENT and SERVER, there is often no direct critical path latency relationship
  // between producer and consumer spans. A PRODUCER span ends when the message was accepted
  // by the broker while the logical processing of the message might span a much longer time.
  SPAN_KIND_PRODUCER = 4;

  // Indicates that the span describes consumer receiving a message from a broker.
  // Like the PRODUCER kind, there is often no direct critical path latency relationship
  // between producer and consumer spans.
  SPAN_KIND_CONSUMER = 5;
}

2.4.2 Event

是对更细粒度的操作的记录，比如一个Rpc操作可以用Event记录调用下游的ip、zone是什么，以及操作类型等

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Event is a time-stamped annotation of the span, consisting of user-supplied
// text description and key-value pairs.
message Event {
  // time_unix_nano is the time the event occurred.
  fixed64 time_unix_nano = 1;

  // name of the event.
  // This field is semantically required to be set to non-empty string.
  string name = 2;

  // attributes is a collection of attribute key/value pairs on the event.
  // Attribute keys MUST be unique (it is not allowed to have more than one
  // attribute with the same key).
  repeated opentelemetry.proto.common.v1.KeyValue attributes = 3;

  // dropped_attributes_count is the number of dropped attributes. If the value is 0,
  // then no attributes were dropped.
  uint32 dropped_attributes_count = 4;
}

相当于skywalking中segment reference的概念，可以将服务等信息存在键值对结构的attributes中，表示了span之间的调用、串联关系

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// A pointer from the current span to another span in the same trace or in a
// different trace. For example, this can be used in batching operations,
// where a single batch handler processes multiple requests from different
// traces or when the handler receives a request from a different project.
message Link {
  // A unique identifier of a trace that this linked span is part of. The ID is a
  // 16-byte array.
  bytes trace_id = 1;

  // A unique identifier for the linked span. The ID is an 8-byte array.
  bytes span_id = 2;

  // The trace_state associated with the link.
  string trace_state = 3;

  repeated opentelemetry.proto.common.v1.KeyValue attributes = 4;

  uint32 dropped_attributes_count = 5;

  fixed32 flags = 6;
}

2.4.4 Status

用于表示Span的状态，skywalking中直接用isError表示为是否是错误的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
message Status {
  reserved 1;

  // A developer-facing human readable error message.
  string message = 2;

  // For the semantics of status codes see
  // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status
  enum StatusCode {
    // The default status.
    STATUS_CODE_UNSET               = 0;
    // The Span has been validated by an Application developer or Operator to 
    // have completed successfully.
    STATUS_CODE_OK                  = 1;
    // The Span contains an error.
    STATUS_CODE_ERROR               = 2;
  };

  // The status code.
  StatusCode code = 3;
}

2.4.5 Span Flag

标识位表示该span是否是远程调用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
enum SpanFlags {
  // The zero value for the enum. Should not be used for comparisons.
  // Instead use bitwise "and" with the appropriate mask as shown above.
  SPAN_FLAGS_DO_NOT_USE = 0;

  // Bits 0-7 are used for trace flags.
  SPAN_FLAGS_TRACE_FLAGS_MASK = 0x000000FF;

  // Bits 8 and 9 are used to indicate that the parent span or link span is remote.
  // Bit 8 (`HAS_IS_REMOTE`) indicates whether the value is known.
  // Bit 9 (`IS_REMOTE`) indicates whether the span or link is remote.
  SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK = 0x00000100;
  SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK = 0x00000200;

  // Bits 10-31 are reserved for future use.
}

3. Xray

  • 本文标题：链路追踪常见协议实现
  • 本文作者：mufiye
  • 创建时间：2024-09-20 23:21:19
  • 本文链接：http://mufiye.github.io/2024/09/20/链路追踪常见协议实现/
  • 版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
 评论