Skip to content

A108-otel-custom-per-call-label.md

Latest commit

 

History

History
149 lines (115 loc) · 5.73 KB

A108-otel-custom-per-call-label.md

File metadata and controls

149 lines (115 loc) · 5.73 KB

A108: OpenTelemetry Custom Per-Call Metric Label

Abstract

Add a custom label for applications to define their own dimension for per-call metrics.

Background

gRFC A66's per-call metrics are being used and found to be useful. However, applications can have their own data relevant to the metric. Baggage is OpenTelemetry's answer to this use-case.

gRFC A79 defined a standard metric architecture across gRPC implementations, but it explicitly did not include per-call metrics. For this reason, it does not include a "context" in its API, which is necessary for propagating baggage.

While supporting baggage is possible in some languages, it is currently counter to the performance optimizations C++ has been considering for its metrics. Determining baggage's cross-language role will be an ongoing discussion, and depending how that discussion finalizes this gRFC will either be a stop-gap or the long-term, cross-language, preferred solution.

It should also be noted that OpenTelemetry is not the only potential metric implementation. Other metric libraries may not have the concept of baggage. Using baggage also requires using an OpenTelemetry API, and a library using gRPC may not also be aware of the precise metric library that will be used by the application, so a gRPC-defined label allows callers to add context to their RPCs without mandating a particular metric implementation.

Related Proposals:

Proposal

gRPC will add an API for applications to provide a custom label value when starting RPCs. That value will be propagated to an optional grpc.client.call.custom label of all client-side call-related instruments. When not specified, the value will be empty string. The instruments that would gain this label:

  • Per-call instruments (gRFC A66)
    • grpc.client.call.duration
    • grpc.client.attempt.started
    • grpc.client.attempt.duration
    • grpc.client.attempt.sent_total_compressed_message_size
    • grpc.client.attempt.rcvd_total_compressed_message_size
  • Retry instruments (gRFC A96)
    • grpc.client.call.retries
    • grpc.client.call.transparent_retries
    • grpc.client.call.hedges
    • grpc.client.call.retry_delay
  • Other per-call instruments, e.g., those added by an LB policy, are encouraged to support this label
    • RLS is the only such case today, but is not defined in a gRFC

The per-call and retry instruments are implemented directly in gRPC's OpenTelemetry module, so gRPC must propagate the value to the module. For LB policies to add it to their own instruments, gRPC must propagate the value to the LB API during picks.

Java

gRPC Java will add a new API io.grpc.Grpc.CALL_OPTION_CUSTOM_LABEL with type CallOptions.Key<String>. The call option will default to the empty string when unset.

The RPC's CallOption is visible to ClientInterceptor (passed explicitly), ClientStreamTracer.Factory (inside StreamInfo), and SubchannelPicker (inside PickSubchannelArgs), so no additional plumbing is necessary.

Go

gRPC Go will add a new API to set and get the custom label value in context.Context. ctx is already passed to grpc.UnaryClientInterceptor/grpc.StreamClientInterceptor, stats.Handler, and balancer.Picker, so no additional plumbing is necessary.

C++

gRPC C++ will add an API to ClientContext.

struct TelemetryLabel {
  std::string value;
};

class ClientContext {
 public:
  // ...

  // New method to set an arbitrary context element.
  // Only works for certain allowed context types
  // (currently, only TelemetryLabel).
  template <typename T>
  void SetContext(T element);
};

Metric Stability

The label added in this proposal will start as experimental. The label is intended to always be explicitly enabled by the application; even once stable, it will not be enabled by default.

Temporary environment variable protection

The new label is only enabled via an API call, so no environment variable protection is necessary.

Rationale

A single label is restrictive. We could allow applications to have multiple (e.g., custom_label1, custom_label2, custom_label3). However, a single label provides a substantial amount of the value to applications. This gRFC is optimized for expediency and low complexity, and adding more labels has diminishing returns. Also, more custom labels can be added in a future gRFC with no additional complexity in gRPC compared to defining them now; it is a deferrable decision.

Since per-RPC metrics do not use the gRFC A79 metric architecture, it would technically be possible to support arbitrary user-defined labels. However, that adds implementation complexity and introduces inefficiencies in a hot path. We'd also like to leave open the door of implementations using the gRFC A79 metric architecture for per-RPC metrics, and supporting arbitrary user-defined metrics would be a significant a change to that API that wouldn't be taken lightly.

There is obvious overlap with baggage. As discussed in the Background section, baggage is relevant, but is considered a future enhancement for expediency.

Implementation

@ejona86 will be implementing immediately in grpc-java. A draft is available in PR 12552. Other languages will follow, with Go work starting potentially in a few weeks.