contrib: add Kafka stats sink for direct metric production via librdkafka#44137
Open
prashanthjos wants to merge 3 commits intoenvoyproxy:mainfrom
Open
contrib: add Kafka stats sink for direct metric production via librdkafka#44137prashanthjos wants to merge 3 commits intoenvoyproxy:mainfrom
prashanthjos wants to merge 3 commits intoenvoyproxy:mainfrom
Conversation
…opics Add a new contrib stats sink (envoy.stat_sinks.kafka) that produces metrics directly to Apache Kafka topics via librdkafka, avoiding the memory overhead of intermediate gRPC collectors at scale. Features: - JSON and Protobuf serialization (Protobuf uses StreamMetricsMessage, wire-compatible with the metrics_service gRPC sink) - Configurable batching, delta/absolute counters, tag-as-label emission - Full librdkafka producer tuning via producer_config map, including TLS and SASL/SCRAM authentication support Signed-off-by: Prashanth Josyula <prashanth.16@gmail.com>
…opics (#XXXXX) Add a new contrib stats sink (envoy.stat_sinks.kafka) that produces metrics directly to Apache Kafka topics via librdkafka, avoiding the memory overhead of intermediate gRPC collectors at scale. Features: - JSON and Protobuf serialization (Protobuf uses StreamMetricsMessage, wire-compatible with the metrics_service gRPC sink) - Configurable batching, delta/absolute counters, tag-as-label emission - Full librdkafka producer tuning via producer_config map, including TLS and SASL/SCRAM authentication support Signed-off-by: Prashanth Josyula <prashanth.16@gmail.com>
…opics Add a new contrib stats sink (envoy.stat_sinks.kafka) that produces metrics directly to Apache Kafka topics via librdkafka, avoiding the memory overhead of intermediate gRPC collectors at scale. Features: - JSON and Protobuf serialization (Protobuf uses StreamMetricsMessage, wire-compatible with the metrics_service gRPC sink) - Configurable batching, delta/absolute counters, tag-as-label emission - Full librdkafka producer tuning via producer_config map, including TLS and SASL/SCRAM authentication support Signed-off-by: Prashanth Josyula <prashanth.16@gmail.com>
prashanthjos
changed the title
Contributor
Author
|
/retest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Message:
add Kafka stats sink for direct metric production to Kafka topics
Additional Description:
This adds a new contrib stats sink (
envoy.stat_sinks.kafka) that produces metrics directly to Apache Kafka topics via librdkafka. The motivation is to avoid the memory overhead of intermediate gRPC metrics collectors in large-scale deployments where the existingmetrics_servicesink creates significant memory pressure on collector infrastructure.The sink supports two serialization formats:
StreamMetricsMessagecontainingMetricFamilyentries, wire-compatible with the gRPCmetrics_servicesink, so existing consumers can deserialize without changes.Other features include configurable batching, delta or absolute counter reporting, tag-as-label emission, and full librdkafka producer tuning (TLS, SASL/SCRAM, compression,etc.) via the
producer_configmap.Risk Level: Low (new contrib extension, no changes to core Envoy code paths)
Testing: Unit tests for both JSON and Protobuf serialization covering counters, gauges, histograms, batching, delta/absolute values, label emission, empty snapshots, and config factory parsing. All tests pass locally.
Docs Changes: Added configuration docs, API reference toctree entry, and changelog entry.
Release Notes: Added to
changelogs/current.yamlundernew_features.Platform Specific Features: N/A