Revolutionizing Data Collection: Event Mesh Enhances OpenTelemetry with Scalable Pub/Sub Architecture
November 3, 2026
The architecture relies on an event mesh of interconnected message buses with persistent storage, enabling tier-2 collectors to process traces in batches after retrieval from the bus and reducing collector state while enabling horizontal scaling.
The design replaces point-to-point OpenTelemetry collector communication with a decoupled Pub/Sub event mesh, improving scalability, resilience, and flexibility in tail-based sampling.
In DLQ design, tier-1 publishes to specific topics with mandatory headers (Span Type, Trace ID); the message bus enforces a 30-second TTL, forwarding unconsumed messages to a Dead Letter Queue for tier-2 processing.
TTL plus DLQ behavior create a buffer window akin to traditional tail-based sampling, allowing dynamic adjustments to buffer duration without altering core architecture.
Tier-2 collection retrieves spans from both active and dead queues, reconstructs traces by trace ID, applies sampling rules, batches qualified traces for the APM backend, and discards non-qualified traces.
Three major components are tier-1 collectors publishing to the message bus, the message bus as the central data distribution and persistence layer, and tier-2 collectors subscribing to streams and applying sampling policies before exporting to the APM backend.
Optionally, Redis may be used as an in-memory cache for global trace IDs and metadata, with DLQ semantics to manage trace buffering within a configurable 30-second window.
The system is designed with stateless, active-active tier-1 and tier-2 collectors, an active-standby message bus for HA/DR, and batch processing to forward sampled traces to the APM backend while maintaining visibility for failed traces.
Summary based on 1 source
