Product teams are looking for a way to quickly view capacity numbers for event platform / kafka /event gate / the error logging stack (which is bottlenecked by logstash).
How should we gather this information, how do we package guidance (e.g. events per stream vs total events, etc) and make it available?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T259734 BUOD-KR1-Q4+: Certify that analytics schema and instruments have been upgraded to use the MEP system (clearing the legacy system for sunsetting) | |||
| Declined | Ottomata | T259157 BUOD-KR1-Q3: Require that all new schema/instruments are created with the MEP system | |||
| Resolved | Ottomata | T259155 Obtain evidence-based guidance on capacity for event streams |
According to @Ottomata , a good starting point is if > 1000 events / s, you need to talk to analytics engineering. There's no real limit, event gate and kafka are horizontally scalable. and we can change that capacity as needed. The other thing is, we can put a number on the total number of events going into e.g. an event gate instance, but from an individual instrument, it will not know how much of the total fraction would be used.
The quickest way to see per stream share usage of the EventGate instance for external clients (like EventLogging) is to check the EventGate grafana dashboard.