It occurred to me we should test benthos@webrequest_live with only one centrallog processing events, especially in light of the recent CPU/threads bump we might not have enough redundancy as it is
Description
Description
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-09-20T08:36:57Z] <godog> stop benthos@webrequest_live.service on centrallog1002 to test redudancy/capacity - T346871
Comment Actions
For reference, the dashboards me and @elukey are looking at:
- https://grafana.wikimedia.org/d/V0TSK7O4z/benthos?from=now-30m&orgId=1&to=now&var-port=4151
- https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=centrallog2002&var-datasource=thanos&var-cluster=syslog
- https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=centrallog1002&var-datasource=thanos&var-cluster=syslog&from=now-1h&to=now
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-09-20T08:47:09Z] <godog> temp bump threads to 15 for benthos@webrequest_live on centrallog2002 - T346871
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-09-20T08:59:59Z] <godog> restore benthos@webrequest_live running on both centrallog hosts - T346871
Comment Actions
Test was successful in the sense that we can keep processing the webrequest firehose from codfw (centrallog2002). Even though by taking a performance hit in terms of messages processed, of course also network bandwidth takes a significant hit, jumping ~ +25MB/s. I believe that in the unlikely event of one centrallog host being unavailable for multiple hours we can still get a reasonable webrequest sampled stream. If even that doesn't work for some reason, it is easy to spin up benthos on different hosts since it is all stateless and trivially horizontally scalable