Canary hosts:
- cp4037 for the text cluster
- cp4045 for the upload cluster
Canary hosts:
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
In Progress | Fabfur | T351117 Move analytics log from Varnish to HAProxy | |||
In Progress | Fabfur | T358109 Install new Benthos instance on cp hosts | |||
Resolved | Fabfur | T359172 Create a new temporary topic on Kafka jumbo cluster to test Benthos | |||
Resolved | Fabfur | T360450 Add $schema key to Benthos payload | |||
Open | Fabfur | T360454 Better Benthos performances | |||
Open | Fabfur | T360642 Remove extra fields currently sent to Kafka | |||
Open | Fabfur | T363420 Use definitive field naming during grok parsing instead of renaming later (in mapping) |
Change 1009293 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy: enable log to benthos socket
Change 1009293 merged by Fabfur:
[operations/puppet@production] haproxy: enable log to benthos socket
Change 1009485 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: fix missing header parsing
Change 1009485 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: fix missing header parsing
Mentioned in SAL (#wikimedia-operations) [2024-03-07T14:56:33Z] <fabfur> repool cp4037 for very short time to process and collect logs from HAProxy/Benthos (T358109)
Mentioned in SAL (#wikimedia-operations) [2024-03-08T10:30:02Z] <fabfur@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on cp4037.ulsfo.wmnet with reason: T358109
Mentioned in SAL (#wikimedia-operations) [2024-03-08T10:30:17Z] <fabfur@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cp4037.ulsfo.wmnet with reason: T358109
Change 1009722 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: fixe metadata field
Change 1009724 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: fix parsing for possible missing headers
Change 1009769 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: temporary disable haproxy logging to benthos for cp4037
Change 1009769 merged by Fabfur:
[operations/puppet@production] hiera: temporary disable haproxy logging to benthos for cp4037
Mentioned in SAL (#wikimedia-operations) [2024-03-18T12:08:31Z] <fabfur> disabling puppet and depooling cp4037 to gradually apply new HAProxy/Benthos configuration (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1011453) T358109
Change 1009722 merged by Fabfur:
[operations/puppet@production] benthos: fixe metadata field
Change 1009724 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: fix parsing for possible missing headers
Mentioned in SAL (#wikimedia-operations) [2024-03-18T16:27:37Z] <fabfur> repooling cp4037 for very short time to collect HAProxy logs with Benthos (T358109)
Change 1012624 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy: avoid UA header truncation
Change 1012625 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: change Benthos prometheus port to avoid conflicts
Change 1012625 merged by Fabfur:
[operations/puppet@production] benthos: change Benthos prometheus port to avoid conflicts
Change 1012628 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: provide fqdn as hostname to backward compatibility
Change 1012624 merged by Fabfur:
[operations/puppet@production] haproxy: avoid UA header truncation
Mentioned in SAL (#wikimedia-operations) [2024-03-19T15:14:58Z] <fabfur> repooling cp4037 for brief time (T358109)
Change 1012628 merged by Fabfur:
[operations/puppet@production] benthos: provide fqdn as hostname to backward compatibility
Mentioned in SAL (#wikimedia-operations) [2024-03-19T16:54:06Z] <fabfur> repooling cp4037 for brief time (T358109)
Mentioned in SAL (#wikimedia-operations) [2024-03-19T17:19:55Z] <fabfur> repooling cp4037 for brief time (T358109)
Change rOMWC101300810221 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: fix hiera lookup
Change 1013034 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: ensure sequence field is an INT
Change 1013034 merged by Fabfur:
[operations/puppet@production] benthos: ensure sequence field is an INT
Change rOMWC101300810221 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: fix hiera lookup
Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:30:05Z] <fabfur> repooling cp4037 for a little longer than last time (T358109)
Change 1013107 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: force sequence key to be casted as INT
Change 1013107 merged by Fabfur:
[operations/puppet@production] benthos: force sequence key to be casted as INT
Change 1013114 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy: add parameter for optional log length
Change 1013114 merged by Fabfur:
[operations/puppet@production] haproxy: add parameter for optional log length
Mentioned in SAL (#wikimedia-operations) [2024-03-21T08:40:37Z] <fabfur> repooling cp4037 for about ~30m (T358109)
Change 1013225 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: using URIPATH and URIPARAM for parsing corresponding fields
Change 1013225 merged by Fabfur:
[operations/puppet@production] benthos: using URIPATH and URIPARAM for parsing corresponding fields
Change 1013228 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: uri_query should be optional
Change 1013228 merged by Fabfur:
[operations/puppet@production] benthos: uri_query should be optional
Mentioned in SAL (#wikimedia-operations) [2024-03-21T10:00:13Z] <fabfur> repooling cp4037 for about ~30m (this is last time I'll notice here, no need for this in the future) (T358109)
Change 1013232 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: fix optional space in grok pattern (when no uri_query present)
Change 1013232 merged by Fabfur:
[operations/puppet@production] benthos: fix optional space in grok pattern (when no uri_query present)
Change 1013275 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: allow truncated http protocol version
Change 1013278 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: added $schema key to unit tests
Change #1013275 abandoned by Fabfur:
[operations/puppet@production] benthos: allow truncated http protocol version
Reason:
superseded by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1013341
Change #1013278 abandoned by Fabfur:
[operations/puppet@production] benthos: added $schema key to unit tests
Reason:
Change #1013503 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: add $schema key to unit tests
Change #1013510 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: drop messages containing specific BADREQ pattern
Change #1013503 merged by Fabfur:
[operations/puppet@production] benthos: add $schema key to unit tests
Change #1013510 merged by Fabfur:
[operations/puppet@production] benthos: drop messages containing specific BADREQ pattern
Change #1013526 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: enable benthos instance on upload host (cp4045)
Change #1013967 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy: fixed typo in log-format
Change #1013967 merged by Fabfur:
[operations/puppet@production] haproxy: fixed typo in log-format
Mentioned in SAL (#wikimedia-operations) [2024-03-25T11:08:07Z] <fabfur> depooling cp4045 to install && test benthos (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1013526) (T358109)
Change #1013526 merged by Fabfur:
[operations/puppet@production] benthos: enable benthos instance on upload host (cp4045)
Mentioned in SAL (#wikimedia-operations) [2024-03-25T11:18:59Z] <fabfur> *repooling* cp4045 with Benthos (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1013526) (T358109)
Change #1015262 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: enable benthos on two new hosts (text|upload)
Change #1015262 merged by Fabfur:
[operations/puppet@production] benthos: enable benthos on two new hosts (text|upload)
Mentioned in SAL (#wikimedia-operations) [2024-03-28T09:27:16Z] <fabfur> temp depooled cp4038 and cp4046 to install benthos (T358109)
Change #1016306 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: add 2 more hosts
Change #1016306 merged by Fabfur:
[operations/puppet@production] benthos: add 2 more hosts
Change #1016760 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: add BENTHOS_SOURCE envvar
Change #1016760 merged by Fabfur:
[operations/puppet@production] benthos: add BENTHOS_SOURCE envvar
Change #1017018 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: add two new hosts (upload and text)
Change #1017018 merged by Fabfur:
[operations/puppet@production] benthos: add two new hosts (upload and text)
Change #1017090 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: fix unit tests to reflect recent changes in schema
Change #1017090 merged by Fabfur:
[operations/puppet@production] benthos: fix unit tests to reflect recent changes in schema
Change #1019765 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: enable Benthos logging on all ulsfo cp hosts
Change #1019765 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: enable Benthos logging on all ulsfo cp hosts
Change #1020692 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: added some labels
Change #1020695 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: fix check for possible empty values
Change #1020692 merged by Fabfur:
[operations/puppet@production] benthos: added some labels
Change #1020695 merged by Fabfur:
[operations/puppet@production] benthos: fix check for possible empty values
Change #1021412 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] haproxy/benthos: uppercase keyx parameter in X-Analytics-TLS hdr
Change #1021412 merged by Fabfur:
[operations/puppet@production] haproxy/benthos: uppercase keyx parameter in X-Analytics-TLS hdr
Change #1021505 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: using hiera aliases for benthos socket address
Change #1021517 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos/haproxy: include haproxy current pid in messages
Change #1021517 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: include haproxy current pid in messages
Change #1021863 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: ensure haproxy_pid is interpreted as number
Change #1021863 merged by Fabfur:
[operations/puppet@production] benthos: ensure haproxy_pid is interpreted as number
Change #1021873 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: fix conf to actually transform haproxy_pid field into number
Change #1021873 merged by Fabfur:
[operations/puppet@production] benthos: fix conf to actually transform haproxy_pid field into number
Change #1023054 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: add envvar for buffer limit
Change #1023060 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: buffer memory limit override for cp4037
Change #1021505 merged by Fabfur:
[operations/puppet@production] benthos/haproxy: using hiera aliases for benthos socket address
Change #1023324 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: applied benthos socket address alias to upload cluster too
Change #1023324 merged by Fabfur:
[operations/puppet@production] hiera: applied benthos socket address alias to upload cluster too
Change #1023054 merged by Fabfur:
[operations/puppet@production] hiera: add envvar for buffer limit
Change #1023060 merged by Fabfur:
[operations/puppet@production] hiera: buffer memory limit increase for cp4037
Mentioned in SAL (#wikimedia-operations) [2024-04-24T15:09:48Z] <fabfur> depooling cp4037 to test tls connection to kafka cluster (T358109)
Change #1023874 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos: kafka brokers certificates can be definitely trusted
Change #1023874 merged by Fabfur:
[operations/puppet@production] benthos:haproxy_cache: kafka brokers certificates can be verified
Change #1023879 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] benthos:haproxy_cache: pass root cas file path as envvar