Page MenuHomePhabricator

Install new Benthos instance on cp hosts
Open, In Progress, MediumPublic

Description

Canary hosts:

  • cp4037 for the text cluster
  • cp4045 for the upload cluster

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+19 -0
operations/puppetproduction+5 -2
operations/puppetproduction+2 -2
operations/puppetproduction+4 -1
operations/puppetproduction+2 -1
operations/puppetproduction+3 -10
operations/puppetproduction+10 -8
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+8 -8
operations/puppetproduction+1 -1
operations/puppetproduction+3 -2
operations/puppetproduction+54 -208
operations/puppetproduction+6 -6
operations/puppetproduction+52 -0
operations/puppetproduction+9 -5
operations/puppetproduction+50 -0
operations/puppetproduction+50 -0
operations/puppetproduction+25 -0
operations/puppetproduction+1 -1
operations/puppetproduction+10 -4
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+9 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+3 -1
operations/puppetproduction+1 -1
operations/puppetproduction+11 -3
operations/puppetproduction+1 -1
operations/puppetproduction+3 -2
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+20 -3
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+5 -2
operations/puppetproduction+5 -0
operations/puppetproduction+27 -9
operations/puppetproduction+1 -1
operations/puppetproduction+7 -10
operations/puppetproduction+3 -3
operations/puppetproduction+216 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 1009293 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: enable log to benthos socket

https://gerrit.wikimedia.org/r/1009293

Change 1009293 merged by Fabfur:

[operations/puppet@production] haproxy: enable log to benthos socket

https://gerrit.wikimedia.org/r/1009293

Change 1009485 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: fix missing header parsing

https://gerrit.wikimedia.org/r/1009485

Change 1009485 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: fix missing header parsing

https://gerrit.wikimedia.org/r/1009485

Mentioned in SAL (#wikimedia-operations) [2024-03-07T14:56:33Z] <fabfur> repool cp4037 for very short time to process and collect logs from HAProxy/Benthos (T358109)

Mentioned in SAL (#wikimedia-operations) [2024-03-08T10:30:02Z] <fabfur@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on cp4037.ulsfo.wmnet with reason: T358109

Mentioned in SAL (#wikimedia-operations) [2024-03-08T10:30:17Z] <fabfur@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cp4037.ulsfo.wmnet with reason: T358109

Change 1009722 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: fixe metadata field

https://gerrit.wikimedia.org/r/1009722

Change 1009724 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: fix parsing for possible missing headers

https://gerrit.wikimedia.org/r/1009724

Change 1009769 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: temporary disable haproxy logging to benthos for cp4037

https://gerrit.wikimedia.org/r/1009769

Change 1009769 merged by Fabfur:

[operations/puppet@production] hiera: temporary disable haproxy logging to benthos for cp4037

https://gerrit.wikimedia.org/r/1009769

Mentioned in SAL (#wikimedia-operations) [2024-03-18T12:08:31Z] <fabfur> disabling puppet and depooling cp4037 to gradually apply new HAProxy/Benthos configuration (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1011453) T358109

Change 1009722 merged by Fabfur:

[operations/puppet@production] benthos: fixe metadata field

https://gerrit.wikimedia.org/r/1009722

Change 1009724 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: fix parsing for possible missing headers

https://gerrit.wikimedia.org/r/1009724

Mentioned in SAL (#wikimedia-operations) [2024-03-18T16:27:37Z] <fabfur> repooling cp4037 for very short time to collect HAProxy logs with Benthos (T358109)

Change 1012624 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: avoid UA header truncation

https://gerrit.wikimedia.org/r/1012624

Fabfur changed the task status from Open to In Progress.Mar 19 2024, 10:55 AM
Fabfur triaged this task as Medium priority.

Change 1012625 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: change Benthos prometheus port to avoid conflicts

https://gerrit.wikimedia.org/r/1012625

Change 1012625 merged by Fabfur:

[operations/puppet@production] benthos: change Benthos prometheus port to avoid conflicts

https://gerrit.wikimedia.org/r/1012625

Change 1012628 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: provide fqdn as hostname to backward compatibility

https://gerrit.wikimedia.org/r/1012628

Change 1012624 merged by Fabfur:

[operations/puppet@production] haproxy: avoid UA header truncation

https://gerrit.wikimedia.org/r/1012624

Mentioned in SAL (#wikimedia-operations) [2024-03-19T15:14:58Z] <fabfur> repooling cp4037 for brief time (T358109)

Change 1012628 merged by Fabfur:

[operations/puppet@production] benthos: provide fqdn as hostname to backward compatibility

https://gerrit.wikimedia.org/r/1012628

Mentioned in SAL (#wikimedia-operations) [2024-03-19T16:54:06Z] <fabfur> repooling cp4037 for brief time (T358109)

Mentioned in SAL (#wikimedia-operations) [2024-03-19T17:19:55Z] <fabfur> repooling cp4037 for brief time (T358109)

Change rOMWC101300810221 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: fix hiera lookup

https://gerrit.wikimedia.org/r/1013008

Change 1013034 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: ensure sequence field is an INT

https://gerrit.wikimedia.org/r/1013034

Change 1013034 merged by Fabfur:

[operations/puppet@production] benthos: ensure sequence field is an INT

https://gerrit.wikimedia.org/r/1013034

Change rOMWC101300810221 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: fix hiera lookup

https://gerrit.wikimedia.org/r/1013008

Mentioned in SAL (#wikimedia-operations) [2024-03-20T15:30:05Z] <fabfur> repooling cp4037 for a little longer than last time (T358109)

Change 1013107 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: force sequence key to be casted as INT

https://gerrit.wikimedia.org/r/1013107

Change 1013107 merged by Fabfur:

[operations/puppet@production] benthos: force sequence key to be casted as INT

https://gerrit.wikimedia.org/r/1013107

Change 1013114 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: add parameter for optional log length

https://gerrit.wikimedia.org/r/1013114

Change 1013114 merged by Fabfur:

[operations/puppet@production] haproxy: add parameter for optional log length

https://gerrit.wikimedia.org/r/1013114

Mentioned in SAL (#wikimedia-operations) [2024-03-21T08:40:37Z] <fabfur> repooling cp4037 for about ~30m (T358109)

Change 1013225 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: using URIPATH and URIPARAM for parsing corresponding fields

https://gerrit.wikimedia.org/r/1013225

Change 1013225 merged by Fabfur:

[operations/puppet@production] benthos: using URIPATH and URIPARAM for parsing corresponding fields

https://gerrit.wikimedia.org/r/1013225

Change 1013228 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: uri_query should be optional

https://gerrit.wikimedia.org/r/1013228

Change 1013228 merged by Fabfur:

[operations/puppet@production] benthos: uri_query should be optional

https://gerrit.wikimedia.org/r/1013228

Mentioned in SAL (#wikimedia-operations) [2024-03-21T10:00:13Z] <fabfur> repooling cp4037 for about ~30m (this is last time I'll notice here, no need for this in the future) (T358109)

Change 1013232 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: fix optional space in grok pattern (when no uri_query present)

https://gerrit.wikimedia.org/r/1013232

Change 1013232 merged by Fabfur:

[operations/puppet@production] benthos: fix optional space in grok pattern (when no uri_query present)

https://gerrit.wikimedia.org/r/1013232

Change 1013275 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: allow truncated http protocol version

https://gerrit.wikimedia.org/r/1013275

Change 1013278 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: added $schema key to unit tests

https://gerrit.wikimedia.org/r/1013278

Change #1013275 abandoned by Fabfur:

[operations/puppet@production] benthos: allow truncated http protocol version

Reason:

superseded by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1013341

https://gerrit.wikimedia.org/r/1013275

Change #1013278 abandoned by Fabfur:

[operations/puppet@production] benthos: added $schema key to unit tests

Reason:

https://gerrit.wikimedia.org/r/1013278

Change #1013503 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: add $schema key to unit tests

https://gerrit.wikimedia.org/r/1013503

Change #1013510 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: drop messages containing specific BADREQ pattern

https://gerrit.wikimedia.org/r/1013510

Change #1013503 merged by Fabfur:

[operations/puppet@production] benthos: add $schema key to unit tests

https://gerrit.wikimedia.org/r/1013503

Change #1013510 merged by Fabfur:

[operations/puppet@production] benthos: drop messages containing specific BADREQ pattern

https://gerrit.wikimedia.org/r/1013510

Change #1013526 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: enable benthos instance on upload host (cp4045)

https://gerrit.wikimedia.org/r/1013526

Change #1013967 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: fixed typo in log-format

https://gerrit.wikimedia.org/r/1013967

Change #1013967 merged by Fabfur:

[operations/puppet@production] haproxy: fixed typo in log-format

https://gerrit.wikimedia.org/r/1013967

Change #1013526 merged by Fabfur:

[operations/puppet@production] benthos: enable benthos instance on upload host (cp4045)

https://gerrit.wikimedia.org/r/1013526

Change #1015262 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: enable benthos on two new hosts (text|upload)

https://gerrit.wikimedia.org/r/1015262

Change #1015262 merged by Fabfur:

[operations/puppet@production] benthos: enable benthos on two new hosts (text|upload)

https://gerrit.wikimedia.org/r/1015262

Mentioned in SAL (#wikimedia-operations) [2024-03-28T09:27:16Z] <fabfur> temp depooled cp4038 and cp4046 to install benthos (T358109)

Change #1016306 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: add 2 more hosts

https://gerrit.wikimedia.org/r/1016306

Change #1016306 merged by Fabfur:

[operations/puppet@production] benthos: add 2 more hosts

https://gerrit.wikimedia.org/r/1016306

Change #1016760 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: add BENTHOS_SOURCE envvar

https://gerrit.wikimedia.org/r/1016760

Change #1016760 merged by Fabfur:

[operations/puppet@production] benthos: add BENTHOS_SOURCE envvar

https://gerrit.wikimedia.org/r/1016760

Change #1017018 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: add two new hosts (upload and text)

https://gerrit.wikimedia.org/r/1017018

Change #1017018 merged by Fabfur:

[operations/puppet@production] benthos: add two new hosts (upload and text)

https://gerrit.wikimedia.org/r/1017018

Change #1017090 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: fix unit tests to reflect recent changes in schema

https://gerrit.wikimedia.org/r/1017090

Change #1017090 merged by Fabfur:

[operations/puppet@production] benthos: fix unit tests to reflect recent changes in schema

https://gerrit.wikimedia.org/r/1017090

Change #1019765 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: enable Benthos logging on all ulsfo cp hosts

https://gerrit.wikimedia.org/r/1019765

Change #1019765 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: enable Benthos logging on all ulsfo cp hosts

https://gerrit.wikimedia.org/r/1019765

Change #1020692 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: added some labels

https://gerrit.wikimedia.org/r/1020692

Change #1020695 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: fix check for possible empty values

https://gerrit.wikimedia.org/r/1020695

Change #1020692 merged by Fabfur:

[operations/puppet@production] benthos: added some labels

https://gerrit.wikimedia.org/r/1020692

Change #1020695 merged by Fabfur:

[operations/puppet@production] benthos: fix check for possible empty values

https://gerrit.wikimedia.org/r/1020695

Change #1021412 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy/benthos: uppercase keyx parameter in X-Analytics-TLS hdr

https://gerrit.wikimedia.org/r/1021412

Change #1021412 merged by Fabfur:

[operations/puppet@production] haproxy/benthos: uppercase keyx parameter in X-Analytics-TLS hdr

https://gerrit.wikimedia.org/r/1021412

Change #1021505 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: using hiera aliases for benthos socket address

https://gerrit.wikimedia.org/r/1021505

Change #1021517 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos/haproxy: include haproxy current pid in messages

https://gerrit.wikimedia.org/r/1021517

Change #1021517 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: include haproxy current pid in messages

https://gerrit.wikimedia.org/r/1021517

Change #1021863 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: ensure haproxy_pid is interpreted as number

https://gerrit.wikimedia.org/r/1021863

Change #1021863 merged by Fabfur:

[operations/puppet@production] benthos: ensure haproxy_pid is interpreted as number

https://gerrit.wikimedia.org/r/1021863

Change #1021873 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: fix conf to actually transform haproxy_pid field into number

https://gerrit.wikimedia.org/r/1021873

Change #1021873 merged by Fabfur:

[operations/puppet@production] benthos: fix conf to actually transform haproxy_pid field into number

https://gerrit.wikimedia.org/r/1021873

Change #1023054 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: add envvar for buffer limit

https://gerrit.wikimedia.org/r/1023054

Change #1023060 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: buffer memory limit override for cp4037

https://gerrit.wikimedia.org/r/1023060

Change #1021505 merged by Fabfur:

[operations/puppet@production] benthos/haproxy: using hiera aliases for benthos socket address

https://gerrit.wikimedia.org/r/1021505

Change #1023324 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: applied benthos socket address alias to upload cluster too

https://gerrit.wikimedia.org/r/1023324

Change #1023324 merged by Fabfur:

[operations/puppet@production] hiera: applied benthos socket address alias to upload cluster too

https://gerrit.wikimedia.org/r/1023324

Change #1023054 merged by Fabfur:

[operations/puppet@production] hiera: add envvar for buffer limit

https://gerrit.wikimedia.org/r/1023054

Change #1023060 merged by Fabfur:

[operations/puppet@production] hiera: buffer memory limit increase for cp4037

https://gerrit.wikimedia.org/r/1023060

Mentioned in SAL (#wikimedia-operations) [2024-04-24T15:09:48Z] <fabfur> depooling cp4037 to test tls connection to kafka cluster (T358109)

Change #1023874 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos: kafka brokers certificates can be definitely trusted

https://gerrit.wikimedia.org/r/1023874

Change #1023874 merged by Fabfur:

[operations/puppet@production] benthos:haproxy_cache: kafka brokers certificates can be verified

https://gerrit.wikimedia.org/r/1023874

Change #1023879 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] benthos:haproxy_cache: pass root cas file path as envvar

https://gerrit.wikimedia.org/r/1023879