Change #1071915 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache:haproxy: introduce extended logging on socket for haproxykafka

https://gerrit.wikimedia.org/r/1071915

Mentioned in SAL (#wikimedia-operations) [2024-09-11T09:42:15Z] <fabfur> depooling cp4037 to test haproxykafka (T374473)

Change #1071915 merged by Fabfur:

[operations/puppet@production] cache:haproxy: introduce extended logging on socket for haproxykafka

https://gerrit.wikimedia.org/r/1071915

Hi,

Are you planning to start shipping to Kafka anytime soon? It would be great to touch base and iron out the timelines in case, just to stay on the same page.

Mentioned in SAL (#wikimedia-operations) [2024-09-11T10:50:21Z] <fabfur> repooling cp4037 to test haproxykafka (T374473)

In T374473#10136693, @gmodena wrote:

Hi,

Are you planning to start shipping to Kafka anytime soon? It would be great to touch base and iron out the timelines in case, just to stay on the same page.

Hi @gmodena sorry for the late answer, I'm testing it on a single host (cp4037) producing messages to the same topics we used during Benthos testing (webrequest_text_test and webrequest_text_test_error). I've not fully switched it on permanently, I'm just doing that for some minutes to gather all the data I need to fine tune the haproxykafka processing/configuration.

I'll let you know when it will be ready to produce messages to that topic and the relevant metrics, consider I'm using the same Benthos message transformations so there should be no big surprises in message format considering what we're used to...

Thanks!

Change #1072697 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: testing haproxykafka on cp4037

https://gerrit.wikimedia.org/r/1072697

@Fabfur thanks for the heads up. We'll need to coordinate a bit on roll out, because the previous ingestion and processing infra has been decommissioned as part of T372456: Rollback haproxy feed automated ingestion.

One thing we should revisit once you feel ready to deploy, is how varnishkafka will be switched off (DC-by-DC vs full fleet). IIRC the options we considered are summed up here (I should move it to Miro).

Change #1072697 merged by Fabfur:

[operations/puppet@production] hiera: testing haproxykafka on cp4037

https://gerrit.wikimedia.org/r/1072697

Change #1072714 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp3066 for testing

https://gerrit.wikimedia.org/r/1072714

Change #1072714 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on cp3066 for testing

https://gerrit.wikimedia.org/r/1072714

Change #1087359 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: haproxykafka defaults to 2 workers

https://gerrit.wikimedia.org/r/1087359

Change #1087359 merged by Fabfur:

[operations/puppet@production] hiera: haproxykafka defaults to 2 workers

https://gerrit.wikimedia.org/r/1087359

Change #1087365 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: hpk batch_deadline on socket set to 1s

https://gerrit.wikimedia.org/r/1087365

Change #1087365 merged by Fabfur:

[operations/puppet@production] hiera: hpk batch_deadline on socket set to 1s

https://gerrit.wikimedia.org/r/1087365

Change #1087371 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxykafka: restart service on config file changes

https://gerrit.wikimedia.org/r/1087371

Change #1087371 abandoned by Fabfur:

[operations/puppet@production] haproxykafka: restart service on config file changes

Reason:

not really needed, and can cause message loss

https://gerrit.wikimedia.org/r/1087371