Page MenuHomePhabricator
Create Task
Maniphest T370668

New software: haproxykafka
Closed, ResolvedPublic
Actions

Description

Umbrella task to follow development of haproxykafka, a software meant to replace Benthos (currently just in testing phase in ulsfo) to parse HAProxy logs over socket and send to specific Kafka topic

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
team-traffic: removed haproxykafka critical alertoperations/alertsmaster+9 -27
cache:haproxy: longer capture buffers for relevant headersoperations/puppetproduction+6 -6
hiera: do not install haproxykafka on cloud instancesoperations/puppetproduction+136 -4
Renamed log fields for pipeline migration (haproxykafka)operations/puppetproduction+1 -1
cache:haproxy: hardcode $schema fieldoperations/puppetproduction+1 -1
hiera: continue haproxykafka tests on cp4037operations/puppetproduction+2 -2
hiera: disabling haproxy logging to socket (haproxykafka)operations/puppetproduction+2 -2
cache:haproxy: fixed the haproxykafka uds pathoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedFabfurT351117 Move analytics log from Varnish to HAProxy
 ResolvedFabfurT370668 New software: haproxykafka
 ResolvedFabfurT372338 haproxykafka: feature: Add prometheus metrics
 DeclinedFabfurT372341 haproxykafka: feature: Read from TCP UDS socket
 ResolvedFabfurT372339 haproxykafka: feature: add ability to add/remove/modify fields
 ResolvedFabfurT372342 haproxykafka: feature: Configuration file
 OpenFabfurT372344 haproxykafka: feature: Ability to print structured messages to stdout
 In ProgressFabfurT374128 haproxykafka minor features
 ResolvedFabfurT374473 Prepare puppet configuration to send haproxy logs to haproxykafka socket
 ResolvedFabfurT378330 Puppet spec tests for haproxykafka module
 ResolvedFabfurT374696 Enable prometheus metrics scraping for haproxykafka
 ResolvedFabfurT377613 Provide Debian packetization
 ResolvedFabfurT377614 Puppet configuration for haproxykafka
 ResolvedFabfurT378758 Set CPU affinity for haproxykafka process
 ResolvedFabfurT378578 Rollout haproxykafka on all hosts
 ResolvedFabfurT379611 Investigate transport errors in eqsin
 ResolvedFabfurT379237 haproxykafka hardening
 OpenFabfurT379776 Enable SSL client authentication on haproxykafka
 ResolvedFabfurT380373 Allow TLS authenticated client to write on new topics
 ResolvedJAllemandouT387454 Add HAproxy termination field to webrequest
 ResolvedJAllemandouT388397 Make webrequest_frontend being ingested using the in-data `dt` field
 ResolvedFabfurT389521 Update HAProxyKafka kafka-timestamp type
 OpenFabfurT386801 Split MessageBuffer configuration for different processing channels
 ResolvedFabfurT391810 Replicate current low-message alerting from VarnishKafka
 ResolvedFabfurT393016 haproxykafka service isn't restarted when upgraded

Event Timeline

Fabfur created this task.Jul 22 2024, 4:10 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJul 22 2024, 4:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ottomata subscribed.EditedJul 22 2024, 4:27 PM

I might be out of my league here, but have yall considered the haproxy Stream Processing Offload Engine?

It looks like a built in way to hook into haproxy requests and do whatever you need. This might be more efficient than listening to a UDP socket with syslog formatted messages. There looks to be SPOE agent libraries in C. Maybe you could even re-use some varnishkafka code to bridge between the SPOE protocol and librdkafka?

Fabfur updated the task description. (Show Details)Jul 22 2024, 4:36 PM
Vgutierrez added a comment.Jul 22 2024, 4:39 PM

https://github.com/negasus/haproxy-spoe-go could be handy if we go down that road and we don't want to get dirty writing C code :)

Fabfur added a comment.Jul 23 2024, 4:48 AM
In T370668#10003489, @Ottomata wrote:

I might be out of my league here, but have yall considered the haproxy Stream Processing Offload Engine?

It looks like a built in way to hook into haproxy requests and do whatever you need. This might be more efficient than listening to a UDP socket with syslog formatted messages. There looks to be SPOE agent libraries in C. Maybe you could even re-use some varnishkafka code to bridge between the SPOE protocol and librdkafka?

Hi Andrew, this is definitely something worth considering, but I would need a serious help w/ C.
ATM we decided to go down the road of rewriting atskafka in it's semplicity to adapt it to HAProxy with structured log format (RFC5424),

In the meantime I'll investigate into SPOE... thanks!

Ottomata added a comment.Jul 23 2024, 6:35 PM

I would need a serious help w/ C.

Ya, me too! Perhaps the SPOE go lib @Vgutierrez mentioned might be easier?

ATM we decided to go down the road of rewriting atskafka in it's semplicity to adapt it to HAProxy with structured log format (RFC5424),

If it works, then it works! I remember when we hired Magnus Edenhill to write varnishkafka, he did a lot of very specific performance and memory focused tuning to make sure stuff would work well. (and um, and I wonder if we could contract him out again? :D Probably not; he's more famous now.)

In either case, we'll be using librdkafka underneath. Perhaps RFC5424 format vs SPOP won't really make much of a performance difference.

• VirginiaPoundstone moved this task from Incoming to NEEDS DISCUSSION on the Test Kitchen board.Jul 26 2024, 4:14 PM
fgiunchedi removed a project: Observability-Logging.Jul 31 2024, 10:09 AM

untagging o11y since this is unrelated

CDanis subscribed.Aug 12 2024, 7:28 PM
Fabfur added a subtask: T372339: haproxykafka: feature: add ability to add/remove/modify fields.Aug 12 2024, 7:32 PM
Fabfur closed subtask T372341: haproxykafka: feature: Read from TCP UDS socket as Declined.Aug 12 2024, 7:38 PM
Fabfur mentioned this in T358109: Install new Benthos instance on cp hosts.Aug 21 2024, 9:56 AM
Fabfur mentioned this in T365968: Install benthos on single esams host to check performances under higher load.
Fabfur mentioned this in T364379: Benthos loses messages when under high load.
Fabfur mentioned this in T360454: Better Benthos performances.
Fabfur mentioned this in T361845: Add metrics to Benthos.
Fabfur mentioned this in T359178: Check statsv and eventlogging VarnishKafka instances.Aug 21 2024, 9:58 AM
Fabfur mentioned this in T365117: HAProxy log format doesn't support "invalid" request path.
Fabfur mentioned this in T365441: Umbrella task for Benthos parsing error.
Fabfur mentioned this in T366031: Upgrade Benthos package on cp hosts.
BTullis subscribed.Aug 21 2024, 10:04 AM
Fabfur closed subtask T372339: haproxykafka: feature: add ability to add/remove/modify fields as Resolved.Sep 5 2024, 3:50 PM
Fabfur closed subtask T372342: haproxykafka: feature: Configuration file as Resolved.
gerritbot added a comment.Sep 11 2024, 9:56 AM

Change #1072158 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] Fixed the haproxykafka uds path to reflect test configuration

https://gerrit.wikimedia.org/r/1072158

gerritbot added a project: Patch-For-Review.Sep 11 2024, 9:56 AM
gerritbot added a comment.Sep 11 2024, 10:08 AM

Change #1072158 merged by Fabfur:

[operations/puppet@production] cache:haproxy: fixed the haproxykafka uds path

https://gerrit.wikimedia.org/r/1072158

gerritbot added a comment.Sep 11 2024, 11:23 AM

Change #1072172 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: disabling haproxy logging to socket (haproxykafka)

https://gerrit.wikimedia.org/r/1072172

gerritbot added a comment.Sep 11 2024, 2:11 PM

Change #1072172 merged by Fabfur:

[operations/puppet@production] hiera: disabling haproxy logging to socket (haproxykafka)

https://gerrit.wikimedia.org/r/1072172

Stashbot added a comment.Sep 11 2024, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-11T14:14:25Z] <fabfur> reverted 1072172 and repooling cp4037 (T370668)

gerritbot added a comment.Sep 12 2024, 8:15 AM

Change #1072484 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: continue haproxykafka tests on cp4037

https://gerrit.wikimedia.org/r/1072484

gerritbot added a comment.Sep 12 2024, 1:22 PM

Change #1072484 merged by Fabfur:

[operations/puppet@production] hiera: continue haproxykafka tests on cp4037

https://gerrit.wikimedia.org/r/1072484

gerritbot added a comment.Sep 12 2024, 5:11 PM

Change #1072577 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache:haproxy: hardcode $schema field

https://gerrit.wikimedia.org/r/1072577

Fabfur closed subtask T372338: haproxykafka: feature: Add prometheus metrics as Resolved.Sep 12 2024, 8:31 PM
gerritbot added a comment.Sep 13 2024, 7:58 AM

Change #1072577 merged by Fabfur:

[operations/puppet@production] cache:haproxy: hardcode $schema field

https://gerrit.wikimedia.org/r/1072577

gerritbot added a comment.Sep 20 2024, 1:02 PM

Change #1074414 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] Renamed log field for pipeline migration (haproxykafka)

https://gerrit.wikimedia.org/r/1074414

gerritbot added a comment.Oct 15 2024, 8:51 AM

Change #1074414 merged by Fabfur:

[operations/puppet@production] Renamed log fields for pipeline migration (haproxykafka)

https://gerrit.wikimedia.org/r/1074414

Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Oct 17 2024, 5:57 PM
Fabfur changed the task status from Open to In Progress.Oct 30 2024, 9:45 AM
Fabfur triaged this task as High priority.
Fabfur changed the status of subtask T374128: haproxykafka minor features from Open to In Progress.
Fabfur closed subtask T374473: Prepare puppet configuration to send haproxy logs to haproxykafka socket as Resolved.
Fabfur closed subtask T374696: Enable prometheus metrics scraping for haproxykafka as Resolved.
Fabfur changed the status of subtask T377613: Provide Debian packetization from Open to In Progress.
Fabfur changed the status of subtask T377614: Puppet configuration for haproxykafka from Open to In Progress.Oct 30 2024, 9:47 AM
Fabfur closed subtask T377613: Provide Debian packetization as Resolved.Nov 6 2024, 8:45 AM
Fabfur closed subtask T377614: Puppet configuration for haproxykafka as Resolved.
gerritbot added a comment.Nov 7 2024, 9:50 AM

Change #1088244 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: do not install haproxykafka on cloud instances

https://gerrit.wikimedia.org/r/1088244

gerritbot added a comment.Nov 12 2024, 3:40 PM

Change #1088244 merged by Fabfur:

[operations/puppet@production] hiera: do not install haproxykafka on cloud instances

https://gerrit.wikimedia.org/r/1088244

Fabfur closed subtask T379237: haproxykafka hardening as Resolved.Nov 13 2024, 4:00 PM
gerritbot added a comment.Dec 3 2024, 1:50 PM

Change #1100113 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache:haproxy: Accept-Language header size to 96

https://gerrit.wikimedia.org/r/1100113

gerritbot added a comment.Dec 4 2024, 3:09 PM

Change #1100113 merged by Fabfur:

[operations/puppet@production] cache:haproxy: longer capture buffers for relevant headers

https://gerrit.wikimedia.org/r/1100113

Fabfur changed the status of subtask T378578: Rollout haproxykafka on all hosts from Open to In Progress.Dec 6 2024, 1:24 PM
Krinkle added a project: MediaWiki-Platform-Team (Radar).Dec 19 2024, 5:22 PM
Krinkle subscribed.
Krinkle mentioned this in T355837: Add Prometheus support to statsd.js via mw.track().Dec 19 2024, 6:00 PM
dr0ptp4kt subscribed.Dec 19 2024, 6:17 PM
• VirginiaPoundstone added a project: Data-Engineering-Radar.Jan 6 2025, 9:13 PM
• VirginiaPoundstone removed a project: Test Kitchen.Jan 6 2025, 9:28 PM
Fabfur closed subtask T378578: Rollout haproxykafka on all hosts as Resolved.Jan 29 2025, 8:45 AM
Fabfur added a subtask: T389521: Update HAProxyKafka kafka-timestamp type.Mar 21 2025, 9:07 AM
Fabfur closed subtask T389521: Update HAProxyKafka kafka-timestamp type as Resolved.Mar 21 2025, 11:32 AM
Fabfur added a subtask: T379776: Enable SSL client authentication on haproxykafka.Apr 3 2025, 8:31 AM
Fabfur added a subtask: T387454: Add HAproxy termination field to webrequest.
Fabfur added a subtask: T388397: Make webrequest_frontend being ingested using the in-data `dt` field.
Fabfur added a subtask: T386801: Split MessageBuffer configuration for different processing channels.
Ahoelzl closed subtask T388397: Make webrequest_frontend being ingested using the in-data `dt` field as Resolved.Apr 3 2025, 3:21 PM
Fabfur closed subtask T393016: haproxykafka service isn't restarted when upgraded as Resolved.May 5 2025, 3:06 PM
Fabfur closed subtask T387454: Add HAproxy termination field to webrequest as Resolved.May 9 2025, 11:14 AM
Fabfur added a project: HaproxyKafka.May 20 2025, 11:06 AM
Fabfur closed this task as Resolved.Jul 25 2025, 11:15 AM

Closing as we can use the HaproxyKafka component to keep track of related tasks

gerritbot added a comment.Sep 1 2025, 1:59 PM

Change #1183689 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/alerts@master] team-traffic: raise haproxykafka alert thresholds

https://gerrit.wikimedia.org/r/1183689

gerritbot added a comment.Sep 8 2025, 1:59 PM

Change #1183689 merged by Fabfur:

[operations/alerts@master] team-traffic: removed haproxykafka critical alert

https://gerrit.wikimedia.org/r/1183689

Fabfur closed subtask T391810: Replicate current low-message alerting from VarnishKafka as Resolved.Sep 11 2025, 8:04 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits