Following also https://github.com/haproxy/haproxy/issues/2612

After upgrading HAProxy to 2.8.10 on whole ulsfo we still see some errors in the kafka DLQ like:

Update on Benthos performances.

This has been reverted due to a bug in HAProxy regarding how log variables are escaped in the rfc5424 format using the +E flag.

A missing Host header in the request result in a 400 from Varnish and a parsing error from varnish:

Update: opened this issue upstream to have another opinion about this and, in case, fix this behavior

Some other information about this:

In T351117#9793797, @gmodena wrote:

In T351117#9781136, @Ottomata wrote:

adopt topic names that follow EP conventions: <dc>.<topic_name>

I'm sorry for not thinking about this earlier. There is a big of a design flaw in the use of data center as a topic prefix, and really, for topics that are never mirrored to other Kafka clusters, there is no need for topic prefixes at all.

I just added documentation about this here:
https://wikitech.wikimedia.org/wiki/Kafka#Data_center_topic_prefixing_design_flaw

Given that, and the ever expanding list of data centers, and the fact that webrequest is the only stream we have that is produced to from non main data centers, I think we should not use topic prefixing for webrequest.

All producers should use the same topic name, independent of which data center they are in.

Thanks for clarifying @Ottomata.

@Ottomata @Fabfur If we remove prefixing, there is a potential clash between varnishkafka and benthos topics.
How about we name the production Haproxy/benthos topics as follows?

• webrequest_frontent_text
• webrequest_frontent_text.error
• webrequest_frontent_upload
• webrequest_frontent_upload.error

I'd say this could be closed...

Little addendum: the latest Benthos release introduced the auto_replay_nacks configuration parameter in (all?) inputs: https://www.benthos.dev/docs/components/inputs/socket_server#auto_replay_nacks

@CDanis helped me a lot in this direction and he found a workaround|solution for this specific issue, optimizing Benthos configuration and using an higher UDP receive buffer to mitigate very high concurrent requests.

• there's couple of CRs pending (linked to this phab) and I'd like to have a second run on the event schema naming conventions (cc / @Fabfur). We might want to drop the webrequest_source since we don't currently use in ETL (it's inferred from the HDFS path, not schema).

The haproxy_id field has been added to messages.

In T351117#9726287, @gmodena wrote:

About the sequence issue, that's the most plausible hypotheses. We could append (or prepend) other information pieces to the sequence number (like the haproxy process id) to avoid duplicates but we couldn't guarantee the monotonic increase (or the increase, even) in this case. I suggest using this current approach for the moment and eventually rework later.

Ack and +1 to your proposal. IMHO it's easier to be resilient to reloads, than working around non-monotonicity. Do you maybe a feel for how often haproxy reloads are expected to happen once it prod? Could we assume they are sporadic events?

I agree with @gmodena on all topics, more specifically:

I'd like to join the chorus of thanks to Papaul, you resolved us a very nasty and long running issue here! Thanks again!

on Linux I usually use xdg-open

cp3066 has been reimaged successfully, no evidence of errors

esams has been repooled at 12:15UTC

ESAMS DC started depooling @05:58UTC

In T360642#9655231, @Ottomata wrote:

meta.id and meta.request_id

meta.id is used to uniquely identify an event, and it is usually used for deduplication.

User added to the ldap/wmf group, please let me know if you can use these services.

User added to the ldap/wmf group, please let me know if you can use these services.

Thanks, I'll notice you soon with the confirmation!

Hello, thanks for this request, could your direct manager please confirm this (it's sufficient to respond to this ticket).

In T360642#9653099, @gmodena wrote:

These are the fields that are sent from Benthos that aren't present in the current webrequest stream:

FWIW meta and $schema are not part of webrequest, but a requirements for EP integration: https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas#Required_event_data. Usually they are managed by our node/java producers.

We don't need for sure two uuid (generate by different parts of the processing pipeline), that are expensive to generate under heavy load and could result in a potential waste of bandwidth/space on Kafka.

I need to investigate the historical reason behind both meta.id and meta.request_id, but if performance is a concern I think we can live without meta.id. meta itself is a required field in the webrequest schema, but payload should validate with missing/empty id.

kafka-jumbo and hadoop should be fine (storage wise), but I do appreciate that at webrequest scale every byte sent over the wire counts (and adds up quickly).
For my own education, do you have some datapoints that show how much overhead uuidv4 in benthos produces (cpu-wise)?

Sent information about dmarc address privately (mail) to the ticket author

Even without metrics generation, this has been fixed with a small processing on the input side.
Leave the ticket in progress to add code for metric generation.