Page MenuHomePhabricator
Create Task
Maniphest T382571

[HAProxy migration] HAProxy and VarnishKafka should produce compatible datasets
Closed, ResolvedPublic
Actions

Description

We need to get a sign off validation plan .

The validation and analysis as been implemented in jupyter notebooks.

Through December 2024 we run validation on multiple hourly haproxykafka samples generated on ulsfo and we report that for all fields datasets match with >99.9 coverage.
When we see differences, it's typically haproxykafka reporting more records (as expected). Traffic volumes (distributions of responses by status code) also match across the two systems.

Dataset were validated both for the refined dataset, as well as kafka topics. Details and steps to reproduce the analysis are available at https://gitlab.wikimedia.org/repos/data-engineering/webrequest-haproxy-logs

Note that due to the sensitive nature of this data (PII) specific output has been omitted.

Details

Other Assignee
JAllemandou
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cache: use fqdn in haproxykafka hostnameoperations/puppetproduction+7 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

gmodena created this task.Dec 20 2024, 7:54 AM
CodeReviewBot added a project: Patch-For-Review.Dec 20 2024, 2:47 PM

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/webrequest-haproxy-logs/-/merge_requests/3

cleanup notebooks

CodeReviewBot added a comment.Dec 20 2024, 2:50 PM

gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/webrequest-haproxy-logs/-/merge_requests/3

cleanup notebooks

Maintenance_bot removed a project: Patch-For-Review.Dec 20 2024, 3:30 PM
Ahoelzl edited projects, added Data-Engineering (Q3 2025 January 1st - March 31th); removed Data-Engineering (Q2 2024 October 1st - December 31th).Jan 18 2025, 12:30 AM
Ahoelzl assigned this task to JAllemandou.Jan 24 2025, 6:33 PM
Ahoelzl updated Other Assignee, added: Antoine_Quhen.
Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q3 2025 January 1st - March 31th) board.
JAllemandou added a comment.Feb 12 2025, 10:37 AM

I've gone through more data validation (webrequest_text only, 2025-02-06T06:00) and found some things to discuss with @Fabfur.

What's expected:

  • The number of requests per host: between 1 and 10% more requests per host from HAproxy than VK, for all hosts.
  • The number of 301 responses is a lot bigger from HAproxy than VK - HAProxy send us tls-redirect rows.

What's not expected

  • The number of requests sending 20X and 30X except 301 response codes is lower on HAProxy than it is on VK, less than 0.5% (still represents almost 1M requests). The difference is stable across hosts and time.
  • Some requests' field don't match with a fairly big amount of volume (~10%). The two culprits are uri_host, which seems to be normalized/changed in VK, and accept_language that also seems to be normalized in VK.

Other questions:

  • The response_size returned by HAProxy is consistantly lower than the one from VK - Could it be compression?
  • When HAProxy logs a 400, the request is still passed to Varnish and we can get a different response - What does the client see? A Bad request response or the response Varnish has generated?
JAllemandou mentioned this in T354694: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition.Feb 12 2025, 10:37 AM
Ahoelzl changed the status of subtask T386343: [HAProxy migration] Validate `pageview` and `unique_devices` generated from `webrequest_frontend` from Open to In Progress.Feb 20 2025, 5:43 PM
Ahoelzl renamed this task from Haproxy kafka and varnishkafka produce compatible datasets to [HAProxy migration] HAProxy and VarnishKafka should produce compatible datasets.Feb 20 2025, 6:07 PM
CodeReviewBot added a project: Patch-For-Review.Feb 24 2025, 12:38 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1086

Draft: Generate HAProxy-webrequest downstreams

Ahoelzl added a project: DPE HAProxy Migration.Feb 24 2025, 6:21 PM
JAllemandou mentioned this in T387454: Add HAproxy termination field to webrequest.Feb 27 2025, 1:38 PM
Ahoelzl added a subtask: T387749: [HAProxy migration] Fix webrequest_frontend deletion job.Mar 3 2025, 3:57 PM
JAllemandou added a subtask: T387750: [HAProxy migration] Compile expected migration delta, switch over plan and communicate.Mar 10 2025, 11:50 AM
Ahoelzl reassigned this task from JAllemandou to Antoine_Quhen.Mar 11 2025, 8:57 PM
Ahoelzl updated Other Assignee, added: JAllemandou; removed: Antoine_Quhen.
CodeReviewBot added a comment.Mar 17 2025, 8:55 PM

aqu closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1086

Draft: Generate HAProxy-webrequest downstreams

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2025, 9:30 PM
CodeReviewBot added a project: Patch-For-Review.Mar 18 2025, 5:44 PM

aqu reopened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1086

Draft: Generate HAProxy-webrequest downstreams

CodeReviewBot added a comment.Mar 19 2025, 9:39 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1086

Generate HAProxy-webrequest downstreams

JAllemandou closed subtask T387749: [HAProxy migration] Fix webrequest_frontend deletion job as Resolved.Mar 26 2025, 8:47 AM
JAllemandou moved this task from In progress to Done on the Data-Engineering (Q3 2025 January 1st - March 31th) board.Mar 26 2025, 8:51 AM
JAllemandou removed a subtask: T387750: [HAProxy migration] Compile expected migration delta, switch over plan and communicate.Mar 26 2025, 8:55 AM
Ahoelzl closed this task as Resolved.Apr 3 2025, 3:22 PM
gerritbot added a comment.Apr 15 2025, 11:01 AM

Change #1136679 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache: use fqdn in syslog hostname

https://gerrit.wikimedia.org/r/1136679

gerritbot added a comment.Apr 28 2025, 3:56 PM

Change #1136679 merged by Fabfur:

[operations/puppet@production] cache: use fqdn in haproxykafka hostname

https://gerrit.wikimedia.org/r/1136679

Stashbot added a comment.Apr 29 2025, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T08:25:57Z] <fabfur> rolling restart haproxykafka on A:cp to apply new configuration https://gerrit.wikimedia.org/r/c/operations/puppet/+/1136679 (T382571)

Fabfur added a project: HaproxyKafka.May 20 2025, 11:07 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits