Currently logic responsible for appending the requestctl rule name to X-Analytics header is handled by Varnish. This means that requests that are filtered by HAProxy and not forwarded to Varnish are not visible from usual analytics tools.
A solution could be to append the requestctl rule name to X-Analytics directly in HAProxy if not set in the response already (by Varnish).
Pay also attention that the X-Analytics header is deleted as usual before response is sent to the client.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| haproxy: stop setting x-requestctl header, use txn var instead | repos/sre/hiddenparma!67 | oblivian | haproxy_logging | main |
Change #1164170 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] cache,haproxy: set requestctl in x-analytics if not set by varnish
Change #1164170 merged by Fabfur:
[operations/puppet@production] cache,haproxy: set requestctl in x-analytics if not set by varnish
Mentioned in SAL (#wikimedia-operations) [2025-06-26T15:12:47Z] <fabfur> temporary disable puppet on cp7001 (T397917)
Mentioned in SAL (#wikimedia-operations) [2025-06-26T15:34:29Z] <fabfur> repooling cp7001 (T397917)
Change #1164253 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] hiera: set requestctl in x-analytics if not set by varnish (cp7006)
Change #1164253 merged by Fabfur:
[operations/puppet@production] hiera: set requestctl in x-analytics if not set by varnish (cp7006)
Mentioned in SAL (#wikimedia-operations) [2025-06-26T16:30:16Z] <fabfur> depool cp7006 for a quick test (T397917)
Change #1164275 had a related patch set uploaded (by Fabfur; author: Fabfur):
[operations/puppet@production] cache,haproxy: use http-after-response capture for x-analytics
Mentioned in SAL (#wikimedia-operations) [2025-06-27T07:26:19Z] <fabfur> depool cp7007 for testing (T397917)
Mentioned in SAL (#wikimedia-operations) [2025-06-27T14:57:16Z] <fabfur> configuration on cp7007 reverted and host repooled (T397917)
Change #1164275 merged by Vgutierrez:
[operations/puppet@production] cache,haproxy: refactor haproxy captures to fix x-analytics logging
Change #1164952 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] Revert "cache,haproxy: refactor haproxy captures to fix x-analytics logging"
Change #1164952 merged by Vgutierrez:
[operations/puppet@production] Revert "cache,haproxy: refactor haproxy captures to fix x-analytics logging"
Change #1166167 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] cache,haproxy: Remove http response captures
oblivian opened https://gitlab.wikimedia.org/repos/sre/hiddenparma/-/merge_requests/67
haproxy: stop setting x-requestctl header, use txn var instead
oblivian merged https://gitlab.wikimedia.org/repos/sre/hiddenparma/-/merge_requests/67
haproxy: stop setting x-requestctl header, use txn var instead
Change #1166167 merged by Vgutierrez:
[operations/puppet@production] cache,haproxy: Remove http response captures
Mentioned in SAL (#wikimedia-operations) [2025-07-07T07:25:42Z] <vgutierrez> depooling cp7006 to test Ia82b93 - T397917
Mentioned in SAL (#wikimedia-operations) [2025-07-07T07:53:23Z] <vgutierrez> repooling cp7006 with Ia82b93 applied - T397917
Change #1166775 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] cache::haproxy: Fix requestctl= sanitization
Change #1166775 merged by Vgutierrez:
[operations/puppet@production] cache::haproxy: Fix requestctl= sanitization
Change #1167200 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] Revert "cache,haproxy: Remove http response captures"
Change #1167203 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):
[operations/puppet@production] cache::haproxy: Replace res.hdr() with res.fhdr()
Change #1167203 merged by Vgutierrez:
[operations/puppet@production] cache::haproxy: Replace hdr() with fhdr()
Change #1167200 abandoned by Vgutierrez:
[operations/puppet@production] Revert "cache,haproxy: Remove http response captures"
Reason:
not needed anymore, issue fixed :)
During the deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1166167, a bug was introduced in the logging logic due to the use of req.hdr() and res.hdr() instead of req.fhdr() and res.fhdr(). The hdr() fetches treat commas as delimiters and only return the last value, while fhdr() correctly returns the full header value as-is.
As a result, from the time the patch was merged (2025-07-07 07:24 UTC) until it was fixed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167203 (2025-07-08 12:03 UTC), requests were sent to the analytics pipeline with truncated header values in some fields:
This may have affected downstream analytics relying on those fields for that ~29-hour window.