[BUG] eventgate-logging-external drops previously collected http request headers
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Ottomata
	Mar 4 2025, 7:06 PM

Description

As reported in T387850: NEL logs are missing geoip information.

We added a enrich_fields_with_http_headers stream configuration feature to eventgate-wikimedia in T382173: Enable Event Platform streams to opt out of collecting User-Agent data.

This change was deployed to eventgate-logging-external on 2025-03-03. It seems likely that this change broke w3c network_error_logging collection of client set headers (T387850).

We should fix this!

Done

eventgate-logging-external streams have the same headers set as before T382173

Details

	Subject	Repo	Branch	Lines +/-
	eventgate-logging-external - upgrade to node20	operations/deployment-charts	master	+1 -16
	EventStreamConfig - keep geoip-* headers in eventgate-logging-external streams	operations/mediawiki-config	master	+54 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Ottomata	T382173 Enable Event Platform streams to opt out of collecting User-Agent data
		Resolved	BUG REPORT	Ottomata	T387908 [BUG] eventgate-logging-external drops previously collected http request headers

Event Timeline

Ottomata created this task.Mar 4 2025, 7:06 PM

Restricted Application changed the subtype of this task from "Task" to "Bug Report". · View Herald TranscriptMar 4 2025, 7:06 PM

Ottomata mentioned this in T387850: NEL logs are missing geoip information.Mar 4 2025, 7:08 PM

Note: this task is currently blocking T383814: Upgrade eventgate-wikimedia to node20 as the node 20 upgrade was done after the addition of enrich_fields_with_http_headers.

We could unblock by backporting the node20 upgrade and releasing without the enrich_fields_with_http_headers feature change, but it would probably be better just to fix the bug!

Ahoelzl assigned this task to Ottomata.Mar 17 2025, 3:19 PM

dr0ptp4kt subscribed.Mar 18 2025, 5:16 PM

Ah! There isn't a bug in eventgate-wikimedia. We just forgot that the headers lost in T263496: Augment NEL reports with GeoIP country code and network AS number were set by the global eventgate-logging-external http_request_headers_to_fields config we removed.

This was originally implemented in 2021 for T263496: Augment NEL reports with GeoIP country code and network AS number.

Because it was an eventgate instance wide setting, any eventgate-logging-external stream that has the http.request_headers field in its schema had the x-geoip-* headers set.

@CDanis do you know if we only need this data for w3c.reportingapi.network_error? If so, we can use the new per stream setting to keep the data.

Actually, the only other active eventgate-logging-external stream is mediawiki.client.error. I can see that these headers are being set in that stream now.

I suppose: we might as well keep the headers in mediawiki.client.error too.

Hm, the kaios_app.error stream also has this data. Let's keep it there too.

Ottomata renamed this task from [BUG] new eventgate-wikimedia header enrich config loses client set headers to [BUG] eventgate-logging-external drops previously collected http request headers.Wed, Mar 26, 7:02 PM

Ottomata updated the task description. (Show Details)

Change #1131413 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] EventStreamConfig - keep geoip-* headers in eventgate-logging-external streams

https://gerrit.wikimedia.org/r/1131413

gerritbot added a project: Patch-For-Review.Wed, Mar 26, 7:05 PM

Scheduled mediawiki-config change for backport window tomorrow.

Ottomata moved this task from Next Up to Ready to Deploy on the Data-Engineering (Q3 2025 January 1st - March 31th) board.Wed, Mar 26, 7:09 PM

Change #1131415 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-logging-external - upgrade to node20

https://gerrit.wikimedia.org/r/1131415

Ottomata changed the task status from Open to In Progress.Wed, Mar 26, 7:22 PM

Ottomata triaged this task as High priority.

Change #1131413 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig - keep geoip-* headers in eventgate-logging-external streams

https://gerrit.wikimedia.org/r/1131413

Change #1131415 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-logging-external - upgrade to node20

https://gerrit.wikimedia.org/r/1131415

Mentioned in SAL (#wikimedia-operations) [2025-03-27T15:28:10Z] <ottomata> upgrading eventgate-logging-external to node20 (using new per stream header enrich setting), first testing in staging. - T383814, T387908