This sounds like either it's a bug in the Metrics Platform code, or this is us mis-reading what this attribute is meant to convey?

We ran into this three years ago when two of the Growth team's schemas were migrated to Event Platform. Might be useful to check what was done there: T288853

CC @phuedx and @Ottomata any ideas here?

Hi! Would have to dive in to find out why/where it is not being set.

Here is the code in eventgate-wikimedia responsible for setting client_ip. And here is the function that determines client_ip from headers.

Just checking: is instrumentation explicitly setting http.client_ip in the event anywhere? If so that will take precedence.

From Slack:

• EventBus is configured to send events submitted to the mediawiki.product_metrics.wikilambda_api stream to eventgate-analytics-external
• EventBus is configured to forward the client IP to the event intake service (EventGate) for that service
• EventGate will set the http.client_ip property on the event if it has a slot for it

It's worth noting that EventBus will only forward the client IP to the event intake service if one is present. I wonder if it is 🤔

Thanks, @Ottomata ! Re: whether instrumentation is explicitly setting http.client_ip - in terms of what the extension (WikiLambda) does, definitely not.

Also from Slack:

I think there might be something going on with client IP detection or forwarding. I would greatly appreciate some help.

The task that @Nettrom linked was about the ServerSideAccountCreation instrument. At the time, @Nettrom verified that the instrument had been fixed.

Fortunately, the instrument is still active so I ran the following via Superset:

SELECT
  SUM(1) AS n,
  SUM(IF(http.client_ip IS NULL, 1, 0)) AS n_null,
  SUM(IF(http.client_ip = '127.0.0.1', 1, 0)) AS n_127_0_0_1,
  SUM(IF(http.client_ip = '127.0.0.1' AND event.isapi, 1, 0)) AS n_127_0_0_1_api
FROM
  event.serversideaccountcreation
WHERE
  year = 2024
  AND month = 6
  AND day < 27
;

and got the following results:

n       n_null  n_127_0_0_1  n_127_0_0_1_api
178919  0       166129       14192

So…

• A large number of events (92.85%) are being sent with X-Client-IP set to 127.0.0.1
• This isn't limited to interactions with the MediaWiki Action API

Unfortunately, event.serversideaccountcreation data is purged after 90 days so it's difficult to detect when this started occurring

Can confirm that this problem has re-emerged in HomepageVisit as well, running the same query that @phuedx
used, but modified to remove the API counts and changed to query event.homepagevisit gives:

n	n_null	n_127_0_0_1
561611	0	518059

That's 92.2% of events having the IP set to 127.0.0.1.

One thought I did have was that EventBus's way of determining the client IP is different from other codebases. EventBus fetches the value of the X-Client-IP header directly. This is the only instance of this in all codebases indexed by Codesearch.

@Ottomata: Would you have any objection to changing the line to:

if ( $this->forwardXClientIP ) {
    $req['headers']['x-client-ip'] = RequestContext::getMain()
        ->getRequest()
        ->getIP();
}

?

+1 looks good to me.

cc @gmodena

Change #1050588 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Increase the eventgate canary log_level to trace, temporarily

https://gerrit.wikimedia.org/r/1050588

Change #1050588 merged by jenkins-bot:

[operations/deployment-charts@master] Increase the eventgate canary log_level to trace, temporarily

https://gerrit.wikimedia.org/r/1050588

Change #1050608 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Revert "Increase the eventgate canary log_level to trace, temporarily"

https://gerrit.wikimedia.org/r/1050608

Change #1050608 merged by jenkins-bot:

[operations/deployment-charts@master] Revert "Increase the eventgate canary log_level to trace, temporarily"

https://gerrit.wikimedia.org/r/1050608

Change #1050632 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventBus@master] Forward info about original request

https://gerrit.wikimedia.org/r/1050632

Change #1050632 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] Forward info about original request

https://gerrit.wikimedia.org/r/1050632