The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

I'm being bold and removing this from the Icebox so we can talk about it.

I've heard a few different folks talk about related needs, but I'll just detail my need here:

We are writing a client-side pageview JS instrument that would send us hopefully enough signals to classify the view as "by a human" or not. We would like to know how this data lines up with the heuristic bot classification we do on top of the webrequest data. It's possible that we'll be able to tell from the overall aggregates or some rough buckets like "Firefox" / "Chrome" or geographic buckets or some combination. But if that's not precise enough, we would ideally want to correlate these two things:

A. pageview event sent from our client-side JS instrument
B. row in webrequest table identified as a pageview

So we tried to think of privacy-preserving ways to do that. X-Request-Id maybe would work if we:

• generated it at the edge
• propagated it to webrequest
• put it in a header that JS can read client-side
• send it from xLab's JS SDK with our instrument's events

What do you think? Does this violate the semantics of X-Request-Id? Would the idea work with a different Id that was better named for this? Or are there caching or other considerations I'm totally missing... (I read up a bit on T113817: Add request_id to webrequest logs as well as other event records ingested into Hadoop and those concerns raised there seem worth re-visiting, the ask now is more focused.)

FWIW HAProxy provides UUIDv4 out of the box so it should be as easy as http-request set-header X-Request-Id %[uuid()].

But for your use case @Milimetric I'm wondering if the networkprobe extension (probenet) is enough: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/refs/heads/master/modules/ext.wikimediaEvents.networkprobe/

It's javascript code that reports the latency from users to our PoPs. That's already a good indicator of clients that are running JS code, but take into account that we have data of some bots that successfully run JS code as well :)

Thanks Valentín, I'm thinking probenet can be a useful signal, but our current focus is to experiment with specific known signals (e.g. presence of DOM properties). We'd like a bit more flexibility to deploy custom logic, and also control over what % of traffic we collect those signals from. More fundamentally, we believe we'll need the X-Request-Id correlation to even be able to evaluate with confidence whether client-side signals, include probenet, are helpful or not.

Dan and I are working on a decision brief to help us navigate the privacy tradeoffs of this work. I'm hoping to have something for review next week.

put it in a header that JS can read client-side
send it from xLab's JS SDK with our instrument's events

(I have some memory of being told I am wrong about this, but...)
If x-request-id is set, eventgate will hoist it into events in the meta.request_id field by default:

So I don't think it strictly needs to be visible to JS and set by JS.

curl 'https://meta.wikimedia.org/w/api.php?action=streamconfigs&format=json&streams=product_metrics.app_base' | jq .streams

  "product_metrics.app_base": {
...
    "producers": {
      "eventgate": {
        "enrich_fields_from_http_headers": {
          "meta.request_id": "x-request-id",
          "http.request_headers.user-agent": "user-agent"
        }
      },
...
}

Do we know what the current behavior is for layers that set X-Request-ID?

The usual approach for HAProxy is to generate a UUID, append it to the request, and echo it back to the user, for example:

http-request set-var(txn.req_id) uuid()
http-request set-header X-Request-ID %[var(txn.req_id)]
http-response set-header X-Request-ID %[var(txn.req_id)]

Currently, we strip any X-Request-ID header from the user response unless X-Wikimedia-Debug is present.

Do layers that traditionally generate X-Request-IDs (like Envoy) echo an incoming X-Request-ID from the CDN?
And should the CDN respect X-Request-ID headers included in requests originating from wikimedia_trust ranges?

What are your thoughts here @CDanis?

In T221976#11248799, @Vgutierrez wrote:

The usual approach for HAProxy is to generate a UUID, append it to the request, and echo it back to the user, for example:

http-request set-var(txn.req_id) uuid()
http-request set-header X-Request-ID %[var(txn.req_id)]
http-response set-header X-Request-ID %[var(txn.req_id)]

LGTM except maybe http-after-response ?

Currently, we strip any X-Request-ID header from the user response unless X-Wikimedia-Debug is present.

+1

Do layers that traditionally generate X-Request-IDs (like Envoy) echo an incoming X-Request-ID from the CDN?

Yes. All of MediaWiki, mw apache, and Envoy tls-listener will honor an existing x-r-id if present, and generate one if not. And return it upstream unconditionally. (In production this results in Envoy generating x-req-id.)

And should the CDN respect X-Request-ID headers included in requests originating from wikimedia_trust ranges?

I think so, yes. This will allow trace context propagation for internal services that call other internal services via the CDN.

Change #1194989 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy: Set and propagate X-Request-ID

https://gerrit.wikimedia.org/r/1194989

Change #1194989 merged by Vgutierrez:

[operations/puppet@production] haproxy: Set and propagate X-Request-ID

https://gerrit.wikimedia.org/r/1194989

T407826: X-Request-Id response header off by 5000 may be related.