Page MenuHomePhabricator

SDS 2.4.4 Edge Uniques Production Cookie Deployment
Closed, ResolvedPublic

Description

Background

The Wikimedia Foundation will implement a first-party cookie for all traffic to our sites named WMF-Uniq. This cookie's implementation is designed to support future A/B testing, DDoS attack mitigation, and visitor trend analysis goals while preserving our users' privacy.

The identifiers in these cookies will only be processed in realtime at our private CDN edge servers. They are designed to contain the minimal amount of metadata necessary to meet functional needs, while meeting or exceeding the privacy requirements outlined in our privacy policy and cookie use guidelines. They will never be stored in traffic logs or databases, and we will not create user profiles of individual readers' browsing histories or patterns that could be linked to a specific person over time. All implementation details will be in public, open-source code. Users can choose to block or clear these cookies without any functional effect on their use of the sites.

KR/Hypothesis from Annual Planning

  • SDS Objective 2: Product managers can quickly, easily, and confidently evaluate the impacts of product features.
  • Key Result 2.4: By end of Q4 FY24/25, successfully enable one product team to run an A/B test on anonymous users for a first paint feature while maintaining privacy compliance and data integrity.
    • Hypothesis 2.4.4: Traffic implementation of edge unique cookie (basically, this task)

Success Metrics

  • Meet or exceed our privacy policy and cookie use guidelines
  • Meet our timeline to use this cookie to complete one A/B test

Requirements

All items are implied to be done according to the design

  • Exchange WMF-Uniq cookies with our users’ agents.
  • Load A/B experiment configurations from a JSON file [which is distributed to CDN nodes initially by puppet]
  • Identify requests which are from agents whose cookies are enrolled in configured experiments. For such requests:
    • Send a Server-Timing header in the response to inform client-side Javascript of the experiment and group names.
    • If the test varies caching (flag in experiment definition):
      • Attach a header to the request before forwarding to the application layer (e.g. MediaWiki) which names the experiment and the group within the experiment, for application layer output-variance purposes.
      • Split CDN caching for experimental groups based on the same experiment/group names in the header above.
    • If the request is an A/B test result metric sent by client-side Javascript to our Eventgate analytics endpoint URI:
      • In addition to the experiment/group name information above, attach a test-group-specific, derived, temporary agent ID to the header as well, which is constructed from a one-way hash of the user's raw cookie identifier and the metadata about the experiment itself.

Actual Task Work for Implementers

  • Cut and package a first release version of the wmfuniq VMOD code. (T392059)
  • Deploy the wmfuniq VMOD code to our production varnish servers.
  • Deploy puppet patches to manage and distribute the server-side secret keys used to validate the cookies' contents.
  • Deploy puppet patches to distribute a static JSON configuration file containing active A/B test configurations (initially empty).
  • Deploy a VCL patch which makes all of our global Varnish servers minimally aware of the WMF-Uniq cookie and able to handle it by discarding and ignoring it.
  • Drop X-Enrollment-Experiments in HAProxy to prevent UAs from polluting the CDN cache
  • Avoid Vary: X-Enrollment-Experiments reaching the users
  • Start varying the cache on X-Enrollment-Experiments
  • Allow /evt-103e/v2/events to reach origin servers and send the traffic to intake-analytics
  • Deploy a VCL patch which uses the VMOD to set and process these cookies correctly, which can be rolled out progressively to the CDN fleet by-edge-host, replacing the above minimal VCL.
  • Progressively roll out to cache nodes and monitor for any functional, site performance, and/or server-load issues which may arise and address them, rolling back if/as necessary.

Timeline

The original timeline for rollout was to be complete by EOQ3 FY24-25 (end of March 2025). This schedule has already slipped, but we're aiming to begin the rollout of the full VCL patch for cookie processing during the week of April 21st. The other, enabling task items before that should be completed earlier. We're aiming to have the whole system functioning in time to conduct the first real A/B test by EOQ4 (end of June 2025).

Artifacts & Resources

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+2 -16
operations/puppetproduction+13 -0
operations/puppetproduction+6 -2
operations/puppetproduction+17 -7
operations/puppetproduction+2 -14
operations/puppetproduction+1 -1
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+0 -3
operations/puppetproduction+2 -1
operations/puppetproduction+4 -4
operations/puppetproduction+14 -0
operations/puppetproduction+4 -0
operations/puppetproduction+11 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+11 -4
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+264 -20
operations/puppetproduction+1 -1
operations/puppetproduction+4 -9
operations/puppetproduction+4 -0
operations/deployment-chartsmaster+45 -1
operations/puppetproduction+6 -3
operations/puppetproduction+11 -1
operations/puppetproduction+23 -34
operations/puppetproduction+34 -23
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+19 -6
operations/puppetproduction+127 -15
operations/puppetproduction+1 -1
operations/puppetproduction+39 -0
labs/privatemaster+2 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Resolvedmpopov
ResolvedMilimetric
Resolvedphuedx
ResolvedNone
Resolvedphuedx
Resolvedphuedx
Resolvedcjming
ResolvedSpikephuedx
Resolvedcjming
ResolvedSfaci
ResolvedSfaci
InvalidNone
InvalidNone
ResolvedMilimetric
ResolvedMilimetric
ResolvedKOfori
ResolvedBUG REPORTBBlack
ResolvedBCornwall
ResolvedBCornwall
ResolvedBTullis
ResolvedBBlack
ResolvedVgutierrez

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1143474 abandoned by Vgutierrez:

[operations/puppet@production] varnish: Allow /beacon/v2/event to hit origin servers

Reason:

this won't be needed, we are using a random prefix to avoid collisions

https://gerrit.wikimedia.org/r/1143474

Change #1143608 merged by Vgutierrez:

[operations/puppet@production] cache::haproxy: Drop incoming X-Experiment-Enrollments header

https://gerrit.wikimedia.org/r/1143608

Change #1143603 abandoned by Vgutierrez:

[operations/puppet@production] hiera: Split ATS cache on X-Experiment-Enrollments

Reason:

we won't be using cachekey for this

https://gerrit.wikimedia.org/r/1143603

Change #1147016 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] Revert "trafficserver: Allow splitting the cache by HTTP header content"

https://gerrit.wikimedia.org/r/1147016

Change #1147022 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Add X-Experiment-Enrollments to Vary header

https://gerrit.wikimedia.org/r/1147022

Change #1147016 merged by Vgutierrez:

[operations/puppet@production] Revert "trafficserver: Allow splitting the cache by HTTP header content"

https://gerrit.wikimedia.org/r/1147016

Change #1147737 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add all WMF domains to the eventgate-analytics-external certificate

https://gerrit.wikimedia.org/r/1147737

Change #1147780 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Prevent Vary: X-E-E from reaching users

https://gerrit.wikimedia.org/r/1147780

Change #1147780 merged by Vgutierrez:

[operations/puppet@production] varnish: Prevent Vary: X-E-E from reaching users

https://gerrit.wikimedia.org/r/1147780

Change #1147022 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Add X-Experiment-Enrollments to Vary header

https://gerrit.wikimedia.org/r/1147022

Change #1147737 merged by jenkins-bot:

[operations/deployment-charts@master] Add all WMF domains to the eventgate-analytics-external certificate

https://gerrit.wikimedia.org/r/1147737

Change #1143483 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Send /evt-103e/v2/events to intake-analytics

https://gerrit.wikimedia.org/r/1143483

Change #1148343 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques on cp4037

https://gerrit.wikimedia.org/r/1148343

Change #1148346 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Deploy wmfuniq-vmod and related files CDN wide

https://gerrit.wikimedia.org/r/1148346

Change #1148346 merged by Vgutierrez:

[operations/puppet@production] varnish: Deploy wmfuniq-vmod and related files CDN wide

https://gerrit.wikimedia.org/r/1148346

Change #1148355 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Fix edge uniques secret deployment

https://gerrit.wikimedia.org/r/1148355

Change #1148355 merged by Vgutierrez:

[operations/puppet@production] varnish: Fix edge uniques secret deployment

https://gerrit.wikimedia.org/r/1148355

Change #1142551 merged by Vgutierrez:

[operations/puppet@production] varnish: Issue and handle WMF-Uniq cookie

https://gerrit.wikimedia.org/r/1142551

Mentioned in SAL (#wikimedia-operations) [2025-05-20T15:57:50Z] <vgutierrez> depooling cp4037 before enabling edge uniques - T391411

Change #1148343 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques on cp4037

https://gerrit.wikimedia.org/r/1148343

Change #1148381 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Fix non wikimedia.org Domain value for WMF-Uniq cookie

https://gerrit.wikimedia.org/r/1148381

Change #1148381 merged by Vgutierrez:

[operations/puppet@production] varnish: Fix non wikimedia.org Domain value for WMF-Uniq cookie

https://gerrit.wikimedia.org/r/1148381

Change #1148399 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Skip PURGE requests for WMF-Uniq handling

https://gerrit.wikimedia.org/r/1148399

Change #1148399 merged by Vgutierrez:

[operations/puppet@production] varnish: Skip PURGE requests for WMF-Uniq handling

https://gerrit.wikimedia.org/r/1148399

Mentioned in SAL (#wikimedia-operations) [2025-05-20T17:31:22Z] <vgutierrez> repool cp4037 with edge uniques enabled, stats available on https://grafana.wikimedia.org/goto/fYSIMlaHR?orgId=1 - T391411

Change #1148814 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques on cp3066

https://gerrit.wikimedia.org/r/1148814

Change #1148814 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques on cp3066

https://gerrit.wikimedia.org/r/1148814

Mentioned in SAL (#wikimedia-operations) [2025-05-21T10:26:10Z] <vgutierrez> enabling edge uniques on cp3066 - T391411

Change #1148869 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques on cp4045

https://gerrit.wikimedia.org/r/1148869

Change #1148869 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques on cp4045

https://gerrit.wikimedia.org/r/1148869

Mentioned in SAL (#wikimedia-operations) [2025-05-21T13:36:26Z] <vgutierrez> enabling edge uniques on cp4045 - T391411

Change #1148877 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques in one server per DC and cluster

https://gerrit.wikimedia.org/r/1148877

Change #1148877 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques in one server per DC and cluster

https://gerrit.wikimedia.org/r/1148877

Mentioned in SAL (#wikimedia-operations) [2025-05-21T14:09:12Z] <vgutierrez> enabling edge uniques in one server per DC and cluster (cp[1100-1101],cp[2027-2028],cp3074,cp[5017,5025],cp[6001,6009],cp[7001,7009])- T391411

Change #1149405 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Allow setting WMF-Uniq cookie for WMCS domains

https://gerrit.wikimedia.org/r/1149405

Change #1149405 merged by Vgutierrez:

[operations/puppet@production] varnish: Allow setting WMF-Uniq cookie for WMCS domains

https://gerrit.wikimedia.org/r/1149405

Change #1150705 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques in another host per DC and cluster

https://gerrit.wikimedia.org/r/1150705

Change #1150705 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques in another host per DC and cluster

https://gerrit.wikimedia.org/r/1150705

Change #1151226 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques in ulsfo

https://gerrit.wikimedia.org/r/1151226

Change #1151226 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques in ulsfo

https://gerrit.wikimedia.org/r/1151226

Change #1151255 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: set SameSite=None for edge unique cookie in upload

https://gerrit.wikimedia.org/r/1151255

Change #1151255 merged by Vgutierrez:

[operations/puppet@production] varnish: set SameSite=None for edge unique cookie in upload

https://gerrit.wikimedia.org/r/1151255

Change #1151598 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques in codfw

https://gerrit.wikimedia.org/r/1151598

Change #1151598 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques in codfw

https://gerrit.wikimedia.org/r/1151598

Change #1151613 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge uniques in eqsin

https://gerrit.wikimedia.org/r/1151613

Change #1151613 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge uniques in eqsin

https://gerrit.wikimedia.org/r/1151613

Change #1151625 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge unique cookies in magru

https://gerrit.wikimedia.org/r/1151625

Change #1151625 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge unique cookies in magru

https://gerrit.wikimedia.org/r/1151625

Change #1151647 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge unique cookies in drmrs

https://gerrit.wikimedia.org/r/1151647

Change #1151647 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge unique cookies in drmrs

https://gerrit.wikimedia.org/r/1151647

Change #1151675 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge unique cookies in esams

https://gerrit.wikimedia.org/r/1151675

Change #1151675 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge unique cookies in esams

https://gerrit.wikimedia.org/r/1151675

Change #1151699 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable edge unique cookies in eqiad

https://gerrit.wikimedia.org/r/1151699

Change #1151699 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable edge unique cookies in eqiad

https://gerrit.wikimedia.org/r/1151699

Change #1151711 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Unify edge uniques settings

https://gerrit.wikimedia.org/r/1151711

Change #1152636 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Set wmfuniq experiment reload period to 30s

https://gerrit.wikimedia.org/r/1152636

Change #1152636 merged by Vgutierrez:

[operations/puppet@production] varnish: Set wmfuniq experiment reload period to 30s

https://gerrit.wikimedia.org/r/1152636

Change #1152648 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Start using edge uniques config fetched from xlabs endpoint

https://gerrit.wikimedia.org/r/1152648

Change #1152648 merged by Vgutierrez:

[operations/puppet@production] varnish: Start using edge uniques config fetched from xlabs endpoint

https://gerrit.wikimedia.org/r/1152648

Change #1152685 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Don't let experiment_fetcher crash if endpoint is unavailable

https://gerrit.wikimedia.org/r/1152685

Change #1152754 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Provide basic logging and metrics for experiment_fetcher

https://gerrit.wikimedia.org/r/1152754

Change #1152685 merged by Vgutierrez:

[operations/puppet@production] varnish: Don't let wmfuniq_experiment_fetcher crash if endpoint is unavailable

https://gerrit.wikimedia.org/r/1152685

Change #1153106 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Fix wmfuniq_experiment_fetcher

https://gerrit.wikimedia.org/r/1153106

Change #1153106 merged by Vgutierrez:

[operations/puppet@production] varnish: Fix wmfuniq_experiment_fetcher

https://gerrit.wikimedia.org/r/1153106

Change #1152754 merged by Vgutierrez:

[operations/puppet@production] varnish: Provide basic logging and metrics for wmfuniq_experiment_fetcher

https://gerrit.wikimedia.org/r/1152754

Change #1151711 merged by Vgutierrez:

[operations/puppet@production] hiera: Unify edge uniques settings

https://gerrit.wikimedia.org/r/1151711

Thanks, everyone!