Page MenuHomePhabricator

Roll out query parameter normalization
Closed, ResolvedPublic

Description

(Extracted from T138093)

Steps

  • Roll out libvmod-querysort package to varnishes.
  • Enable query-sorting in Beta Cluster varnish.
  • Enable query-sorting in production via X-Wikimedia-Debug.
  • Set $wgCdnMatchParameterOrder = false on Beta. Test.
  • Cherry-pick I3c52ca47e to deployment branches.
  • Set $wgCdnMatchParameterOrder = false on testwiki.
  • Enable querysort on testwiki.
  • Set $wgCdnMatchParameterOrder = false on mediawiki.org.
  • Enable querysort on mediawiki.org.
  • Set $wgCdnMatchParameterOrder = false everywhere.
  • Incrementally enable querysort for all domains (T138093#8117992)
  • Add documentation on Wikitech (done)

Monitoring

Testing

MediaWiki

$wgCdnMatchParameterOrder test script: P32345 (run on mwdebug)

Checking Cache-Control header from an app server:

$ curl -s -i --connect-to ::$HOSTNAME -H 'X-Forwarded-Proto: https' 'https://test.wikipedia.org/w/index.php?action=history&title=Main_Page' | grep -i cache-control
Cache-Control: private, must-revalidate, max-age=0

(should be Cache-Control: s-maxage=1209600, must-revalidate, max-age=0 once a87177564 is deployed)

Varnish
# Initial request is a cache miss (this may vary, obviously):
$ curl -si 'https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Hobby&action=history' | grep -i x-cache
x-cache: deployment-cache-text06 miss, deployment-cache-text06 miss
x-cache-status: miss

# Subsequent request is a cache hit:
$ curl -si 'https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Hobby&action=history' | grep -i x-cache
x-cache: deployment-cache-text06 miss, deployment-cache-text06 hit/1
x-cache-status: hit-remote

# Subsequent request, now with sorted query parameters, should also be a hit:
$ curl -si 'https://en.wikipedia.beta.wmflabs.org/w/index.php?action=history&title=Hobby' | grep -i x-cache
x-cache: deployment-cache-text06 miss, deployment-cache-text06 hit/2
x-cache-status: hit-remote

Details

Other Assignee
Vgutierrez
ProjectBranchLines +/-Subject
operations/puppetproduction+6 -6
operations/puppetproduction+2 -43
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+21 -20
operations/mediawiki-configmaster+1 -3
operations/puppetproduction+14 -2
operations/mediawiki-configmaster+1 -0
operations/puppetproduction+7 -2
mediawiki/corewmf/1.39.0-wmf.23+126 -7
operations/mediawiki-configmaster+5 -0
operations/mediawiki-configmaster+3 -0
Show related patches Customize query in gerrit

Event Timeline

ori renamed this task from Roll out query parameter normalization to Roll out query parameter normalization.Aug 9 2022, 2:59 PM
ori changed the task status from Open to In Progress.

Change 821757 had a related patch set uploaded (by Ori; author: Ori):

[operations/mediawiki-config@master] [beta] Set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/821757

Change 821757 merged by Ori:

[operations/mediawiki-config@master] [beta] Set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/821757

Change 821731 had a related patch set uploaded (by Ori; author: Ori):

[mediawiki/core@wmf/1.39.0-wmf.23] Support CDN query parameter re-ordering

https://gerrit.wikimedia.org/r/821731

Change 822093 had a related patch set uploaded (by Ori; author: Ori):

[operations/mediawiki-config@master] testwiki: set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/822093

Change 822093 merged by jenkins-bot:

[operations/mediawiki-config@master] testwiki: set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/822093

Change 821731 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.23] Support CDN query parameter re-ordering

https://gerrit.wikimedia.org/r/821731

Mentioned in SAL (#wikimedia-operations) [2022-08-10T20:34:42Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:822093|testwiki: set $wgCdnMatchParameterOrder to false (T314868)]] (duration: 03m 20s)

Change 819677 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Enable query sorting for all testwiki requests

https://gerrit.wikimedia.org/r/819677

Change 819677 merged by Vgutierrez:

[operations/puppet@production] Enable query sorting for all testwiki requests

https://gerrit.wikimedia.org/r/819677

Change 823656 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Enable query sorting for all mediawikiwiki requests

https://gerrit.wikimedia.org/r/823656

Change 823658 had a related patch set uploaded (by Ori; author: Ori):

[operations/mediawiki-config@master] mediawikiwiki: set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/823658

Change 823658 merged by jenkins-bot:

[operations/mediawiki-config@master] mediawikiwiki: set $wgCdnMatchParameterOrder to false

https://gerrit.wikimedia.org/r/823658

Change 823656 merged by Vgutierrez:

[operations/puppet@production] Enable query sorting for all mediawikiwiki requests

https://gerrit.wikimedia.org/r/823656

Change 824865 had a related patch set uploaded (by Ori; author: Ori):

[operations/mediawiki-config@master] Set $wgCdnMatchParameterOrder to false by default

https://gerrit.wikimedia.org/r/824865

Change 822434 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Incremental roll-out of query-sorting (0%)

https://gerrit.wikimedia.org/r/822434

Change 825232 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Incremental roll-out of query-sorting (1%)

https://gerrit.wikimedia.org/r/825232

Change 824865 merged by jenkins-bot:

[operations/mediawiki-config@master] Set $wgCdnMatchParameterOrder to false by default

https://gerrit.wikimedia.org/r/824865

Change 822434 merged by Vgutierrez:

[operations/puppet@production] Incremental roll-out of query-sorting (0%)

https://gerrit.wikimedia.org/r/822434

Change 825232 merged by Vgutierrez:

[operations/puppet@production] Incremental roll-out of query-sorting (1%)

https://gerrit.wikimedia.org/r/825232

Mentioned in SAL (#wikimedia-operations) [2022-08-22T10:00:53Z] <vgutierrez> Incremental roll-out of query-sorting (1%) - T314868

Change 825404 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Incremental roll-out of query-sorting (5%)

https://gerrit.wikimedia.org/r/825404

Change 825404 merged by Vgutierrez:

[operations/puppet@production] Incremental roll-out of query-sorting (5%)

https://gerrit.wikimedia.org/r/825404

Mentioned in SAL (#wikimedia-operations) [2022-08-23T09:40:42Z] <vgutierrez> Incremental roll-out of query-sorting (5%) - T314868

Change 825730 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Incremental roll-out of query-sorting (15%)

https://gerrit.wikimedia.org/r/825730

Change 825730 merged by Vgutierrez:

[operations/puppet@production] Incremental roll-out of query-sorting (15%)

https://gerrit.wikimedia.org/r/825730

Mentioned in SAL (#wikimedia-operations) [2022-08-23T12:33:36Z] <vgutierrez> Incremental roll-out of query-sorting (15%) - T314868

Change 825917 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Restart incremental roll-out of query-sorting at 1%

https://gerrit.wikimedia.org/r/825917

Change 825917 merged by Vgutierrez:

[operations/puppet@production] Restart incremental roll-out of query-sorting at 1%

https://gerrit.wikimedia.org/r/825917

Mentioned in SAL (#wikimedia-operations) [2022-08-24T09:46:27Z] <vgutierrez> Restart incremental roll-out of query-sorting at 1% - T314868

Change 826398 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase roll-out of query-sorting to 5%

https://gerrit.wikimedia.org/r/826398

Change 826398 merged by Vgutierrez:

[operations/puppet@production] Increase roll-out of query-sorting to 5%

https://gerrit.wikimedia.org/r/826398

Mentioned in SAL (#wikimedia-operations) [2022-08-25T08:23:10Z] <vgutierrez> Increase roll-out of query-sorting to 5% - T314868

Change 826601 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase roll-out of query-sorting to 15%

https://gerrit.wikimedia.org/r/826601

Change 826601 merged by Vgutierrez:

[operations/puppet@production] Increase roll-out of query-sorting to 15%

https://gerrit.wikimedia.org/r/826601

Mentioned in SAL (#wikimedia-operations) [2022-08-26T07:23:36Z] <vgutierrez> Increase roll-out of query-sorting to 15% - T314868

Change 826783 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase roll-out of query-sorting to 30%

https://gerrit.wikimedia.org/r/826783

Change 826783 merged by Vgutierrez:

[operations/puppet@production] Increase roll-out of query-sorting to 30%

https://gerrit.wikimedia.org/r/826783

Mentioned in SAL (#wikimedia-operations) [2022-08-26T08:47:08Z] <vgutierrez> Increase roll-out of query-sorting to 30% - T314868

Change 826994 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase roll-out of query-sorting to 50%

https://gerrit.wikimedia.org/r/826994

Change 826996 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase roll-out of query-sorting to 75%

https://gerrit.wikimedia.org/r/826996

Change 826997 had a related patch set uploaded (by Ori; author: Ori):

[operations/puppet@production] Increase query-sorting to 100%, remove sampling code

https://gerrit.wikimedia.org/r/826997

Change 826994 merged by Vgutierrez:

[operations/puppet@production] Increase roll-out of query-sorting to 50%

https://gerrit.wikimedia.org/r/826994

Mentioned in SAL (#wikimedia-operations) [2022-08-29T07:58:24Z] <vgutierrez> Increase roll-out of query-sorting to 50% - T314868

Change 826996 merged by Vgutierrez:

[operations/puppet@production] Increase roll-out of query-sorting to 75%

https://gerrit.wikimedia.org/r/826996

Mentioned in SAL (#wikimedia-operations) [2022-08-29T13:12:48Z] <vgutierrez> Increase roll-out of query-sorting to 75% - T314868

Change 826997 merged by Vgutierrez:

[operations/puppet@production] Increase query-sorting to 100%, remove sampling code

https://gerrit.wikimedia.org/r/826997

Mentioned in SAL (#wikimedia-operations) [2022-08-30T13:28:31Z] <vgutierrez> Increase roll-out of query-sorting to 100% - T314868

ori triaged this task as Medium priority.
ori updated Other Assignee, added: Vgutierrez.
ori updated the task description. (Show Details)

This is now complete. Many thanks to @Vgutierrez for partnering with me to get this rolled out.

Change 835572 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] varnish: Fix VCL tests broken by querysort

https://gerrit.wikimedia.org/r/835572

Change 835572 merged by Vgutierrez:

[operations/puppet@production] varnish: Fix VCL tests broken by querysort

https://gerrit.wikimedia.org/r/835572