Page MenuHomePhabricator
Create Task
Maniphest T383845

MediaWiki on PHP 8.1 production traffic ramp-up
Closed, ResolvedPublic
Actions

Description

This task will track execution of the production traffic ramp for migrating to PHP 8.1.

How?

We have 2 different mechanisms in place to assist us in directing traffic to PHP8.1 mediawiki pods in a controlled manner

Cookie-based enrollment

Applicable to cookie enabled incoming traffic towards following deployments:

  • mw-web → mw-web-next.discovery.wmnet
  • mw-api-ext → mw-api-ext-next.discovery.wmnet

As described in T377040, this traffic will be routed to pods with the release tag suffix -next. Users will randomly get the PHP_ENGINE=8.1 cookie, based on the amount of PHP8.1 traffic we are targeting for.

Capacity-based migration

Applicable to external cookie-less, and internal traffic, targeting trafficing towards the following deployments:

  • mw-api-int → mw-api-int-migration
  • mw-parsoid → mw-parsoid-migration
  • mw-jobrunner → mw-jobrunner-migration
  • mw-web → mw-web-migration
  • mw-api-ext → mw-api-ext-migration

We will be scaling the relevant -migration release of a given service, based on the portion of traffic we want to send to PHP8.1 pods.

Everything else

One-offs

  • mw-wikifunctions
  • mw-misc

Non-request-serving use cases (tracked in separate tasks)

  • mw-script
  • mw-cron
  • dumps

Support

Contact: @Scott_French, @jijiki, serviceops
Issues: Please report under T379874

Emergency Rollback

Below, you will find example patches should a rollback be necessary. Kindly proceed with caution and ensure you have SRE approval.
Deployment instructions can be found in T383845#10526890, as well as in each respective commit message:

Timeline and Milestones

We are aiming for a ~5 week long migration. The following schedule is subject to change depending on how smoothy everything goes. We shall pause for a week due to the SRE summit while we are hopeful to have completed the migration before the forthcoming March Datacenter Switchover

Any given moment, you may visit MediaWiki PHP8.1 Rollout Progress which provides a good overview of this work, though noting that any fully migrated deployments (e.g., mw-api-ext and mw-web) will appear as if reverted to 7.4.

Note: Rollouts will generally be taking place Mon-Tue-Wed each week

Note: As of 2025-02-26 we have resumed the traffic ramp after resolving what we believe to be the cause of T385395.

Week of 27th Jan (reverted)
Week of 3rd Feb (reverted)

2025-02-06: Reverted due to T385395.

Week of 10th Feb

No progress due to SRE Summit

Week of 17th Feb

No progress - working on T385395 resolution

Week of 24th Feb

Roll forward to state as of 6th February.

  • 50% cookie-enrolled web & ext-api traffic
  • 5% int-api traffic
  • 2% jobrunner
  • 2% parsoid
Week of 3rd March
  • 100% cookie-enrolled web & ext-api traffic
  • 5% residual web & ext-api traffic
  • 50% int-api traffic
  • 25% jobrunner
  • 25% parsoid
Week of 10th March
  • 100% residual web & ext-api traffic
  • 100% int-api traffic
  • 100% jobrunner
  • 100% parsoid
Week of 17th March (Point of No Return)

Note: This week contains the Datacenter Switchover, so we will likely opt to accelerate or defer some portions of this.

  • 100% mw-misc
  • 100% mw-wikifunctions
  • 100% mw-script - T387917
  • 100% mw-cron - T387916
  • 100% mw-videoscaler - mw-videoscaler is already using PHP 8.1

Details

Other Assignee
jijiki
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[DNM] hieradata: revert in-place from 8.1 to 7.4 (1 of 2)operations/puppetproduction+7 -7
[DNM] mw-*: revert in-place from 8.1 to 7.4 (2 of 2)operations/deployment-chartsmaster+10 -14
mw-(api-ext|web): additional post-migration cleanupoperations/deployment-chartsmaster+4 -6
mw-*: normalize the next and migration releasesoperations/deployment-chartsmaster+73 -62
mw-misc: migrate to PHP 8.1 (2 of 2)operations/deployment-chartsmaster+1 -0
hieradata: migrate mw-misc to PHP 8.1 (1 of 2)operations/puppetproduction+1 -1
mw-wikifunctions: migrate to PHP 8.1 (2 of 2)operations/deployment-chartsmaster+1 -0
hieradata: migrate mw-wikifunctions to PHP 8.1 (1 of 2)operations/puppetproduction+1 -1
Disable cookie-based enrollment in 8.1 (cleanup)operations/mediawiki-configmaster+1 -1
mw-(api-int|parsoid|jobrunner): switch all pods to -main (3/3)operations/deployment-chartsmaster+23 -29
mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)operations/deployment-chartsmaster+7 -0
hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)operations/puppetproduction+6 -9
hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)operations/puppetproduction+6 -9
mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)operations/deployment-chartsmaster+7 -0
mw-(api-ext|web): scale next down to 1 replicaoperations/deployment-chartsmaster+4 -22
trafficserver: revert cookie-enrolled traffic to mainoperations/puppetproduction+1 -1
mw-(api-ext|web): scale main up to normal multi-DC sizingoperations/deployment-chartsmaster+2 -16
mw-(api-ext|web): direct residual traffic back to mainoperations/deployment-chartsmaster+4 -16
mw-(api-ext|web): serve 100% of residual traffic on 8.1 (2 of 2)operations/deployment-chartsmaster+10 -0
hieradata: switch all releases of mw-(api-ext|web) to 8.1 (1 of 2)operations/puppetproduction+6 -10
hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/2)operations/puppetproduction+6 -9
mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/2)operations/deployment-chartsmaster+7 -0
mw-(api-ext|web): serve 50% of residual traffic on 8.1operations/deployment-chartsmaster+10 -10
mw-(api-ext|web): serve 25% of residual traffic on 8.1operations/deployment-chartsmaster+10 -10
mw-(api-int|parsoid|jobrunner): increase PHP8.1 trafficoperations/deployment-chartsmaster+12 -12
mw-*: Tune 8.1 releases to avoid deployment timeoutsoperations/deployment-chartsmaster+29 -8
mw-(api-ext|web): serve 5% of residual traffic on 8.1operations/deployment-chartsmaster+16 -4
trafficserver: respect PHP_ENGINE_STICKY cookie valueoperations/puppetproduction+1 -1
mw-web: additional right-sizing tweaks for nextoperations/deployment-chartsmaster+4 -4
mw-(api-ext|web): right-size given current trafficoperations/deployment-chartsmaster+47 -15
mw-api-int: serve 25% of traffic on PHP 8.1operations/deployment-chartsmaster+4 -4
mw-(api-ext|web): shrink next to 80% and increase maxUnavailableoperations/deployment-chartsmaster+6 -4
mw-api-int: serve 10% of traffic on PHP 8.1operations/deployment-chartsmaster+4 -4
Enroll 100% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
mw-(api-ext|web): scale next to 40% of mainoperations/deployment-chartsmaster+6 -6
Re-enable cookie-based enrollment in 8.1 at 50%operations/mediawiki-configmaster+1 -1
mw-(api-int|jobrunner|parsoid): resume php8.1 rolloutoperations/deployment-chartsmaster+15 -6
mw-(api-int|jobrunner|parsoid): resume php8.1 rolloutoperations/deployment-chartsmaster+15 -7
Re-enroll 5% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
trafficserver: re-enable cookie-enrolled traffic to 8.1operations/puppetproduction+1 -1
trafficserver: revert cookie-enrolled traffic to 7.4operations/puppetproduction+1 -1
Route PHP8 Excimer profiles to separate ArcLamp sinksoperations/mediawiki-configmaster+7 -0
mw-(api-int|jobrunner|parsoid): serve 100% of traffic on 7.4operations/deployment-chartsmaster+6 -9
Disable cookie-based enrollment in 8.1operations/mediawiki-configmaster+1 -1
jobrunner: remove PHP major version httpbb assertionoperations/puppetproduction+4 -3
Enroll 50% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
mw-api-int: serve 5% of traffic on PHP 8.1operations/deployment-chartsmaster+4 -4
mw-(api-ext|web): scale next to 25% of mainoperations/deployment-chartsmaster+4 -4
mw-parsoid & mw-jobrunner serve 2% of traffic on PHP 8.1operations/deployment-chartsmaster+4 -4
mw-api-int: serve 2% of traffic on PHP 8.1operations/deployment-chartsmaster+4 -4
Enroll 25% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
mw-(api-ext|web): scale next to 15% of mainoperations/deployment-chartsmaster+4 -4
mw-parsoid: serve 1% of traffic on PHP 8.1operations/deployment-chartsmaster+2 -2
mw-jobrunner: serve 1% of traffic on PHP 8.1operations/deployment-chartsmaster+2 -2
mw-api-int: serve 1% of traffic on PHP 8.1operations/deployment-chartsmaster+11 -2
Enroll 10% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
Enroll 5% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
Enroll 2% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
Enroll 1% of client sessions in PHP 8.1operations/mediawiki-configmaster+1 -1
Enroll 0.1% of client sessions in PHP 8.1operations/mediawiki-configmaster+2 -2
mw-(api-ext|web): temporarily increase maxUnavailableoperations/deployment-chartsmaster+12 -0
mw-(api-ext|web): scale next to 10% of mainoperations/deployment-chartsmaster+4 -2
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT396296 Upgrade symfony/* to PHP 8.1 versions
 ResolvedReedyT373753 Drop symfony/polyfill-php 73-81 from mediawiki-vendor
 StalledNoneT255792 Quibble runs core:unit tests twice!
 OpenNoneT376276 PHP 8.4: Implicitly nullable parameter declarations deprecated
 OpenNoneT379587 PHP 8.4: Implicitly nullable parameter declarations deprecated in 3rd party vendor libs
 ResolvedReedyT378424 PHP 8.4: Implicitly nullable parameter declarations deprecated in qrcode vendor libs
 OpenNoneT358666 Drop PHP 8.1 support from MediaWiki
 OpenKrinkleT360995 Migrate Wikimedia production from PHP 8.1 to PHP 8.3
 ResolvedJdforrester-WMFT353362 Make PHP 8.3 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries)
 ResolvedJdforrester-WMFT352085 Make PHP 8.2 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries)
 ResolvedJdforrester-WMFT360560 Make PHP 8.2 voting on development branch of Wikibase
 ResolvedLucas_Werkmeister_WMDET324202 Remove use of utf8_encode and utf8_decode in Wikibase
 ResolvedNoneT353161 [CLIENT] Creation of dynamic property MediaWiki\Title\Title::$wikibasePushedDeleteToRepo is deprecated
 ResolvedJdforrester-WMFT360709 Make PHP 8.2 voting on development (master) branch of Math extension
 ResolvedDaimonaT381728 Use PHP 8.3 in MediaWiki-Docker
 ResolvedJdforrester-WMFT398491 Move CI images to WMF PHP 8.3 packages
 OpenNoneT401255 Switch PHP 8.3 image of MediaWiki-Docker from Sury to WMF
 ResolvedScott_FrenchT401254 Upgrade mw-debug/next to PHP 8.3
 ResolvedScott_FrenchT401252 Configure title-case consistency mapping for PHP 8.1 -> 8.3 transition
 ResolvedScott_FrenchT399884 Configure production MediaWiki image builds for PHP 8.3
 OpencscottT401855 ☂ PHP 8.3 issues found during WMF rollout
 ResolvedScott_FrenchT402424 PHP Deprecated: Using ${var} in strings is deprecated, use {$var} instead in /srv/monitoring/lib.php on line 99
 DuplicatePRODUCTION ERRORNoneT403514 Expectation (readQueryRows <= 10000) by MediaWiki\Actions\ActionEntryPoint::execute not met (actual: 12860)
 ResolvedPRODUCTION ERRORMichaelT407403 Error: Invalid serialization data for DatePeriod object
 OpenPRODUCTION ERRORNoneT408852 PHP Deprecated: DateTime::__construct(): Passing null to parameter #1 ($datetime) of type string is deprecated
 OpenMichaelT401005 Investigate Production Errors from ComputedUserImpactLookup::getPageViewData
 DuplicatePRODUCTION ERRORNoneT409248 PHP Warning: Trying to access array offset on null
 ResolvedTgrT402597 Exploratory testing on PHP 8.3 for MediaWiki Platform Team components
 ResolvedNoneT402804 Exploratory testing on PHP 8.3 for MediaWiki authentication stack
 ResolvedDAlangi_WMFT403484 Exploratory testing on PHP 8.3 for MediaWiki authentication stack: Login
 ResolvedTgrT403485 Exploratory testing on PHP 8.3 for MediaWiki authentication stack: Central session
 ResolvedDAlangi_WMFT403486 Exploratory testing on PHP 8.3 for MediaWiki authentication stack: Signup
 ResolvedTgrT403487 Exploratory testing on PHP 8.3 for MediaWiki authentication stack: OAuth
 ResolvedTgrT403488 Exploratory testing on PHP 8.3 for MediaWiki authentication stack: 2FA
 ResolvedAtienoT402809 Exploratory testing on PHP 8.3 for MediaWiki Interfaces Team components
 OpenNoneT402810 Exploratory testing on PHP 8.3 for Content Transform Team components
 ResolvedbrouberolT403110 Prepare dumps-on-k8s (mediawiki-dumps-legacy) for migration to PHP 8.3
 ResolvedScott_FrenchT403655 Configure mw-next-routing for the PHP 8.3 migration
 ResolvedScott_FrenchT403657 Configure the WikimediaEvents extension for the PHP 8.3 migration
 ResolvedScott_FrenchT403284 Migrate production Shellbox services to PHP 8.3
 ResolvedScott_FrenchT403283 Prepare PHP 8.3 service images for Shellbox
 ResolvedScott_FrenchT398246 Prepare PHP 8.3 production images
 ResolvedScott_FrenchT398245 Prepare WMF PHP 8.3 packages for bullseye
 ResolvedKrinkleT400109 Ensure MediaWiki-Core-Profiler works with php-xhprof 2.x (PHP 8.3)
 ResolvedKrinkleT401152 Switch wmf-config/Profiler from Tideways to XHProf
 ResolvedScott_FrenchT403772 Migrate parsoidtest1001 to PHP 8.3
 ResolvedKrinkleT405031 lcobucci/clock needs upgrading for PHP 8.3
 ResolvedTgrT363639 web-auth/webauthn-lib must be upgraded to 4+ for PHP 8.2+ support
 ResolvedScott_FrenchT405955 MediaWiki on PHP 8.3 production workload migration
 ResolvedjnucheT411277 Patchdemo wiki creation fails due to PHP 8.1 not supported by MW
 ResolvedJdforrester-WMFT411372 wikilambda-catalyst-end-to-end broken due to new version of composer after PHP 8.1 -> 8.3 migration
 ResolvedBUG REPORTbd808T411235 Beta cluster scap using php8.1 container; php8.2 is now required
 ResolvedReedyT380185 Prepare for newer geoip2/geoip2 library
 ResolvedReedyT338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
 Resolvedseanleong-WMDET379930 Use WeakMap instead of an array in RestrictedEntityLookupFactory
 StalledNoneT334726 Use return type `never` in Wikibase
 OpenNoneT328919 Upgrade PHPUnit in MediaWiki code to PHPUnit 10
 ResolvedLucas_Werkmeister_WMDET328741 Test failures under PHPUnit 9.6: "Expecting E_DEPRECATED and E_USER_DEPRECATED is deprecated and will no longer be possible in PHPUnit 10"
 ResolvedDaimonaT342110 Upgrade to PHPUnit 9.6
 ResolvedDaimonaT342259 Move MediaWikiIntegrationTestCase setup and teardown away from run()
 ResolvedDaimonaT342301 Remove MediaWikiIntegrationTestCase::$tablesUsed in favour of automatic query tracking
 OpenNoneT332865 PHPUnit data providers should be simple static functions that return plain data
 ResolvedUmherirrenderT337130 Make PHPUnit dataProvider LexemeFieldTestBase::getTestData static (WikibaseLexemeCirrusSearch extension)
 OpenNoneT337135 Make PHPUnit dataProvider on UstringLibraryTest and LuaEngineUnitTestBase static
 ResolvedUmherirrenderT337144 Make PHPUnit dataProvider static in AbuseFilter tests
 ResolvedUmherirrenderT337155 Make PHPUnit dataProvider static in ReadingLists tests
 Resolved• mszaboT337156 Make PHPUnit dataProvider static in IPInfo tests
 ResolvedtstarlingT337157 Make PHPUnit dataProvider static in LoginNotify tests
 ResolvedUmherirrenderT337158 Make PHPUnit dataProvider static in Flow tests
 ResolvedDreamy_JazzT337159 Make PHPUnit dataProvider static in CheckUser tests
 ResolvedDreamy_JazzT346044 Remove CheckUserUnionQueryBuilder
 ResolvedPhysikerweltT337160 Make PHPUnit dataProvider static in Math tests
 ResolvedTgrT337161 Make PHPUnit dataProvider static in OAuth tests
 ResolvedUmherirrenderT337162 Make PHPUnit dataProvider static in ProofreadPage tests
 ResolvedUmherirrenderT337163 Make PHPUnit dataProvider static in CirrusSearch tests
 ResolvedNoneT337164 Make PHPUnit dataProvider static in FileImporter tests
 ResolvedUmherirrenderT337165 Make PHPUnit dataProvider static in GrowthExperiments tests
 ResolvedNoneT337166 Make PHPUnit dataProvider static in CampaignEvents tests
 ResolvedNoneT371467 [GENERAL] Make PHPUnit dataProvider static in Wikibase-related extensions
 ResolvedkarapayneWMDET337154 Make PHPUnit dataProvider static in Wikibase tests
 ResolvedAudreyPenven_WMDET380604 Make provideLuaData static
 ResolvedAudreyPenven_WMDET380605 Make EntityMetaTagsCreatorTestCase's provideTestGetMetaTags dataProvider static
 ResolvedLucas_Werkmeister_WMDET337153 Make PHPUnit dataProvider static in WikibaseQualityConstraints tests
 ResolvedLucas_Werkmeister_WMDET337151 Make PHPUnit dataProvider static in WikibaseLexeme tests
 ResolvedUmherirrenderT337152 Make PHPUnit dataProvider static in WikibaseMediaInfo tests
 ResolvedthiemowmdeT337150 Make PHPUnit dataProvider static in Wikidata.org tests
 ResolvedLucas_Werkmeister_WMDET337148 Make PHPUnit dataProvider static in WikibaseCirrusSearch tests
 ResolvedUmherirrenderT393065 Make PHPUnit dataProvider on ExtensionJsonTestBase static
 ResolvedUmherirrenderT393079 Make PHPUnit dataProvider static in CommunityConfiguration tests
 ResolvedUmherirrenderT393080 Make PHPUnit dataProvider static in AutoModerator tests
 ResolvedUmherirrenderT393081 Make PHPUnit dataProvider on AbstractChangesListSpecialPageTestCase static
 ResolvedUmherirrenderT393082 Make PHPUnit dataProvider on OutputTransformStageTestBase static
 ResolvedUmherirrenderT393084 Make PHPUnit dataProvider on TypeDefTestCaseTrait static
 Resolved• mszaboT393115 Make PHPUnit dataProvider on PasswordTestCase static
 ResolvedxSavitarT393122 Make PHPUnit dataProvider on BundleSizeTestBase static
 ResolvedUmherirrenderT393132 Make PHPUnit dataProvider on RevisionRecordTests static
 ResolvedUmherirrenderT393207 Make PHPUnit dataProvider on ExtensionServicesTestBase static
 ResolvedUmherirrenderT393219 Make PHPUnit dataProvider on RestStructureTest static
 OpenNoneT410731 Make production extensions PHPUnit tests data providers real providers (and use static methods)
 OpenNoneT345481 Migrate Parser and extension tests away from deprecated PHPUnit TestSuite subclassing
 OpenNoneT358394 Stop using PHPUnit's TestSuite in Scribunto
 OpenDannyS712T363064 MediaWiki.Commenting.MissingCovers.MissingCovers does not support PHPUnit 10.x attributes
 ResolvedUmherirrenderT363641 IPUtils: Expecting E_WARNING and E_USER_WARNING is deprecated and will no longer be possible in PHPUnit 10
 Resolved• larissagauliaT363650 Minify: Expecting E_WARNING and E_USER_WARNING is deprecated and will no longer be possible in PHPUnit 10
 ResolvedUmherirrenderT362858 equivset: Expecting E_ERROR and E_USER_ERROR is deprecated
 ResolvedJdforrester-WMFT395743 Use of package ockcyp/covers-validator in libaries is not compatible with PHPUnit 10
 ResolvedJdforrester-WMFT319055 Upgrade to psr/container 2.x
 ResolvedJdforrester-WMFT328921 Drop PHP 7.4 and PHP 8.0 support from MediaWiki
 ResolvedKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
 ResolvedScott_FrenchT383845 MediaWiki on PHP 8.1 production traffic ramp-up
 ResolvedScott_FrenchT377042 Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext
 ResolvedScott_FrenchT377040 Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services
 ResolvedKrinkleT377987 Prepare sticky cookie for gradual PHP 8.1 rollout
 ResolvedMSantosT379986 Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next
 ResolvedTgrT383633 Test MediaWiki authentication on PHP 8.1 in mwdebug-next
 ResolvedScott_FrenchT372605 Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment
 ResolvedScott_FrenchT372604 Turn up PHP 8.1-flavored mw-debug k8s deployment
 ResolveddduvallT370934 Build and publish multiple MediaWiki production images for a given set of PHP versions
 ResolvedClement_GoubertT376895 mw-debug-repl fails with `container mediawiki-pinkunicorn-app is not valid for pod mw-debug.codfw.next-5d785576b4-sq6dv`
 ResolvedScott_FrenchT372602 Prepare PHP 8.1 production images
 ResolvedScott_FrenchT372507 Prepare WMF PHP 8.1 packages for Bullseye
 ResolvedScott_FrenchT372603 Regenerate UcfirstOverrides.php for PHP 7.4 -> 8.1 transition
 ResolvedScott_FrenchT384532 Align mw-on-k8s alerts with PHP 8.1 migration
 ResolvedoriT385199 Gather PHP 8.1 profiling data
ResolvedClement_GoubertT387916 Migrate mw-cron to PHP 8.1
 ResolvedScott_FrenchT387917 Migrate mw-script to PHP 8.1
 In ProgressScott_FrenchT388799 php-wmerrors rsyslog rule selects on php7 only

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Clement_Goubert closed subtask T387916: Migrate mw-cron to PHP 8.1 as Resolved.Mar 10 2025, 6:02 PM
Scott_French updated the task description. (Show Details)Mar 10 2025, 6:29 PM
Scott_French changed the status of subtask T387917: Migrate mw-script to PHP 8.1 from Open to In Progress.Mar 10 2025, 8:50 PM
jijiki updated the task description. (Show Details)Mar 10 2025, 10:00 PM
gerritbot added a comment.Mar 11 2025, 5:21 PM

Change #1126607 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] hieradata: switch mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/2)

https://gerrit.wikimedia.org/r/1126607

gerritbot added a comment.Mar 11 2025, 5:32 PM

Change #1125504 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): serve 50% of residual traffic on 8.1

https://gerrit.wikimedia.org/r/1125504

Stashbot added a comment.Mar 11 2025, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-11T17:48:12Z] <swfrench-wmf> mw-(api-ext|web): migrated 50% of residual PHP 7.4 traffic to 8.1 - T383845

gerritbot added a comment.Mar 11 2025, 8:16 PM

Change #1126650 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/2)

https://gerrit.wikimedia.org/r/1126650

gerritbot added a comment.Mar 12 2025, 10:17 AM

Change #1126607 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/2)

https://gerrit.wikimedia.org/r/1126607

gerritbot added a comment.Mar 12 2025, 10:33 AM

Change #1126650 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/2)

https://gerrit.wikimedia.org/r/1126650

Stashbot added a comment.Mar 12 2025, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T10:44:24Z] <jiji@deploy2002> Started scap sync-world: (T383845) mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1

TheDJ mentioned this in T388659: MWExceptionRenderer not adding styles for message boxes (will appear unstyled in future MediaWiki version).EditedMar 12 2025, 12:34 PM
TheDJ subscribed.

ping @jijiki as scap deployer for the possible change that kicked this error rate of T388659 up

Clement_Goubert subscribed.Mar 12 2025, 12:38 PM
In T383845#10628231, @TheDJ wrote:

ping @jijiki as scap deployer for the possible change that kicked this error rate of T388659 up

This is an unrelated incident (as far as we can tell) we are currently working on https://wikimedia.statuspage.io/incidents/t13w599t74sq

gerritbot added a comment.Mar 12 2025, 4:33 PM

Change #1125501 merged by Scott French:

[operations/puppet@production] hieradata: switch all releases of mw-(api-ext|web) to 8.1 (1 of 2)

https://gerrit.wikimedia.org/r/1125501

gerritbot added a comment.Mar 12 2025, 4:37 PM

Change #1125505 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): serve 100% of residual traffic on 8.1 (2 of 2)

https://gerrit.wikimedia.org/r/1125505

Stashbot added a comment.Mar 12 2025, 4:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T16:43:45Z] <swfrench@deploy2002> Started scap sync-world: No-sync scap run to update helmfile release values for mw-(api-ext|web) - T383845

Stashbot added a comment.Mar 12 2025, 5:02 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:02:28Z] <swfrench@deploy2002> Started scap sync-world: helmfile-only deployment to apply remaining 8.1 diffs on mw-(api-ext|web) - T383845

Stashbot added a comment.Mar 12 2025, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:06:06Z] <swfrench@deploy2002> Finished scap sync-world: helmfile-only deployment to apply remaining 8.1 diffs on mw-(api-ext|web) - T383845 (duration: 05m 03s)

Stashbot added a comment.Mar 12 2025, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:06:37Z] <swfrench-wmf> mw-(api-ext|web): migrated 100% of residual PHP 7.4 traffic to 8.1 - T383845

gerritbot added a comment.Mar 12 2025, 5:16 PM

Change #1125506 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): direct residual traffic back to main

https://gerrit.wikimedia.org/r/1125506

Stashbot added a comment.Mar 12 2025, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:28:20Z] <swfrench-wmf> mw-(api-ext|web): reverted all non-cookie-migrated traffic back to 'main' release - T383845

gerritbot added a comment.Mar 12 2025, 5:30 PM

Change #1125507 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): scale main up to normal multi-DC sizing

https://gerrit.wikimedia.org/r/1125507

Stashbot added a comment.Mar 12 2025, 5:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:35:05Z] <swfrench-wmf> mw-(api-ext|web): scaled 'main' releases back to normal size - T383845

Stashbot added a comment.Mar 12 2025, 5:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T17:37:36Z] <swfrench-wmf> ran cumin 'A:cp-text' 'disable-puppet "merging ATS Lua config change - T383845"'

gerritbot added a comment.Mar 12 2025, 5:38 PM

Change #1125502 merged by Scott French:

[operations/puppet@production] trafficserver: revert cookie-enrolled traffic to main

https://gerrit.wikimedia.org/r/1125502

Stashbot added a comment.Mar 12 2025, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T18:07:45Z] <swfrench-wmf> ran cumin -b8 -s90 'A:cp-text' 'run-puppet-agent -e "merging ATS Lua config change - T383845"'

Scott_French added a comment.Mar 12 2025, 6:31 PM

As of ~ 17:06 UTC, all external web and API traffic - i.e., traffic served by mw-api-ext and mw-web - has been migrated to PHP 8.1.

There are some lingering cleanups to take care of related to capacity management, but I hope to take care of those later today.

I've refreshed the "in-place revert" patches listed in the Emergency Rollback section of the task description to reflect the latest state. As before, if something comes up and we need to revert, that is the preferred strategy.

gerritbot added a comment.Mar 12 2025, 6:40 PM

Change #1125508 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): scale next down to 1 replica

https://gerrit.wikimedia.org/r/1125508

Stashbot added a comment.Mar 12 2025, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-12T18:48:30Z] <swfrench-wmf> mw-(api-ext|web): scaled latent 'next' deployments down to 1 pod - T383845

Scott_French updated the task description. (Show Details)Mar 12 2025, 7:35 PM
gerritbot added a comment.Mar 13 2025, 10:15 AM

Change #1127476 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)

https://gerrit.wikimedia.org/r/1127476

gerritbot added a comment.Mar 13 2025, 10:22 AM

Change #1127478 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)

https://gerrit.wikimedia.org/r/1127478

gerritbot added a comment.Mar 13 2025, 10:40 AM

Change #1127476 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)

https://gerrit.wikimedia.org/r/1127476

gerritbot added a comment.Mar 13 2025, 11:01 AM

Change #1127478 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)

https://gerrit.wikimedia.org/r/1127478

Stashbot added a comment.Mar 13 2025, 11:15 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-13T11:15:44Z] <jiji@deploy2002> Started scap sync-world: (T383845) mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1

matmarex updated the task description. (Show Details)Mar 13 2025, 12:57 PM
gerritbot added a comment.Mar 13 2025, 4:46 PM

Change #1127557 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)

https://gerrit.wikimedia.org/r/1127557

gerritbot added a comment.Mar 13 2025, 4:46 PM

Change #1127558 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)

https://gerrit.wikimedia.org/r/1127558

gerritbot added a comment.Mar 13 2025, 5:01 PM

Change #1127558 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: switch 100% mw-(api-int|parsoid|jobrunner) to PHP 8.1 (1/3)

https://gerrit.wikimedia.org/r/1127558

gerritbot added a comment.Mar 13 2025, 5:07 PM

Change #1127557 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all releases to PHP 8.1 (2/3)

https://gerrit.wikimedia.org/r/1127557

Stashbot added a comment.Mar 13 2025, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-13T17:47:08Z] <jiji@deploy2002> Started scap sync-world: No-sync scap run to switch image flavours to PHP 8.1 - T383845

Stashbot added a comment.Mar 13 2025, 6:09 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-13T18:09:15Z] <jiji@deploy2002> Started scap sync-world: scap run to deploy switch to PHP 8.1 images - T383845

Stashbot added a comment.Mar 13 2025, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-13T18:17:38Z] <jiji@deploy2002> Finished scap sync-world: scap run to deploy switch to PHP 8.1 images - T383845 (duration: 10m 28s)

gerritbot added a comment.Mar 13 2025, 7:07 PM

Change #1125423 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-int|parsoid|jobrunner): switch all pods to -main (3/3)

https://gerrit.wikimedia.org/r/1125423

Stashbot added a comment.Mar 13 2025, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-13T19:24:04Z] <swfrench-wmf> mw-(api-int|jobrunner|parsoid): reverted all traffic back to 'main' release - T383845

Scott_French changed the status of subtask T388799: php-wmerrors rsyslog rule selects on php7 only from Open to In Progress.Mar 13 2025, 7:57 PM
Scott_French updated the task description. (Show Details)Mar 13 2025, 11:14 PM
Scott_French added a comment.Mar 14 2025, 2:12 AM

As of ~ 18:20 UTC today, mw-api-int, mw-parsoid, and mw-jobrunner have been fully migrated to PHP 8.1. This concludes the large / high-traffic services, leaving lower-traffic services (or non-traffic-serving use cases) that we'll coordinate plans for over the coming days (see task description).

I've once again refreshed the "in-place revert" patches listed in the Emergency Rollback section of the task description to reflect the latest state. I've also added warning note around the specific scap command used.

gerritbot added a comment.Mar 17 2025, 2:57 PM

Change #1128439 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-wikifunctions: migrate to PHP 8.1 (2 of 2)

https://gerrit.wikimedia.org/r/1128439

gerritbot added a comment.Mar 17 2025, 2:57 PM

Change #1128440 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: migrate mw-wikifunctions to PHP 8.1 (1 of 2)

https://gerrit.wikimedia.org/r/1128440

gerritbot added a comment.Mar 17 2025, 3:29 PM

Change #1128451 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/mediawiki-config@master] Disable cookie-based enrollment in 8.1

https://gerrit.wikimedia.org/r/1128451

gerritbot added a comment.Mar 17 2025, 5:47 PM

Change #1128451 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable cookie-based enrollment in 8.1 (cleanup)

https://gerrit.wikimedia.org/r/1128451

Stashbot added a comment.Mar 17 2025, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T17:48:02Z] <swfrench@deploy2002> Started scap sync-world: Backport for [[gerrit:1128451|Disable cookie-based enrollment in 8.1 (cleanup) (T383845)]]

Stashbot added a comment.Mar 17 2025, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T17:52:00Z] <swfrench@deploy2002> swfrench: Backport for [[gerrit:1128451|Disable cookie-based enrollment in 8.1 (cleanup) (T383845)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mar 17 2025, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T18:00:38Z] <swfrench@deploy2002> Finished scap sync-world: Backport for [[gerrit:1128451|Disable cookie-based enrollment in 8.1 (cleanup) (T383845)]] (duration: 12m 35s)

Stdolan subscribed.Mar 18 2025, 1:35 AM
gerritbot added a comment.Mar 18 2025, 5:01 PM

Change #1128440 merged by Scott French:

[operations/puppet@production] hieradata: migrate mw-wikifunctions to PHP 8.1 (1 of 2)

https://gerrit.wikimedia.org/r/1128440

gerritbot added a comment.Mar 18 2025, 5:06 PM

Change #1128439 merged by jenkins-bot:

[operations/deployment-charts@master] mw-wikifunctions: migrate to PHP 8.1 (2 of 2)

https://gerrit.wikimedia.org/r/1128439

Stashbot added a comment.Mar 18 2025, 5:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-18T17:12:21Z] <swfrench@deploy2002> Started scap sync-world: Switch mw-wikifuncions to PHP 8.1 - T383845

Stashbot added a comment.Mar 18 2025, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-18T17:20:37Z] <swfrench@deploy2002> Finished scap sync-world: Switch mw-wikifuncions to PHP 8.1 - T383845 (duration: 11m 46s)

gerritbot added a comment.Mar 18 2025, 5:40 PM

Change #1128923 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: migrate mw-misc to PHP 8.1 (1 of 2)

https://gerrit.wikimedia.org/r/1128923

gerritbot added a comment.Mar 18 2025, 5:40 PM

Change #1128924 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-misc: migrate to PHP 8.1 (2 of 2)

https://gerrit.wikimedia.org/r/1128924

Scott_French updated the task description. (Show Details)Mar 18 2025, 5:40 PM
gerritbot added a comment.Mar 20 2025, 6:40 PM

Change #1128923 merged by Scott French:

[operations/puppet@production] hieradata: migrate mw-misc to PHP 8.1 (1 of 2)

https://gerrit.wikimedia.org/r/1128923

gerritbot added a comment.Mar 20 2025, 6:43 PM

Change #1128924 merged by jenkins-bot:

[operations/deployment-charts@master] mw-misc: migrate to PHP 8.1 (2 of 2)

https://gerrit.wikimedia.org/r/1128924

Stashbot added a comment.Mar 20 2025, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-20T18:48:36Z] <swfrench@deploy2002> Started scap sync-world: Switch mw-misc to PHP 8.1 - T383845

Stashbot added a comment.Mar 20 2025, 6:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-20T18:51:05Z] <swfrench@deploy2002> Finished scap sync-world: Switch mw-misc to PHP 8.1 - T383845 (duration: 03m 22s)

Scott_French added a comment.Mar 20 2025, 7:09 PM

Alright, after a bit or manual testing (h/t to Joe for doing so as well), mw-misc - i.e., noc.wikimedia.org - appears to be working as expected.

I'll keep an eye out for issues that might reflect subtly broken clients (since we have various automated use cases that fetch configs from noc), but given what I've seen so far in manual testing, I believe that would be unlikely.

Scott_French updated the task description. (Show Details)Mar 20 2025, 7:09 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 20 2025, 7:31 PM
gerritbot added a comment.Mar 20 2025, 11:13 PM

Change #1129959 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-*: normalize the next and migration releases

https://gerrit.wikimedia.org/r/1129959

gerritbot added a project: Patch-For-Review.Mar 20 2025, 11:13 PM
abi_ mentioned this in T389944: Update Translate to use PHP 8.1 features.Mar 25 2025, 12:58 PM
gerritbot added a comment.Mar 25 2025, 5:03 PM

Change #1129959 merged by jenkins-bot:

[operations/deployment-charts@master] mw-*: normalize the next and migration releases

https://gerrit.wikimedia.org/r/1129959

Stashbot added a comment.Mar 25 2025, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T17:06:15Z] <swfrench@deploy1003> Started scap sync-world: Helmfile-only deployment for next and migration release cleanups - T383845

Stashbot added a comment.Mar 25 2025, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T17:08:07Z] <swfrench@deploy1003> Finished scap sync-world: Helmfile-only deployment for next and migration release cleanups - T383845 (duration: 02m 45s)

Scott_French mentioned this in T390225: Migrate scap's maintenance script invocations to PHP 8.1.Mar 27 2025, 6:11 PM
jijiki changed the status of subtask T383875: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades from Open to Stalled.Apr 2 2025, 12:33 PM
Scott_French updated the task description. (Show Details)Apr 3 2025, 11:23 PM
Scott_French closed subtask T387917: Migrate mw-script to PHP 8.1 as Resolved.Apr 3 2025, 11:30 PM
gerritbot added a comment.Apr 4 2025, 1:07 AM

Change #1134100 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] mw-(api-ext|web): additional post-migration cleanup

https://gerrit.wikimedia.org/r/1134100

gerritbot added a comment.Apr 4 2025, 2:56 PM

Change #1134100 merged by jenkins-bot:

[operations/deployment-charts@master] mw-(api-ext|web): additional post-migration cleanup

https://gerrit.wikimedia.org/r/1134100

jasmine_ subscribed.Apr 7 2025, 3:30 PM
Jdforrester-WMF added a comment.Apr 8 2025, 9:07 PM

Is this as-written Resolved? The remaining work AIUI is on migrating the last remaining cron jobs over to the new infrastructure (T341555).

Scott_French closed this task as Resolved.Apr 8 2025, 11:35 PM

Thanks, @Jdforrester-WMF - Indeed it is, though I was holding onto it for a bit to track immediate follow-on cleanups to get our configuration in a more understandable state. In any case, I think those should all be done at this point, and the bulk of the remaining cleanup is captured in T391057: Turn down MediaWiki image builds for PHP 7.4. Separately, there will be work to make migration-related infrastructure more generic in advance of the next one.

Also yes, the two remaining critical items are:

In any case, for clarity, let's close this out.

ori closed subtask T385199: Gather PHP 8.1 profiling data as Resolved.Apr 23 2025, 4:39 PM
gerritbot added a comment.Jun 23 2025, 8:17 PM

Change #1114029 abandoned by Scott French:

[operations/deployment-charts@master] [DNM] mw-*: revert in-place from 8.1 to 7.4 (2 of 2)

Reason:

This was not needed

https://gerrit.wikimedia.org/r/1114029

gerritbot added a comment.Jun 23 2025, 8:17 PM

Change #1114028 abandoned by Scott French:

[operations/puppet@production] [DNM] hieradata: revert in-place from 8.1 to 7.4 (1 of 2)

Reason:

This was not needed

https://gerrit.wikimedia.org/r/1114028

jijiki removed a subtask: T383875: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades.Jul 17 2025, 7:55 AM
jijiki removed a subtask: T383805: Performance assessment of PHP 8.Jul 17 2025, 8:01 AM
Scott_French mentioned this in T405955: MediaWiki on PHP 8.3 production workload migration.Sep 29 2025, 6:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits