Page MenuHomePhabricator
Create Task
Maniphest T319432

Migrate WMF production from PHP 7.4 to PHP 8.1
Closed, ResolvedPublic
Actions

Assigned To
Krinkle
Authored By
Krinkle
Oct 5 2022, 3:28 PM
Tags
Referenced Files
F56618453: Screenshot 2024-07-23 at 14.47.10.png
Jul 23 2024, 1:47 PM
F56618452: Screenshot 2024-07-23 at 14.46.28.png
Jul 23 2024, 1:46 PM
F53712686: 2023_php_EOL_Composer.png
May 18 2024, 3:17 PM
F53712005: 2023_php_EOL_wmf.png
May 18 2024, 3:17 PM
Subscribers
A_smart_kitten
abi_
Aklapper
akosiaris
ArielGlenn
• bd808
BTullis
View All 38 Subscribers

Description

T271736: Migrate WMF production from PHP 7.2 to PHP 7.4 | T360995: Migrate Wikimedia production from PHP 8.1 to PHP 8.3

This is scheduled after T290536: Serve production traffic via Kubernetes, kept out of dependency tree for readability.

The next target for WMF production after PHP 7.4, is PHP 8.1. This will incorporate:

  • Assurance that this should work
    • CI configured to require PHP 8.1 to pass for all production code
      • Making CI passing and enforced ("voting") for MediaWiki core on PHP 8.0 and PHP 8.1.
      • Making CI passing wmf-deployed MediaWiki extensions and skins on PHP 8.0 and PHP 8.1.
    • Other tooling changes as needed
  • Preparation of Wikimedia production
  • Pre-launch testing
    • Benchmarks for different workloads and different clusters (non blocking) T383805 T321709
    • Add an option to WikimediaDebug to select which version is used. T372605
    • Add ability to do consistent client-side ramp-up via a PHP_ENGINE cookie. T377987
    • Add request routing for PHP_ENGINE cookie. T377042
  • MediaWiki production migration T383845
    • cookie-based traffic (mw-(web|api-ext)-next svc)
      • mw-web
      • mw-api-ext
    • cookie-less external traffic, and non-user facing traffic (via -migration releases)
      • mw-web
      • mw-api-ext
      • mw-api-int
      • mw-parsoid
      • mw-jobrunner
    • Everything else
      • Snapshot hosts ("dumps")
      • mw-videoscaler
      • mw-script
      • mw-wikifunctions
      • mw-cron
  • Migrate remaining cronjobs to mw-cron T341555
  • Decommission support for PHP 7.4 from production
  • Drop PHP 7.4 and 8.0 testing from CI, if not otherwise needed for release branches, and any relevant documentation

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update the PHP version in the other dumps configuration fileoperations/puppetproduction+1 -1
Upgrade PHP on the misc dumps worker - snapshot1016operations/puppetproduction+2 -1
Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016operations/puppetproduction+15 -1
MediaWiki-Docker: Switch PHP images to PHP 8.1mediawiki/coremaster+4 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add PHP 8.1 imagesrepos/releng/dev-images!23jforresterphp81main
Customize query in GitLab

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedJdforrester-WMFT328921 Drop PHP 7.4 and PHP 8.0 support from MediaWiki
ResolvedReedyT380185 Prepare for newer geoip2/geoip2 library
ResolvedKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
ResolvedtstarlingT292552 Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11
ResolvedJdforrester-WMFT316078 Make PHP 8.1 voting on development (master) branch of MW ecosystem (core, extensions, skins, libraries)
ResolvedJdforrester-WMFT320818 Configure CI to test the Wikimedia production gate code against PHP 8.1 instead of PHP 7.4
Resolved• larissagauliaT321709 PHP 8.1 Upgrade Performance <-> SRE Service Ops coordination
ResolvedJMeybohmT345561 Upgrade the MediaWiki servers to ICU 67
ResolvedNoneT373752 Build php-uuid package, and add to WMF production and CI
DuplicateNoneT378043 Route requests with PHP_ENGINE=8.1 cookie to "next" deployment via ATS plugin
ResolvedNoneT379874 ☂ PHP 8.1 issues found during WMF rollout/ramp up
ResolvedScott_FrenchT372604 Turn up PHP 8.1-flavored mw-debug k8s deployment
ResolveddduvallT370934 Build and publish multiple MediaWiki production images for a given set of PHP versions
ResolvedScott_FrenchT372603 Regenerate UcfirstOverrides.php for PHP 7.4 -> 8.1 transition
ResolvedFeature• bd808T378752 Install PHP 8.1 on deployment-prep bullseye MediaWiki instances
ResolvedScott_FrenchT383845 MediaWiki on PHP 8.1 production traffic ramp-up
ResolvedScott_FrenchT377042 Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext
ResolvedKrinkleT377987 Prepare sticky cookie for gradual PHP 8.1 rollout
ResolvedMSantosT379986 Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next
ResolvedScott_FrenchT380485 Transition parsoidtest1001 to PHP 8.1
ResolvedScott_FrenchT390225 Migrate scap's maintenance script invocations to PHP 8.1
ResolvedScott_FrenchT391057 Turn down MediaWiki image builds for PHP 7.4
ResolvedScott_FrenchT392938 Remove PHP 7.4 from deployment hosts
ResolvedScott_FrenchT394556 Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition
· · ·

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Reedy added a comment.Nov 22 2024, 1:18 PM

Noting also that getting the k8s stuff done was done before this in attempt to make future upgrades easier..

I’m not sure we’re immediately start going to go to 8.3 or similar when we’re done with the 8.1 upgrade, but it should result in a much shorter timeline going forward

Reedy mentioned this in T379509: [GENERAL] Fix implicit nullable types in Wikibase.Nov 24 2024, 9:10 PM
Krinkle mentioned this in T380185: Prepare for newer geoip2/geoip2 library.Nov 26 2024, 7:30 PM
Scott_French updated the task description. (Show Details)Dec 3 2024, 6:06 PM
Daimona mentioned this in T381728: Use PHP 8.3 in MediaWiki-Docker.Dec 7 2024, 11:03 PM
• bd808 added a subtask: T378752: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances.Dec 16 2024, 5:09 PM
A_smart_kitten subscribed.Jan 3 2025, 5:13 PM
ArthurTaylor mentioned this in T379481: [GENERAL] PHP 8.4 issues in WMDE/Wikidata libraries.Jan 6 2025, 10:42 AM
jijiki updated the task description. (Show Details)Jan 15 2025, 9:35 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJan 15 2025, 9:35 PM
jijiki updated the task description. (Show Details)Jan 15 2025, 9:37 PM
Scott_French added a subtask: T383845: MediaWiki on PHP 8.1 production traffic ramp-up.Jan 16 2025, 12:18 AM
jijiki added a subtask: T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).Jan 16 2025, 11:30 AM
jijiki changed the status of subtask T383845: MediaWiki on PHP 8.1 production traffic ramp-up from Open to In Progress.Jan 16 2025, 12:13 PM
Jdforrester-WMF mentioned this in T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).Jan 16 2025, 2:06 PM
Jdforrester-WMF updated the task description. (Show Details)Jan 16 2025, 2:08 PM
Krinkle closed subtask T379986: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next as Resolved.Jan 16 2025, 9:09 PM
Scott_French added a subtask: T380485: Transition parsoidtest1001 to PHP 8.1.Jan 17 2025, 3:45 PM
Scott_French mentioned this in T380485: Transition parsoidtest1001 to PHP 8.1.
Scott_French closed subtask T377042: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext as Resolved.Jan 22 2025, 6:41 PM
jijiki changed the status of subtask T379874: ☂ PHP 8.1 issues found during WMF rollout/ramp up from Open to In Progress.Jan 24 2025, 2:14 PM
jijiki changed the task status from Stalled to In Progress.Jan 24 2025, 2:18 PM
jijiki updated the task description. (Show Details)
Reedy mentioned this in T327094: Allow psr/container 2.x in Metrics Platform.Jan 24 2025, 3:49 PM
Scott_French updated the task description. (Show Details)Jan 24 2025, 3:57 PM
Tacsipacsi mentioned this in T385294: Enable sniffs to enforce use of type declarations.Feb 6 2025, 12:56 PM
Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Feb 11 2025, 7:36 PM
Ottomata added a project: Data-Engineering-Radar.
Daimona mentioned this in T387243: Do not display response statistics tab when all participants unregistered.Feb 26 2025, 3:21 AM
Reedy mentioned this in T387395: Merge master into wmf_deploy (Feb 26, 2025).Feb 26 2025, 9:55 PM
Jdforrester-WMF closed subtask T320818: Configure CI to test the Wikimedia production gate code against PHP 8.1 instead of PHP 7.4 as Resolved.Mar 3 2025, 9:05 PM
• bd808 closed subtask T378752: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances as Resolved.Mar 4 2025, 5:23 PM
Reedy mentioned this in T388723: Merge master into wmf_deploy (Mar 12, 2025).Mar 12 2025, 9:18 PM
Jdforrester-WMF updated the task description. (Show Details)Mar 17 2025, 1:09 PM
Novem_Linguae updated the task description. (Show Details)Mar 18 2025, 6:36 PM
Jdforrester-WMF updated the task description. (Show Details)Mar 18 2025, 7:12 PM
Krinkle added a subtask: T389243: Update Excimer to 1.2.3 in production.Mar 18 2025, 8:26 PM
Krinkle removed a subtask: T389243: Update Excimer to 1.2.3 in production.
Novem_Linguae mentioned this in T271736: Migrate WMF production from PHP 7.2 to PHP 7.4.Mar 19 2025, 12:24 AM
gerritbot added a comment.Mar 19 2025, 6:44 PM

Change #1129343 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016

https://gerrit.wikimedia.org/r/1129343

gerritbot added a project: Patch-For-Review.Mar 19 2025, 6:44 PM
gerritbot added a comment.Mar 20 2025, 7:02 AM

Change #1129343 merged by Btullis:

[operations/puppet@production] Upgrade snapshot hosts to PHP version 8.1 - except snapshot1016

https://gerrit.wikimedia.org/r/1129343

BTullis subscribed.Mar 20 2025, 7:10 AM

I am rolling out PHP version 8.1 to all snapshot hosts except snapshot1016. They should all start the 20250320 dump using the new version.
https://gerrit.wikimedia.org/r/1129343 applies cleanly.

We are still working on T352650: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes to migrate them to Kubernetes and Airflow, but for now they are still using the Debian packages.

BTullis added a comment.Mar 20 2025, 10:09 AM

7 of the 8 snapshot servers are now running PHP 8.1 for dumps.

btullis@cumin1002:~$ sudo cumin A:snapshot 'php --version'
8 hosts will be targeted:
snapshot[1010-1017].eqiad.wmnet
OK to proceed on 8 hosts? Enter the number of affected hosts to confirm or "q" to quit: 8
===== NODE GROUP =====                                                                                                                                                                                             
(1) snapshot1016.eqiad.wmnet                                                                                                                                                                                       
----- OUTPUT of 'php --version' -----                                                                                                                                                                              
PHP 7.4.33 (cli) (built: Apr 18 2024 14:41:42) ( NTS )                                                                                                                                                             
Copyright (c) The PHP Group                                                                                                                                                                                        
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies
===== NODE GROUP =====                                                                                                                                                                                             
(7) snapshot[1010-1015,1017].eqiad.wmnet                                                                                                                                                                           
----- OUTPUT of 'php --version' -----                                                                                                                                                                              
PHP 8.1.31 (cli) (built: Feb 25 2025 14:52:55) (NTS)                                                                                                                                                               
Copyright (c) The PHP Group                                                                                                                                                                                        
Zend Engine v4.1.31, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.31, Copyright (c), by Zend Technologies
================                                                                                                                                                                                                   
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (8/8) [00:00<00:00, 18.49hosts/s]
FAIL |                                                                                                                                                                             |   0% (0/8) [00:00<?, ?hosts/s]
100.0% (8/8) success ratio (>= 100.0% threshold) for command: 'php --version'.
100.0% (8/8) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

I had to upgrade libpcre2-8-0 manually after installation, due to T387276: PHP on Beta cluster fails due to mismatching PCRE - This wouldn't have been a problem if we had reimaged, but the upgrade didn't automaticaly install the newly pinned libpcre2-8-0 package.

We will follow up with snapshot1016 as soon as it is practicable.

Scott_French added a subtask: T390225: Migrate scap's maintenance script invocations to PHP 8.1.Mar 27 2025, 6:11 PM
gerritbot added a comment.Mar 31 2025, 10:12 AM

Change #1132568 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade PHP on the misc dumps worker - snapshot1016

https://gerrit.wikimedia.org/r/1132568

gerritbot added a comment.Mar 31 2025, 10:23 AM

Change #1132568 merged by Btullis:

[operations/puppet@production] Upgrade PHP on the misc dumps worker - snapshot1016

https://gerrit.wikimedia.org/r/1132568

BTullis updated the task description. (Show Details)Mar 31 2025, 10:28 AM

All snapshot hosts have now been upgraded to PHP 8.1.

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2025, 10:31 AM
gerritbot added a comment.Mar 31 2025, 10:46 AM

Change #1132584 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the PHP version in the other dumps configuration file

https://gerrit.wikimedia.org/r/1132584

gerritbot added a project: Patch-For-Review.Mar 31 2025, 10:46 AM

Change #1132584 merged by Btullis:

[operations/puppet@production] Update the PHP version in the other dumps configuration file

https://gerrit.wikimedia.org/r/1132584

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2025, 11:30 AM
Jdforrester-WMF updated the task description. (Show Details)Apr 1 2025, 7:43 PM
taavi updated the task description. (Show Details)Apr 2 2025, 3:59 PM
MSantos subscribed.Apr 7 2025, 4:07 PM
Scott_French added a subtask: T391057: Turn down MediaWiki image builds for PHP 7.4.Apr 7 2025, 5:42 PM
MShilova_WMF subscribed.Apr 8 2025, 4:08 PM
Scott_French closed subtask T383845: MediaWiki on PHP 8.1 production traffic ramp-up as Resolved.Apr 8 2025, 11:35 PM
Krinkle mentioned this in T391516: https://performance.wikimedia.org/php-profiling/ leads to 404 for all listed sources.Apr 9 2025, 7:13 PM
Scott_French closed subtask T390225: Migrate scap's maintenance script invocations to PHP 8.1 as Resolved.Apr 10 2025, 4:54 PM
Scott_French added a comment.EditedApr 15 2025, 6:55 PM

Following up on discussion out of band about remaining hosts that install PHP 7.4, here using php7.4-common [0] as a representative package:

Ignoring hosts that install "stock" debian packages (rather than our custom builds in component/icu67 or component/php74) for use cases unrelated to running MediaWiki code, we see:

  • mwdebug1001 and mwdebug2001: These are the remaining two mwdebug hosts that have not yet been reimaged to 8.1 (in progress).
  • mwmaint1002 and mwmaint2002: These hosts will remain on PHP 7.4 until they are ready for turndown, as periodic maintenance jobs and one-off maintenance scripts move to k8s (in progress).
  • deploy1003 and deploy2002: These hosts have 7.4 installed to support running maintenance scripts. Scap's use of maintenance scripts has been containerized and already migrated to 8.1. Users who might run one-off maintenance scripts on these hosts - rather than mwmaint hosts - will similarly onboard to k8s via the mwscript-k8s tool.
  • cloudweb1003.wikimedia.org, cloudweb1004.wikimedia.org, cloudweb2002-dev.wikimedia.org: This appears to be leftovers from the migration of wikitech to k8s. Specifically, these hosts still have apache2 and php7.4-fpm installed, however the apache2 configuration relevant to serving wikitech is disabled. Similarly, although /srv/mediawiki exists on these hosts, it has clearly not been touched since October 2024. I'll follow up on what the plan is for cleaning these up (edit: this is most likely T371378: Cleanup: Wikitech code leftovers).

[0] https://debmonitor.wikimedia.org/packages/php7.4-common

Scott_French closed subtask T380485: Transition parsoidtest1001 to PHP 8.1 as Resolved.Apr 19 2025, 1:01 AM
Krinkle removed a parent task: T328922: Drop PHP 8.0 support from MediaWiki.Apr 28 2025, 4:38 PM
Scott_French mentioned this in T392938: Remove PHP 7.4 from deployment hosts.Apr 29 2025, 5:38 PM
Scott_French added a subtask: T392938: Remove PHP 7.4 from deployment hosts.
Scott_French closed subtask T392938: Remove PHP 7.4 from deployment hosts as Resolved.May 7 2025, 7:26 PM
Scott_French added a subtask: T394556: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition.May 16 2025, 9:30 PM
Scott_French closed subtask T391057: Turn down MediaWiki image builds for PHP 7.4 as Resolved.May 22 2025, 9:33 PM
Jdforrester-WMF removed a parent task: T319055: Upgrade to psr/container 2.x.Jun 4 2025, 4:35 PM
Jdforrester-WMF updated the task description. (Show Details)Jun 4 2025, 4:39 PM
Clement_Goubert updated the task description. (Show Details)Jun 5 2025, 10:52 AM
Ladsgroup awarded a token.Jun 5 2025, 12:08 PM
MSantos awarded a token.Jun 5 2025, 12:42 PM
Daimona awarded a token.Jun 5 2025, 12:57 PM
Jdforrester-WMF closed subtask T373752: Build php-uuid package, and add to WMF production and CI as Resolved.Jun 5 2025, 7:11 PM
Danielyepezgarces awarded a token.Jun 7 2025, 3:25 AM
Chlod subscribed.Jun 7 2025, 4:13 AM
matmarex mentioned this in T396813: Make use of PHP 8.0 and 8.1 features in existing code.Jun 12 2025, 11:33 PM
jijiki removed a subtask: T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).Jun 16 2025, 3:40 PM
Krinkle mentioned this in T363639: web-auth/webauthn-lib must be upgraded to 4+ for PHP 8.2+ support.Jun 23 2025, 3:05 AM
Jdforrester-WMF updated the task description. (Show Details)Jun 24 2025, 2:55 PM
Scott_French closed subtask T394556: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition as Resolved.Jul 2 2025, 1:25 AM
Krinkle removed a subtask: T384294: Add support for JIT in PHP 8.4 images.Jul 17 2025, 3:29 AM
Novem_Linguae added a comment.Jul 17 2025, 3:35 AM

Looks like we've ticked most the boxes and resolved most the subtasks. Might be able to mark this one resolved soon :)

Krinkle updated the task description. (Show Details)Jul 17 2025, 4:55 AM
jijiki updated the task description. (Show Details)Jul 17 2025, 7:59 AM
jijiki updated the task description. (Show Details)
Umherirrender closed subtask T379874: ☂ PHP 8.1 issues found during WMF rollout/ramp up as Resolved.Jul 28 2025, 3:54 PM
Krinkle updated the task description. (Show Details)Jul 28 2025, 11:46 PM
Krinkle closed this task as Resolved.Aug 3 2025, 10:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits