Page MenuHomePhabricator

Migrate to PHP 7 in WMF production
Open, NormalPublic

Subscribers
Tokens
"Love" token, awarded by Ladsgroup."Love" token, awarded by Michael."Love" token, awarded by jcrespo."Love" token, awarded by kostajh."Yellow Medal" token, awarded by Liuxinyu970226."Mountain of Wealth" token, awarded by D3r1ck01."Yellow Medal" token, awarded by Ricordisamoa.
Assigned To
None
Authored By
tstarling, Sep 20 2017

Description

Following the strategy announcement by the HHVM team and subsequent discussion on wikitech-l, including benchmarks indicating HHVM no longer has a performance advantage, it is proposed that we migrate the WMF MediaWiki appservers from HHVM to PHP 7.

The current plan involves first migrating the appservers from jessie to stretch (T174431).

High-level checklist

Phase 1: PHP 7 and extensions/integrations installed:

  • Plain PHP 7.0 available on the command-line on all mw servers (e.g. for manual use via mwscript).
  • Deciding how to install PHP 7 and how to do CGI from Apache.
  • Plain PHP 7.0 available on mwdebug servers via php-fpm.

Phase 2: Verification of PHP 7 interaction with MediaWiki.

Features to be installed, tested, and known to work correctly; any behaviour/config differences between PHP7 and HHVM to be equalised or justified and documented.

  • Memcached client works fine with MW.
  • mysqli driver works fine with MW.
  • APCu works fine with MW.
  • Session handler work fine with MW.
  • Post-send functions works fine with MW. - T209981
  • Error and fatal error reporting works fine with MW. – T187147
  • Audit and fix as needed differences in INI settings. – T211488

Phase 3: Public opt-in via WikimediaDebug.

  • Add nginx handling to expose PHP7 for opted-in requests.
  • Provide opt-in via X-Wikimedia-Debug for mwdebug servers.
  • Debug profiling works fine with MW (e.g. xhprof, or tideways) – T206152
  • Sampling profiling works fine with MW. – T176916
  • Any misc issues we find from opt-in tests and anything blocking from PHP 7.2 support.

Phase 4: Public opt-in via beta feature.

  • Expose PHP7 vs. HHVM generation to JS in a header mw.config key.
  • Look at whether we should distinguish NavTiming metrics based on whether they're served by php7 or HHVM. (Not sure whether this is valuable.)
  • PHP 7.2 available on all mw servers via php-fpm (web, api, jobrunners, videoscalers)
  • Provide opt-in via MediaWiki Beta Feature for any web/api appserver.
  • Upgrade from PHP 7.0 to PHP 7.2.

Phase 5: PHP 7.2 on by default:

  • Ramp up A/B test of the beta feature for web/api app servers.
  • Switch default for all users on web appservers and api appservers.
  • Switch mwscript CLI on maintenance hosts and deployment hosts. – T195392
  • Switch job runners and video scalers.

Summary

Taking into account the lack funding for appserver work, as well as the end of the year fundraising and Christmas freezes, the (tentative!) timeline I proposed is:

  • Upgrade the appserver fleet (w/ HHVM) to Debian stretch, including the ICU migration, in Q3 FY17-18 (circa February/March 2018)
  • Begin PHP7 planning and initial implementation work in Q4 FY17-18, e.g. including a few test servers
  • Fund the work in FY18-19 and complete it early in the year (Q1 or Q2 at the latest)

That's from an Operations perspective -- I think other tasks related to this migration (such as CI, beta, code/extension changes etc.) can proceed semi-independently. Given our velocity, I think we'll end up be the ones blocking others, rather than us waiting on these.

During the SRE offsite/onsite we came up with the following plan:

  • We need to remove PHP 5 usage by May 2018 (branch date for the next Mediawiki LTS)
    • Only open issue is dumps, which is showing good progress (needs wikidiff build and review of php7 puppet patch)
  • We're planning the ICU migration (detailed in T177498) to happen this quarter:
    • HHVM package for icu57 needs to be upgraded to 3.18.8
    • Initial testing will happen on mwdebug
    • Also deployment-prep will be upgraded to the new packages (allows preparation of patches to use more recent Unicode)
    • For actual migration Community Liaisons need to be involved (user-visible effects)
    • First all app servers need to be upgraded, the run the conversion script (will take several days)
  • Once we have upgraded to HHVM/ICU57 on jessie, we can upgrade to HHVM 3.18/stretch (we should be able to complete that by Q4)
  • Once we're on stretch we can upgrade to HHVM 3.24
  • We came up with the following migration plan towards PHP 7:
    • Running php7-fpm on different port
    • Depending on cookie route to HHVM or php7-fpm for initial testing
    • Add PHP7 as a beta feature for more widespread exposure
    • Gradually ramp up the PHP usage towards all users
    • If all goes fine we should be complete before the fundraising silent period, otherwise after that
    • Profiling is currently a known blocker – T176916

Related Objects

StatusAssignedTask
StalledNone
StalledNone
StalledNone
OpenNone
OpenNone
StalledNone
OpenNone
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
ResolvedMoritzMuehlenhoff
ResolvedNone
ResolvedQuiddity
ResolvedLadsgroup
ResolvedJoe
ResolvedLegoktm
ResolvedLegoktm
Resolvedhashar
Resolvedhashar
Resolvedssastry
ResolvedSmalyshev
ResolvedLegoktm
Resolvedtstarling
Resolvedtstarling
Opentstarling
Resolvedtstarling
OpenNone
ResolvedNone
OpenNone
ResolvedNone
ResolvedDzahn
ResolvedRobH
ResolvedCmjohnson
ResolvedMoritzMuehlenhoff
ResolvedPapaul
ResolvedSmalyshev
Resolvedjcrespo
ResolvedJdforrester-WMF
ResolvedNone
Resolvedaaron
ResolvedJoe
ResolvedJoe
ResolvedKrinkle
ResolvedBPirkle
ResolvedJoe
ResolvedJoe
Resolved Gilles
ResolvedJoe
ResolvedAnomie
ResolvedAnomie
ResolvedKrinkle
Resolvedjijiki
ResolvedTgr
Resolvedjijiki
ResolvedMoritzMuehlenhoff
ResolvedArielGlenn
DuplicateNone
ResolvedReedy
ResolvedJoe
ResolvedKrinkle
OpenNone
OpenNone
Resolvedjijiki
ResolvedNone
ResolvedNone
Openjijiki

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Joe updated the task description. (Show Details)Nov 14 2018, 5:02 PM
BPirkle updated the task description. (Show Details)Nov 27 2018, 10:27 PM
Joe updated the task description. (Show Details)Nov 29 2018, 12:06 PM

Sorry, I found this very interesting case, but I don't know a better ticket to to report it. I saw on mediawiki errors logs that every minute, a request to [[User:Acer/Simple1]] on enwiki was done (oldid 844560394, in case it is edited or deleted). It returned 500 errors every time- the page is very large, contains 50000 wikilinks (I am guessing because it took a long time to render).

I tried once with PHP on mwdebug and it rendered in around 10 seconds only (now it works on HHVM, directly from cache). I confirmed this was not a one time thing, by repeating the expriment on /Simple2. This migration looks very promising, thank you!

Imarlier updated the task description. (Show Details)Dec 13 2018, 5:45 PM

Change 467096 merged by jenkins-bot:
[mediawiki/vagrant@master] Switch Vagrant to PHP 7.2

https://gerrit.wikimedia.org/r/467096

Reedy added a comment.Jan 5 2019, 7:00 PM

Luasandbox seems to be segfaulting on vagrant... dunno if it’s more widely replicable yet, or applicable to prod

Will try some more testing. See T212950 for more info

Dzahn added a subscriber: Dzahn.Jan 22 2019, 4:22 PM
Krinkle updated the task description. (Show Details)Jan 25 2019, 1:51 AM

Change 488524 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] mediawiki: Remove beta-cluster specific auto_prepend_file override

https://gerrit.wikimedia.org/r/488524

Change 488524 merged by Ottomata:
[operations/puppet@production] mediawiki: Remove beta-cluster specific auto_prepend_file override

https://gerrit.wikimedia.org/r/488524

Change 491462 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Absent libpcre3-dbg from hhvm::debug

https://gerrit.wikimedia.org/r/491462

GTirloni removed a subscriber: GTirloni.Feb 21 2019, 8:42 AM

Change 491462 abandoned by Muehlenhoff:
Absent libpcre3-dbg from hhvm::debug

Reason:
https://phabricator.wikimedia.org/T216712 obsoletes this

https://gerrit.wikimedia.org/r/491462

Ladsgroup added a subscriber: Ladsgroup.
TheDJ added a subscriber: TheDJ.Mar 8 2019, 12:57 PM

Just stumbled across this in CommonSettings:

$wgSVGConverters['rsvg-broken'] = '$path/rsvg-convert -w $width -h $height -o $output < $input';
if ( defined( 'HHVM_VERSION' ) ) {
	# Newer librsvg supports a sane security model by default and doesn't need our security patch
	$wgSVGConverters['rsvg-secure'] = '$path/rsvg-convert -u -w $width -h $height -o $output $input';
} else {
	# This converter will only work when rsvg has a suitable security patch
	$wgSVGConverters['rsvg-secure'] = '$path/rsvg-convert --no-external-files -w $width -h $height -o $output $input';
}

That's probably gonna need an update ?

greg removed a subscriber: greg.Mar 27 2019, 11:20 PM
Krinkle updated the task description. (Show Details)Apr 10 2019, 4:01 PM
Krinkle updated the task description. (Show Details)

Removing item for checking php70 and php71 issues. Any of those affecting php72, I've co-tagged with PHP 7.2 support.

awight added a subscriber: awight.May 11 2019, 8:05 PM
Krinkle updated the task description. (Show Details)May 13 2019, 10:39 AM