Page MenuHomePhabricator

[EPIC] Migrate MediaWiki-Vagrant base image to Debian Stretch
Closed, ResolvedPublic

Description

Track T174431: Upgrade mw* servers to Debian Stretch (using HHVM) and T176370: Migrate to PHP 7 in WMF production in MediaWiki-Vagrant.

The process should be similar to the Trusty->Jessie migration (T136429). Major changes in this migration include upgrading from Puppet 3.x to 4.x and PHP 5.x to PHP 7.x.

Major changes

  • Debian Stretch (Debian 9) base image
  • Default PHP runtime is Zend PHP 7.0 (HHVM available via role)
  • Database is MariaDB 10.1
  • Puppet 4

Testing the Stretch base image and Puppet profiles

Its recommended to test with a fresh MediaWiki-Vagrant checkout so if things go badly you can easily switch back to your original install and keep working.

$ git clone --recursive https://gerrit.wikimedia.org/r/mediawiki/vagrant mwv-stretch
$ cd mwv-stretch
$ git checkout stretch-migration
$ ./setup.sh
$ vagrant up

You can run vagrant roles list -e -1 to get a nice list of the roles you have enabled on your normal Trusty VM install to copy over to your Stretch testing VM. This one-liner liner might even do it for you:

$ cd mwv-stretch
$ vagrant roles enable $(cd ../vagrant; vagrant roles list -e -1)
$ vagrant provision

Role testing

Pure PHP roles (most extensions) should be fine, but it doesn't hurt to test them. The main things which need testing are roles which install additional system packages (e.g. elasticsearch), configure system packages (e.g. https), or install custom services (e.g. parsoid, restbase).

Provisioning: OK โœ…, warnings โš ๏ธ, errors ๐Ÿ’ฃ. Source: stretch, jessie.

StatusRoleStretchJessie
abusefilterโœ…โœ…
accountinfoโœ…โœ…
advancedsearchโœ…โœ…
analytics๐Ÿ’ฃ๐Ÿ’ฃ
antispamโœ…โœ…
antispoofโœ…โœ…
apexโœ…โœ…
apparmor๐Ÿ’ฃ๐Ÿ’ฃ
articlecreationworkflowโœ…โœ…
OKarticleplaceholder๐Ÿ’ฃ๐Ÿ’ฃ
babelโœ…โœ…
betafeaturesโœ…โœ…
bouncehandlerโœ…โœ…
buggyโœ…โœ…
campaignsโœ…โœ…
cassandra๐Ÿ’ฃโœ…
categorytreeโœ…โœ…
OKcentralauth๐Ÿ’ฃโœ…
centralnoticeโœ…โœ…
checkuserโœ…โœ…
cirrussearch๐Ÿ’ฃ๐Ÿ’ฃ T183306
citeโœ…โœ…
citoid๐Ÿ’ฃ๐Ÿ’ฃ
cldrโœ…โœ…
codeeditorโœ…โœ…
codemirror๐Ÿ’ฃโœ…
cologneblueโœ…โœ…
commons๐Ÿ’ฃ๐Ÿ’ฃ
commons_datasets๐Ÿ’ฃ๐Ÿ’ฃ
commonsmetadataโœ…โœ…
confirmeditโœ…โœ…
contactpageโœ…โœ…
contenttranslation๐Ÿ’ฃ๐Ÿ’ฃ
disableaccountโœ…โœ…
disambiguatorโœ…โœ…
doublewikiโœ…โœ…
easytimelineโœ…โœ…
echo๐Ÿ’ฃโœ…
education๐Ÿ’ฃโœ…
OKelk๐Ÿ’ฃโœ… T183071
emailauthโœ…โœ…
embedvideoโœ…โœ…
eventbus๐Ÿ’ฃ๐Ÿ’ฃ
OKeventloggingโœ…โœ… T183380
externalstoreโœ…โœ…
featuredfeedsโœ…โœ…
fileannotationsโœ…โœ…
fileimporterโœ…โœ…
flaggedrevsโœ…โœ…
flow๐Ÿ’ฃ๐Ÿ’ฃ
fss๐Ÿ’ฃ๐Ÿ’ฃ
fundraising๐Ÿ’ฃ๐Ÿ’ฃ
gadgetsโœ…โœ…
gadgets2โœ…โœ…
geodataโœ…โœ…
geodata_elastic๐Ÿ’ฃ๐Ÿ’ฃ T183306
geshiโœ…โœ…
gettingstarted๐Ÿ’ฃ๐Ÿ’ฃ
globalblocking๐Ÿ’ฃโœ…
globalcssjs๐Ÿ’ฃโœ…
globalpreferences๐Ÿ’ฃโœ…
globalusageโœ…โœ…
globaluserpageโœ…โœ…
gpgmailโœ…โœ…
graphโœ…โœ…
graphoidโš ๏ธ๐Ÿ’ฃ
greystuffโœ…โœ…
guidedtourโœ…โœ…
gwtoolset๐Ÿ’ฃโœ…
hadoop๐Ÿ’ฃ๐Ÿ’ฃ
headertabsโœ…โœ…
OKhhvmโœ…โœ…
hive๐Ÿ’ฃ๐Ÿ’ฃ
horizon๐Ÿ’ฃ๐Ÿ’ฃ
OKhttpsโš ๏ธโœ… T183304, T183302
hue๐Ÿ’ฃ๐Ÿ’ฃ
iabotโœ…โœ…
iegreview๐Ÿ’ฃ๐Ÿ’ฃ
imagemetricsโœ…โœ…
inputboxโœ…โœ…
interwikiโœ…โœ…
invitesignupโœ…โœ…
jsduckโœ…โœ…
jsonconfigโœ…โœ…
kafka๐Ÿ’ฃโœ…
kartographerโœ…โœ…
kartographerwvโœ…โœ…
keystone๐Ÿ’ฃ๐Ÿ’ฃ
l10nupdateโœ…โœ…
labeledsectiontransclusionโœ…โœ…
langwikisโœ…โœ…
ldapauthโœ…โœ…
liquidthreads๐Ÿ’ฃโœ…
lockdownโœ…โœ…
loginnotify๐Ÿ’ฃ๐Ÿ’ฃ
mapsโœ…โœ…
massactionโœ…โœ…
massmessage๐Ÿ’ฃโœ…
math๐Ÿ’ฃโœ…
mathoidโš ๏ธ๐Ÿ’ฃ
mathsearch๐Ÿ’ฃโœ…
OKmediawiki (default)โœ…โœ…
memcachedโœ…โœ…
minervaโœ…โœ…
mlebโœ…โœ…
mobileappโœ…โœ…
mobilecontentservice๐Ÿ’ฃโœ…
OKmobilefrontendโœ…๐Ÿ’ฃ
modernโœ…โœ…
molhandlerโœ…โœ…
monobookโœ…โœ…
OKmultimediaโœ…โœ…
OKmultimediaviewerโœ…โœ…
mysql_strictโœ…โœ…
OKnavigationtimingโœ…โœ…
newsletterโœ…โœ…
newusermessageโœ…โœ…
notebookโœ…๐Ÿ’ฃ
nukeโœ…โœ…
oathauthโœ…โœ…
oauthโœ…โœ…
oauthauthenticationโœ…โœ…
offline๐Ÿ’ฃโœ…
oozie๐Ÿ’ฃ๐Ÿ’ฃ
openbadgesโœ…โœ…
oresโœ…โœ…
ores_serviceโœ…โœ…
pageassessmentsโœ…โœ…
pagedtiffhandlerโœ…โš ๏ธ
pageimagesโœ…โœ…
pagetriageโœ…โœ…
pageviewinfoโœ…โœ…
parserfunctionsโœ…โœ…
parsoidโš ๏ธ๐Ÿ’ฃ
payments๐Ÿ’ฃ๐Ÿ’ฃ
OKpdfhandlerโœ…โœ…
OKperformanceinspectorโœ…โœ…
phabricator๐Ÿ’ฃโœ…
phptagsโœ…โœ…
phragile๐Ÿ’ฃ๐Ÿ’ฃ
pipeescapeโœ…โœ…
poemโœ…โœ…
poolcounter๐Ÿ’ฃโœ… T183384
OKpopupsโœ…โœ…
privateโœ…โœ…
proofreadpageโœ…โœ…
psr3โœ…โœ…
questycaptcha๐Ÿ’ฃ๐Ÿ’ฃ
quicksurveysโœ…โœ…
quips๐Ÿ’ฃโœ…
quizโœ…โœ…
raita๐Ÿ’ฃ๐Ÿ’ฃ
readinglistsโœ…โœ…
OKrelatedarticlesโœ…โœ…
renameuserโœ…โœ…
restbase๐Ÿ’ฃโœ…
revisionsliderโœ…โœ…
sal๐Ÿ’ฃโœ…
sandboxlinkโœ…โœ…
scholarshipsโœ…โœ…
scoreโœ…โœ…
OKscribunto๐Ÿ’ฃโœ… T183888
securepollโœ…โœ…
semanticextraspecialpropertiesโœ…โœ…
semanticmediawikiโœ…โœ…
semanticresultformatsโœ…โœ…
semantictitleโœ…โœ…
sentry๐Ÿ’ฃโœ…
shorturlโœ…โœ…
simple_miser๐Ÿ’ฃโš ๏ธ
simple_performant๐Ÿ’ฃโœ…
sitematrixโš ๏ธ๐Ÿ’ฃ
spark๐Ÿ’ฃ๐Ÿ’ฃ
OKstatsdโœ…โœ…T183374
striker๐Ÿ’ฃ๐Ÿ’ฃ
OKsvgโœ…โœ…
OKswiftโœ…โœ… T183354
syntaxhighlightโœ…โœ…
templatedataโœ…โœ…
templatesandboxโœ…โœ…
templatestylesโœ…โœ…
testwikiโœ…โœ…
textextractsโœ…โœ…
OKthree_dโœ…โœ… T183446
throttleoverrideโœ…โœ…
OKthumb_on_404โœ…โœ…
thumbor๐Ÿ’ฃโœ… T183462
tidyโœ…โœ…
OKtimedmediahandlerโœ…โœ…
timelessโœ…โœ…
titleblacklistโœ…โœ…
torblockโœ…โœ…
translateโœ…โœ…
ulsโœ…โœ…
uploadslinkโœ…โœ…
OKuploadwizardโœ…โš ๏ธ
urlgetparametersโœ…โœ…
urlshortenerโœ…โœ…
usermergeโœ…โœ…
variablesโœ…โœ…
OKvarnishโš ๏ธโœ…
OKvipsscalerโœ…โœ…
visualeditor๐Ÿ’ฃโœ…
warnings_as_errorsโœ…โœ…
whoopsโœ…โœ…
widgetsโœ…โœ…
wikibase_repo๐Ÿ’ฃโœ…
OKwikidataโœ…โœ… T183307
wikidatapagebannerโœ…โœ…
wikidiff2๐Ÿ’ฃโœ…
wikieditorโœ…โœ…
wikihieroโœ…โœ…
wikilabelsโœ…โœ…
wikiloveโœ…โœ…
wikimediaeventsโœ…โœ…
wikimediaflow๐Ÿ’ฃ๐Ÿ’ฃ
wikimediaincubatorโœ…โœ…
wikimediamaintenanceโš ๏ธโœ…
wikimediamessagesโœ…โœ…
wikimediaproduction๐Ÿ’ฃ๐Ÿ’ฃ
wikimetrics๐Ÿ’ฃ๐Ÿ’ฃ
wikipediaextractsโœ…โœ…
wikispeechโœ…โœ…
wikitech๐Ÿ’ฃ๐Ÿ’ฃ
xanalyticsโœ…โœ…
xhprofguiโœ…โœ…
youtubeโœ…โœ…
zero๐Ÿ’ฃโœ… T181915 (miss role::zend)

Reporting issues

Please create a subtask of this tracking task. It will be difficult to keep track of bugs reported as individual comments here.

Related Objects

StatusAssignedTask
Resolvedbd808
Resolvedbd808
DuplicateNone
Resolvedbd808
Resolvedbd808
ResolvedTgr
Resolvedbd808
ResolvedPaladox
DeclinedNone
InvalidNone
Resolvedbd808
OpenNone
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedGilles
ResolvedPaladox
Resolvedbd808
Resolvedbd808
Resolvedmobrovac
ResolvedMoritzMuehlenhoff
ResolvedLegoktm
Resolvedawight
OpenNone
OpenNone
OpenNone
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
ResolvedGilles
ResolvedGilles
Resolvedfgiunchedi
OpenNone
OpenNone
Resolvedbd808
Resolvedzeljkofilipin
Resolvedbd808

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I would assume they just haven't been re-packaged for Stretch and uploaded to our Debian repo. That was the issue with Poolcounter as well.

Gilles updated the task description. (Show Details)Dec 21 2017, 12:32 PM

Out of curiosity, do all roles provision correctly on master? (Jessie)

Out of curiosity, do all roles provision correctly on master? (Jessie)

I had the same thought. I can not run two mediawiki-vagrant machines at the same time (because of conflicting ports), I will run the same script for master/jessie as soon as this one is finished.

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Out of curiosity, do all roles provision correctly on master? (Jessie)

Done. Take a look at task description.

hashar updated the task description. (Show Details)Dec 22 2017, 2:38 PM
Tgr added a comment.Dec 22 2017, 5:12 PM

I had the same thought. I can not run two mediawiki-vagrant machines at the same time (because of conflicting ports)

You need to do something like

vagrant config http_port XXX
vagrant config https_port YYY
vagrant config static_ip ZZZ

and after that they work fine in parallel.

I have created a test suite that attempts to compile the puppet catalog for each of the roles ( T183570 and https://gerrit.wikimedia.org/r/#/c/399834/ ).

It has a few failures worth looking at P6503:

role::phragileInvalid parameter deploy_dir on Phabricator::Config[phabricator.show-prototypes] at puppet/modules/role/manifests/phragile.pp:17
role:sparkCould not find resource 'Class[Cdh::Hive]' for relationship on 'Class[Cdh::Spark]'
role:thumborDuplicate declaration: Package[python-logstash] is already declared; cannot redeclare at puppet/modules/thumbor/manifests/init.pp:63
role::wikimediaproductionDuplicate declaration: Mediawiki::Extension[GlobalUsage] is already declared in file puppet/modules/role/manifests/commons.pp:38; cannot redeclare at puppet/modules/role/manifests/globalusage.pp:5
role::wikitechDuplicate declaration: Mediawiki::Extension[Renameuser] is already declared in file puppet/modules/role/manifests/renameuser.pp:6; cannot redeclare at puppet/modules/role/manifests/wikitech.pp:58

the role::https failure is due to the test suite not setting facts

hashar updated the task description. (Show Details)Dec 24 2017, 2:29 PM
bd808 updated the task description. (Show Details)Dec 25 2017, 3:50 AM
bd808 updated the task description. (Show Details)Dec 25 2017, 3:52 AM
bd808 updated the task description. (Show Details)
bd808 updated the task description. (Show Details)Dec 25 2017, 5:58 AM

You need to do something like

vagrant config http_port XXX
vagrant config https_port YYY
vagrant config static_ip ZZZ

Thanks!

Change 401510 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vagrant@stretch-migration] services ports must be Integer

https://gerrit.wikimedia.org/r/401510

Change 401510 merged by jenkins-bot:
[mediawiki/vagrant@stretch-migration] services ports must be Integer

https://gerrit.wikimedia.org/r/401510

bd808 updated the task description. (Show Details)Jan 3 2018, 4:15 AM
Gilles changed the status of subtask T183384: Fix Poolcounter role for Stretch from Open to Stalled.Jan 8 2018, 9:34 AM
Gilles changed the status of subtask T183462: Fix Thumbor role for Stretch from Open to Stalled.
Gilles updated the task description. (Show Details)

Change 404399 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[mediawiki/vagrant@master] Merge remote-tracking branch 'origin/stretch-migration'

https://gerrit.wikimedia.org/r/404399

Change 404399 merged by jenkins-bot:
[mediawiki/vagrant@master] Merge remote-tracking branch 'origin/stretch-migration'

https://gerrit.wikimedia.org/r/404399

bd808 moved this task from In Progress to Done on the MediaWiki-Vagrant board.Jan 16 2018, 4:43 AM

Change 404414 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vagrant@master] Bump Vagrant gem version to the one used in Stretch

https://gerrit.wikimedia.org/r/404414

I have migrated the CI job to use a Docker container based on Debian Stretch. That required Vagrant to be bumped to 1.9.x https://gerrit.wikimedia.org/r/#/c/404414/

There might be some oddity for people not having ruby 2.3 (Vagrant depends on listen which requires ruby 2.3) but nothing that rvm can't solve.

Change 404414 merged by jenkins-bot:
[mediawiki/vagrant@master] Bump Vagrant gem version to the one used in Stretch

https://gerrit.wikimedia.org/r/404414

Krinkle renamed this task from [EPIC] Migrate base image to Debian Stretch to [EPIC] Migrate MediaWiki-Vagrant base image to Debian Stretch.Jul 26 2018, 11:17 PM
bd808 closed this task as Resolved.Jul 26 2018, 11:20 PM
bd808 claimed this task.

There are open bugs, but this epic has served its purpose. Thanks to everyone who helped with this project!