Page MenuHomePhabricator

[EPIC] Migrate MediaWiki-Vagrant base image to Debian Stretch
Closed, ResolvedPublic

Description

Track T174431: Upgrade mw* servers to Debian Stretch (using HHVM) and T176370: Migrate to PHP 7 in WMF production in MediaWiki-Vagrant.

The process should be similar to the Trusty->Jessie migration (T136429). Major changes in this migration include upgrading from Puppet 3.x to 4.x and PHP 5.x to PHP 7.x.

Major changes

  • Debian Stretch (Debian 9) base image
  • Default PHP runtime is Zend PHP 7.0 (HHVM available via role)
  • Database is MariaDB 10.1
  • Puppet 4

Testing the Stretch base image and Puppet profiles

Its recommended to test with a fresh MediaWiki-Vagrant checkout so if things go badly you can easily switch back to your original install and keep working.

$ git clone --recursive https://gerrit.wikimedia.org/r/mediawiki/vagrant mwv-stretch
$ cd mwv-stretch
$ git checkout stretch-migration
$ ./setup.sh
$ vagrant up

You can run vagrant roles list -e -1 to get a nice list of the roles you have enabled on your normal Trusty VM install to copy over to your Stretch testing VM. This one-liner liner might even do it for you:

$ cd mwv-stretch
$ vagrant roles enable $(cd ../vagrant; vagrant roles list -e -1)
$ vagrant provision

Role testing

Pure PHP roles (most extensions) should be fine, but it doesn't hurt to test them. The main things which need testing are roles which install additional system packages (e.g. elasticsearch), configure system packages (e.g. https), or install custom services (e.g. parsoid, restbase).

Provisioning: OK ✅, warnings ⚠️, errors 💣. Source: stretch, jessie.

StatusRoleStretchJessie
abusefilter
accountinfo
advancedsearch
analytics💣💣
antispam
antispoof
apex
apparmor💣💣
articlecreationworkflow
OKarticleplaceholder💣💣
babel
betafeatures
bouncehandler
buggy
campaigns
cassandra💣
categorytree
OKcentralauth💣
centralnotice
checkuser
cirrussearch💣💣 T183306
cite
citoid💣💣
cldr
codeeditor
codemirror💣
cologneblue
commons💣💣
commons_datasets💣💣
commonsmetadata
confirmedit
contactpage
contenttranslation💣💣
disableaccount
disambiguator
doublewiki
easytimeline
echo💣
education💣
OKelk💣T183071
emailauth
embedvideo
eventbus💣💣
OKeventloggingT183380
externalstore
featuredfeeds
fileannotations
fileimporter
flaggedrevs
flow💣💣
fss💣💣
fundraising💣💣
gadgets
gadgets2
geodata
geodata_elastic💣💣 T183306
geshi
gettingstarted💣💣
globalblocking💣
globalcssjs💣
globalpreferences💣
globalusage
globaluserpage
gpgmail
graph
graphoid⚠️💣
greystuff
guidedtour
gwtoolset💣
hadoop💣💣
headertabs
OKhhvm
hive💣💣
horizon💣💣
OKhttps⚠️T183304, T183302
hue💣💣
iabot
iegreview💣💣
imagemetrics
inputbox
interwiki
invitesignup
jsduck
jsonconfig
kafka💣
kartographer
kartographerwv
keystone💣💣
l10nupdate
labeledsectiontransclusion
langwikis
ldapauth
liquidthreads💣
lockdown
loginnotify💣💣
maps
massaction
massmessage💣
math💣
mathoid⚠️💣
mathsearch💣
OKmediawiki (default)
memcached
minerva
mleb
mobileapp
mobilecontentservice💣
OKmobilefrontend💣
modern
molhandler
monobook
OKmultimedia
OKmultimediaviewer
mysql_strict
OKnavigationtiming
newsletter
newusermessage
notebook💣
nuke
oathauth
oauth
oauthauthentication
offline💣
oozie💣💣
openbadges
ores
ores_service
pageassessments
pagedtiffhandler⚠️
pageimages
pagetriage
pageviewinfo
parserfunctions
parsoid⚠️💣
payments💣💣
OKpdfhandler
OKperformanceinspector
phabricator💣
phptags
phragile💣💣
pipeescape
poem
poolcounter💣T183384
OKpopups
private
proofreadpage
psr3
questycaptcha💣💣
quicksurveys
quips💣
quiz
raita💣💣
readinglists
OKrelatedarticles
renameuser
restbase💣
revisionslider
sal💣
sandboxlink
scholarships
score
OKscribunto💣T183888
securepoll
semanticextraspecialproperties
semanticmediawiki
semanticresultformats
semantictitle
sentry💣
shorturl
simple_miser💣⚠️
simple_performant💣
sitematrix⚠️💣
spark💣💣
OKstatsdT183374
striker💣💣
OKsvg
OKswiftT183354
syntaxhighlight
templatedata
templatesandbox
templatestyles
testwiki
textextracts
OKthree_dT183446
throttleoverride
OKthumb_on_404
thumbor💣T183462
tidy
OKtimedmediahandler
timeless
titleblacklist
torblock
translate
uls
uploadslink
OKuploadwizard⚠️
urlgetparameters
urlshortener
usermerge
variables
OKvarnish⚠️
OKvipsscaler
visualeditor💣
warnings_as_errors
whoops
widgets
wikibase_repo💣
OKwikidataT183307
wikidatapagebanner
wikidiff2💣
wikieditor
wikihiero
wikilabels
wikilove
wikimediaevents
wikimediaflow💣💣
wikimediaincubator
wikimediamaintenance⚠️
wikimediamessages
wikimediaproduction💣💣
wikimetrics💣💣
wikipediaextracts
wikispeech
wikitech💣💣
xanalytics
xhprofgui
youtube
zero💣T181915 (miss role::zend)

Reporting issues

Please create a subtask of this tracking task. It will be difficult to keep track of bugs reported as individual comments here.

Related Objects

StatusAssignedTask
Resolvedbd808
Resolvedbd808
DuplicateNone
Resolvedbd808
Resolvedbd808
ResolvedTgr
Resolvedbd808
ResolvedPaladox
DeclinedNone
InvalidNone
Resolvedbd808
OpenNone
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolved Gilles
Resolved Gilles
Resolved Gilles
Resolved Gilles
ResolvedPaladox
Resolvedbd808
Resolvedbd808
Resolvedmobrovac
ResolvedMoritzMuehlenhoff
ResolvedLegoktm
Resolvedawight
OpenNone
OpenNone
OpenNone
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolved Gilles
Resolved Gilles
Resolvedfgiunchedi
OpenNone
OpenNone
Resolvedbd808
Resolvedzeljkofilipin
Resolvedbd808

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I would assume they just haven't been re-packaged for Stretch and uploaded to our Debian repo. That was the issue with Poolcounter as well.

Gilles updated the task description. (Show Details)Dec 21 2017, 12:32 PM

Out of curiosity, do all roles provision correctly on master? (Jessie)

Out of curiosity, do all roles provision correctly on master? (Jessie)

I had the same thought. I can not run two mediawiki-vagrant machines at the same time (because of conflicting ports), I will run the same script for master/jessie as soon as this one is finished.

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Out of curiosity, do all roles provision correctly on master? (Jessie)

Done. Take a look at task description.

hashar updated the task description. (Show Details)Dec 22 2017, 2:38 PM
Tgr added a comment.Dec 22 2017, 5:12 PM

I had the same thought. I can not run two mediawiki-vagrant machines at the same time (because of conflicting ports)

You need to do something like

vagrant config http_port XXX
vagrant config https_port YYY
vagrant config static_ip ZZZ

and after that they work fine in parallel.

I have created a test suite that attempts to compile the puppet catalog for each of the roles ( T183570 and https://gerrit.wikimedia.org/r/#/c/399834/ ).

It has a few failures worth looking at P6503:

role::phragileInvalid parameter deploy_dir on Phabricator::Config[phabricator.show-prototypes] at puppet/modules/role/manifests/phragile.pp:17
role:sparkCould not find resource 'Class[Cdh::Hive]' for relationship on 'Class[Cdh::Spark]'
role:thumborDuplicate declaration: Package[python-logstash] is already declared; cannot redeclare at puppet/modules/thumbor/manifests/init.pp:63
role::wikimediaproductionDuplicate declaration: Mediawiki::Extension[GlobalUsage] is already declared in file puppet/modules/role/manifests/commons.pp:38; cannot redeclare at puppet/modules/role/manifests/globalusage.pp:5
role::wikitechDuplicate declaration: Mediawiki::Extension[Renameuser] is already declared in file puppet/modules/role/manifests/renameuser.pp:6; cannot redeclare at puppet/modules/role/manifests/wikitech.pp:58

the role::https failure is due to the test suite not setting facts

hashar updated the task description. (Show Details)Dec 24 2017, 2:29 PM
bd808 updated the task description. (Show Details)Dec 25 2017, 3:50 AM
bd808 updated the task description. (Show Details)Dec 25 2017, 3:52 AM
bd808 updated the task description. (Show Details)
bd808 updated the task description. (Show Details)Dec 25 2017, 5:58 AM

You need to do something like

vagrant config http_port XXX
vagrant config https_port YYY
vagrant config static_ip ZZZ

Thanks!

Change 401510 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vagrant@stretch-migration] services ports must be Integer

https://gerrit.wikimedia.org/r/401510

Change 401510 merged by jenkins-bot:
[mediawiki/vagrant@stretch-migration] services ports must be Integer

https://gerrit.wikimedia.org/r/401510

bd808 updated the task description. (Show Details)Jan 3 2018, 4:15 AM
Gilles changed the status of subtask T183384: Fix Poolcounter role for Stretch from Open to Stalled.Jan 8 2018, 9:34 AM
Gilles changed the status of subtask T183462: Fix Thumbor role for Stretch from Open to Stalled.
Gilles updated the task description. (Show Details)

Change 404399 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[mediawiki/vagrant@master] Merge remote-tracking branch 'origin/stretch-migration'

https://gerrit.wikimedia.org/r/404399

Change 404399 merged by jenkins-bot:
[mediawiki/vagrant@master] Merge remote-tracking branch 'origin/stretch-migration'

https://gerrit.wikimedia.org/r/404399

bd808 moved this task from In Progress to Done on the MediaWiki-Vagrant board.Jan 16 2018, 4:43 AM

Change 404414 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/vagrant@master] Bump Vagrant gem version to the one used in Stretch

https://gerrit.wikimedia.org/r/404414

I have migrated the CI job to use a Docker container based on Debian Stretch. That required Vagrant to be bumped to 1.9.x https://gerrit.wikimedia.org/r/#/c/404414/

There might be some oddity for people not having ruby 2.3 (Vagrant depends on listen which requires ruby 2.3) but nothing that rvm can't solve.

Change 404414 merged by jenkins-bot:
[mediawiki/vagrant@master] Bump Vagrant gem version to the one used in Stretch

https://gerrit.wikimedia.org/r/404414

Krinkle renamed this task from [EPIC] Migrate base image to Debian Stretch to [EPIC] Migrate MediaWiki-Vagrant base image to Debian Stretch.Jul 26 2018, 11:17 PM
bd808 closed this task as Resolved.Jul 26 2018, 11:20 PM
bd808 claimed this task.

There are open bugs, but this epic has served its purpose. Thanks to everyone who helped with this project!