Page MenuHomePhabricator

Upgrade dump hosts to stretch with php7
Closed, ResolvedPublic

Description

For multiple reasons we have held back on upgrading our dumps infrastructure from Ubuntu Trusty to a newer distribution. Also, because of some HHVM bugs, we never switched to use HHVM for dumps and we're still producing those with PHP 5.5.

Since we now want to drop PHP 5.x support from MediaWiki 1.31, we need to either upgrade the dump hosts to a newer distribution with a recent version of HHVM (which will be able to run dumps) or to one using PHP 7.0. So our options will be:

  • Upgrade to debian jessie and run dumps with HHVM
  • Upgrade to debian stretch and run dumps with PHP 7.0

This task is a blocker to the actual implementation of the RfC, and should track progress on this.

Event Timeline

Joe created this task.Nov 21 2017, 9:55 AM

My preference is stretch with php7 if the packages are available relatively soon (December).

matmarex removed a subscriber: matmarex.Nov 21 2017, 7:00 PM
Joe removed a project: Proposal.Nov 22 2017, 9:25 PM
Joe moved this task from Inbox to Backlog on the TechCom-RFC board.
Joe removed a project: TechCom-RFC.
ArielGlenn renamed this task from Upgrade dump hosts to a newer distribution to Upgrade dump hosts to stretch with php7.Jan 5 2018, 9:35 AM
Krinkle removed a subscriber: Krinkle.Jan 8 2018, 7:41 PM
hoo added a subscriber: hoo.Jan 16 2018, 4:03 AM
Joe added a comment.EditedApr 19 2018, 1:51 PM

Status update:

  • SREs are working on the blocker task T184854 deploying the new HHVM version to production. We hope to have completed the rolling cache wipe early next week (so, by April 25th)
  • Once that is done, proceed with T161509 immediately and run some tests
  • In the meantime, also reimage snapshot1005 so that the dumps run starting on May 1st there will be run with PHP 7.0; the other dumps will be run with php 5.5. This is the last batch of jobs that will be run on php5.
  • When those end (around May 15th), we will reimage the remaining servers.

This means that after May 1st the dependency on php 5.5 in MediaWiki can probably be dropped.

Change 428595 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] make addschanges dumps use configured php on snaphot hosts

https://gerrit.wikimedia.org/r/428595

Change 428595 merged by ArielGlenn:
[operations/puppet@production] make addschanges dumps use configured php on snaphot hosts

https://gerrit.wikimedia.org/r/428595

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1005.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201804280435_ariel_14927_snapshot1005_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['snapshot1005.eqiad.wmnet']

and were ALL successful.

Change 429539 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7 for all dumps-relate things on snapshot1005

https://gerrit.wikimedia.org/r/429539

Change 429539 merged by ArielGlenn:
[operations/puppet@production] use php7 for all dumps-related things on snapshot1005

https://gerrit.wikimedia.org/r/429539

EddieGP removed a subscriber: EddieGP.Apr 29 2018, 5:40 PM

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1006.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201804301443_ariel_19438_snapshot1006_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['snapshot1006.eqiad.wmnet']

and were ALL successful.

Change 429822 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7.0 for all dumps-related things on snapshot1006

https://gerrit.wikimedia.org/r/429822

Change 429822 merged by ArielGlenn:
[operations/puppet@production] use php7.0 for all dumps-related things on snapshot1006

https://gerrit.wikimedia.org/r/429822

The last host to upgrade, snapshot1007, can be done when the current dump run completes, or at least the jobs on that host complete. The misc cron jobs host, see T190112, is due to arrive today and with any luck will be available for install early next week, at which point cron jobs could be moved there.

Currently waiting for the following dumps to complete: commonswiki, dewiki, frwiki, ruwiki, and the dreaded mediawikiwiki flow history dumps. I estimate three days for all of these.

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1007.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201805200735_ariel_4481_snapshot1007_eqiad_wmnet.log.

Change 434141 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7.0 for dumps and related jobs on snapshot1007 now

https://gerrit.wikimedia.org/r/434141

Change 434141 merged by ArielGlenn:
[operations/puppet@production] use php7.0 for dumps and related jobs on snapshot1007 now

https://gerrit.wikimedia.org/r/434141

Completed auto-reimage of hosts:

['snapshot1007.eqiad.wmnet']

and were ALL successful.

ArielGlenn closed this task as Resolved.May 20 2018, 9:15 AM

This task is finally complete!

ArielGlenn moved this task from Active to Done on the Dumps-Generation board.May 20 2018, 9:16 AM