Page MenuHomePhabricator
Create Task
Maniphest T181029

Upgrade dump hosts to stretch with php7
Closed, ResolvedPublic
Actions

Description

For multiple reasons we have held back on upgrading our dumps infrastructure from Ubuntu Trusty to a newer distribution. Also, because of some HHVM bugs, we never switched to use HHVM for dumps and we're still producing those with PHP 5.5.

Since we now want to drop PHP 5.x support from MediaWiki 1.31, we need to either upgrade the dump hosts to a newer distribution with a recent version of HHVM (which will be able to run dumps) or to one using PHP 7.0. So our options will be:

  • Upgrade to debian jessie and run dumps with HHVM
  • Upgrade to debian stretch and run dumps with PHP 7.0

This task is a blocker to the actual implementation of the RfC, and should track progress on this.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
use php7.0 for dumps and related jobs on snapshot1007 nowArielGlennoperations/puppetproduction+1 -1
use php7.0 for all dumps-related things on snapshot1006ArielGlennoperations/puppetproduction+1 -1
use php7 for all dumps-related things on snapshot1005ArielGlennoperations/puppetproduction+1 -1
make addschanges dumps use configured php on snaphot hostsArielGlennoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Joe created this task.Nov 21 2017, 9:55 AM
ArielGlenn added projects: Dumps-Generation, User-ArielGlenn.Nov 21 2017, 9:57 AM

My preference is stretch with php7 if the packages are available relatively soon (December).

matmarex unsubscribed.Nov 21 2017, 7:00 PM
Joe removed a project: Proposal.Nov 22 2017, 9:25 PM
Joe moved this task from P1: Define to Old on the TechCom-RFC board.
Joe removed a project: TechCom-RFC.
ArielGlenn renamed this task from Upgrade dump hosts to a newer distribution to Upgrade dump hosts to stretch with php7.Jan 5 2018, 9:35 AM
ArielGlenn added a subtask: T161509: Test php7 on snapshot1001 for dumps.
Krinkle unsubscribed.Jan 8 2018, 7:41 PM
ArielGlenn mentioned this in T162245: Enable GC for HHVM CLI (at least for dump runners).Jan 15 2018, 4:52 PM
hoo subscribed.Jan 16 2018, 4:03 AM
Dzahn mentioned this in T186288: replace all Ubuntu (trusty) hosts in production with Debian.Feb 8 2018, 10:00 PM
ArielGlenn moved this task from Backlog to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Mar 12 2018, 11:23 AM
ArielGlenn removed a project: User-ArielGlenn.Mar 29 2018, 2:43 PM
Addshore unsubscribed.Apr 3 2018, 9:32 AM
dcausse subscribed.Apr 13 2018, 8:52 AM
Joe added a comment.EditedApr 19 2018, 1:51 PM

Status update:

  • SREs are working on the blocker task T184854 deploying the new HHVM version to production. We hope to have completed the rolling cache wipe early next week (so, by April 25th)
  • Once that is done, proceed with T161509 immediately and run some tests
  • In the meantime, also reimage snapshot1005 so that the dumps run starting on May 1st there will be run with PHP 7.0; the other dumps will be run with php 5.5. This is the last batch of jobs that will be run on php5.
  • When those end (around May 15th), we will reimage the remaining servers.

This means that after May 1st the dependency on php 5.5 in MediaWiki can probably be dropped.

Reedy mentioned this in T172165: Require either PHP 7.0+ or HHVM in MW 1.31.Apr 20 2018, 7:36 AM
MGChecker subscribed.Apr 20 2018, 7:57 AM
gerritbot subscribed.Apr 24 2018, 10:24 AM

Change 428595 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] make addschanges dumps use configured php on snaphot hosts

https://gerrit.wikimedia.org/r/428595

gerritbot added a project: Patch-For-Review.Apr 24 2018, 10:24 AM
gerritbot added a comment.Apr 24 2018, 10:29 AM

Change 428595 merged by ArielGlenn:
[operations/puppet@production] make addschanges dumps use configured php on snaphot hosts

https://gerrit.wikimedia.org/r/428595

ArielGlenn closed subtask T184258: get a snapshot instance running in beta with stretch, php7 as Resolved.Apr 24 2018, 3:33 PM
ops-monitoring-bot subscribed.Apr 28 2018, 4:35 AM

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1005.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201804280435_ariel_14927_snapshot1005_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Apr 28 2018, 5:09 AM

Completed auto-reimage of hosts:

['snapshot1005.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Apr 28 2018, 5:21 AM

Change 429539 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7 for all dumps-relate things on snapshot1005

https://gerrit.wikimedia.org/r/429539

gerritbot added a comment.Apr 28 2018, 5:24 AM

Change 429539 merged by ArielGlenn:
[operations/puppet@production] use php7 for all dumps-related things on snapshot1005

https://gerrit.wikimedia.org/r/429539

EddieGP unsubscribed.Apr 29 2018, 5:40 PM
ops-monitoring-bot added a comment.Apr 30 2018, 2:44 PM

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1006.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201804301443_ariel_19438_snapshot1006_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Apr 30 2018, 3:18 PM

Completed auto-reimage of hosts:

['snapshot1006.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Apr 30 2018, 3:20 PM

Change 429822 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7.0 for all dumps-related things on snapshot1006

https://gerrit.wikimedia.org/r/429822

gerritbot added a comment.Apr 30 2018, 3:25 PM

Change 429822 merged by ArielGlenn:
[operations/puppet@production] use php7.0 for all dumps-related things on snapshot1006

https://gerrit.wikimedia.org/r/429822

ArielGlenn closed subtask T161509: Test php7 on snapshot1001 for dumps as Resolved.May 10 2018, 6:05 AM
ArielGlenn added a comment.May 10 2018, 6:08 AM

The last host to upgrade, snapshot1007, can be done when the current dump run completes, or at least the jobs on that host complete. The misc cron jobs host, see T190112, is due to arrive today and with any luck will be available for install early next week, at which point cron jobs could be moved there.

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Active on the Dumps-Generation board.May 14 2018, 8:41 AM
ArielGlenn added a comment.May 14 2018, 8:44 AM

Currently waiting for the following dumps to complete: commonswiki, dewiki, frwiki, ruwiki, and the dreaded mediawikiwiki flow history dumps. I estimate three days for all of these.

ops-monitoring-bot added a comment.May 20 2018, 7:35 AM

Script wmf-auto-reimage was launched by ariel on neodymium.eqiad.wmnet for hosts:

snapshot1007.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201805200735_ariel_4481_snapshot1007_eqiad_wmnet.log.

gerritbot added a comment.May 20 2018, 8:06 AM

Change 434141 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] use php7.0 for dumps and related jobs on snapshot1007 now

https://gerrit.wikimedia.org/r/434141

gerritbot added a comment.May 20 2018, 8:07 AM

Change 434141 merged by ArielGlenn:
[operations/puppet@production] use php7.0 for dumps and related jobs on snapshot1007 now

https://gerrit.wikimedia.org/r/434141

ops-monitoring-bot added a comment.May 20 2018, 8:14 AM

Completed auto-reimage of hosts:

['snapshot1007.eqiad.wmnet']

and were ALL successful.

ArielGlenn closed this task as Resolved.May 20 2018, 9:15 AM

This task is finally complete!

ArielGlenn moved this task from Active to Done on the Dumps-Generation board.May 20 2018, 9:16 AM
• Aklapper removed subscribers: Anomie, • ksmith.Oct 16 2020, 5:42 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 16 2020, 6:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits