Page MenuHomePhabricator
Create Task
Maniphest T144006

Move the MW Beta appservers to Debian
Closed, ResolvedPublic
Actions

Description

Macro list of clusters:

  • Deployment servers - tin / mira
  • Appservers - deployment-mediawiki 01 02 03
  • Jobrunners - deployment-jobrunner01.deployment-prep.eqiad.wmflabs
  • Videoscalers (Got removed, never setup in deployment-prep)

The hostnames can be checked in hieradata/labs/deployment-prep/common.yaml

scap::dsh::groups:
    mediawiki-installation:
        hosts:
            - deployment-jobrunner01.deployment-prep.eqiad.wmflabs
            - deployment-mediawiki01.deployment-prep.eqiad.wmflabs
            - deployment-mediawiki02.deployment-prep.eqiad.wmflabs
            - deployment-mediawiki03.deployment-prep.eqiad.wmflabs
            - deployment-tmh01.deployment-prep.eqiad.wmflabs
            - deployment-tin.deployment-prep.eqiad.wmflabs
            - mira.deployment-prep.eqiad.wmflabs

Deployment steps

Jenkins runs scap as user jenkins-deploy on deployment-tin. You will need to manually accept the new host ssh fingerprint by running:

sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs

Manually refresh all CA certificates symlinks due to T145609:

update-ca-certificates --verbose --fresh
  • For MediaWiki web servers **

Once the puppet patch(es) are merged, the beta cluster puppetmaster needs a rebase:

ssh deployment-puppetmaster.deployment-prep.eqiad.wmflabs
sudo su -
cd /var/lib/git/operations/puppet
git pull

Then get the host pooled on the Varnish cache by running puppet on deployment-cache-text04.deployment-prep.eqiad.wmflabs. That will update the Varnish list of directors. Then: service varnish reload.

Test:

curl --silent https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version?date=`date +%s`|grep deployment

And alternatively look at https://logstash-beta.wmflabs.org/

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
hashar added a comment.Sep 12 2016, 1:56 PM

We had two 8 CPU / 16 G instances created to migrate the databases to Jessie T138778 that is scheduled for Thursday. Once migrated I guess they will be deleted and free up 16 CPU / 32 G of RAM :]

There is also a task to purge/consolidate instances on beta T142288 which is ongoing. Will free up even more of the quota.

Then I guess we can ask to lower the quota to keep the project under control.

gerritbot subscribed.Sep 12 2016, 2:18 PM

Change 309999 had a related patch set uploaded (by Elukey):
Add mediawiki04 to the list of labs appservers in deployment-prep

https://gerrit.wikimedia.org/r/309999

gerritbot added a project: Patch-For-Review.Sep 12 2016, 2:18 PM
Stashbot subscribed.Sep 12 2016, 2:41 PM

Mentioned in SAL [2016-09-12T14:41:35Z] <elukey> applied base::firewall, beta::deployaccess, mediawiki::conftool, role::mediawiki::appserver to deployment-mediawiki04.deployment-prep.eqiad.wmflabs (Debian jessie instance) - T144006

gerritbot added a comment.Sep 12 2016, 4:20 PM

Change 309999 abandoned by Elukey:
Add mediawiki04 to the list of labs appservers in deployment-prep

Reason:
Will split into two CRs

https://gerrit.wikimedia.org/r/309999

gerritbot added a comment.Sep 12 2016, 4:23 PM

Change 310034 had a related patch set uploaded (by Elukey):
Add deployment-mediawiki04 to the deployment-prep scap dsh

https://gerrit.wikimedia.org/r/310034

gerritbot added a comment.Sep 12 2016, 4:23 PM

Change 310035 had a related patch set uploaded (by Elukey):
Add deployment-mediawiki04 to the deployment-prep Varnish config

https://gerrit.wikimedia.org/r/310035

hashar added a comment.Sep 12 2016, 4:37 PM

https://wikitech.wikimedia.org/wiki/HHVM/Troubleshooting has some interesting bits

furl http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page

Does fcgi requests on localhost \O/ bypassing apache

gerritbot added a comment.Sep 13 2016, 8:12 AM

Change 310034 merged by Elukey:
Add deployment-mediawiki04 to the deployment-prep scap dsh

https://gerrit.wikimedia.org/r/310034

hashar updated the task description. (Show Details)Sep 13 2016, 9:05 AM
gerritbot added a comment.Sep 13 2016, 9:10 AM

Change 310035 merged by Elukey:
Add deployment-mediawiki04 to the deployment-prep Varnish config

https://gerrit.wikimedia.org/r/310035

hashar updated the task description. (Show Details)Sep 13 2016, 9:16 AM
hashar added a comment.Sep 13 2016, 9:22 AM

After some mess with scap mwdeploy keys solved by running on deployment-tin:

sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs

deployment-mediawiki04 is in the pool:

$ curl --silent https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version|grep deployment
... mw.config.set({"wgBackendResponseTime":674,"wgHostname":"deployment-mediawiki04"}) ...
                                                                        ^^^^^^^^^^^
hashar updated the task description. (Show Details)Sep 13 2016, 9:25 AM

Noticed in logstash:

Warning: failed to mkdir "/srv/mediawiki/php-master/images/thumb/2/20/Order_of_St_John_(UK)_ribbon.png"
mode 0777 [Called from wfMkdirParents in /srv/mediawiki/php-master/includes/Glob
hashar added a comment.Sep 13 2016, 9:34 AM
In T144006#2632108, @hashar wrote:

Noticed in logstash:

Warning: failed to mkdir "/srv/mediawiki/php-master/images/thumb/2/20/Order_of_St_John_(UK)_ribbon.png"
mode 0777 [Called from wfMkdirParents in /srv/mediawiki/php-master/includes/Glob

That is unrelated to the Jessie reimaging. Filled T145496 about it.

gerritbot added a comment.Sep 13 2016, 9:34 AM

Change 310256 had a related patch set uploaded (by Elukey):
Replace mediawiki01 with mediawiki04 (Debian Jessie) in deployment-prep

https://gerrit.wikimedia.org/r/310256

gerritbot added a comment.Sep 13 2016, 9:37 AM

Change 310256 merged by Elukey:
Replace mediawiki01 with mediawiki04 (Debian Jessie) in deployment-prep

https://gerrit.wikimedia.org/r/310256

gerritbot added a comment.Sep 13 2016, 10:12 AM

Change 310264 had a related patch set uploaded (by Hashar):
beta: change canary mw server from 01 to 04

https://gerrit.wikimedia.org/r/310264

gerritbot added a comment.Sep 13 2016, 10:14 AM

Change 310264 merged by Elukey:
beta: change canary mw server from 01 to 04

https://gerrit.wikimedia.org/r/310264

hashar mentioned this in T145611: Please raise quota for deployment-prep.Sep 14 2016, 8:31 AM
hashar added a subtask: T145609: Puppet sslcert::ca does not refresh the certificate symlinks when a .crt is updated.Sep 14 2016, 9:25 AM
hashar updated the task description. (Show Details)
Stashbot added a comment.Sep 14 2016, 9:27 AM

Mentioned in SAL (#wikimedia-releng) [2016-09-14T09:27:50Z] <hashar> Deleting deployment-mediawiki01 , replaced by deployment-mediawiki04 T144006

elukey added a subtask: T145611: Please raise quota for deployment-prep.Sep 15 2016, 8:23 AM

The task is currently blocked by T145611

elukey claimed this task.Sep 15 2016, 8:23 AM
gerritbot added a comment.Sep 15 2016, 8:38 AM

Change 310749 had a related patch set uploaded (by Elukey):
Remove mediawiki03 from deployment-prep

https://gerrit.wikimedia.org/r/310749

gerritbot added a comment.Sep 15 2016, 8:44 AM

Change 310749 merged by Elukey:
Remove mediawiki03 from deployment-prep

https://gerrit.wikimedia.org/r/310749

gerritbot added a comment.Sep 15 2016, 8:59 AM

Change 310756 had a related patch set uploaded (by Elukey):
Add mediawiki06 to the deployment-prep scap dsh

https://gerrit.wikimedia.org/r/310756

gerritbot added a comment.Sep 15 2016, 9:02 AM

Change 310756 merged by Elukey:
Add mediawiki06 to the deployment-prep scap dsh

https://gerrit.wikimedia.org/r/310756

Stashbot added a comment.Sep 15 2016, 9:33 AM

Mentioned in SAL (#wikimedia-releng) [2016-09-15T09:33:36Z] <hashar> T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki06.deployment-prep.eqiad.wmflabs

gerritbot added a comment.Sep 15 2016, 10:11 AM

Change 310773 had a related patch set uploaded (by Elukey):
Set mediawiki06 (replacement of mediawiki03) as security audit target

https://gerrit.wikimedia.org/r/310773

gerritbot added a comment.Sep 15 2016, 10:16 AM

Change 310773 merged by Elukey:
Set mediawiki06 (replacement of mediawiki03) as security audit target

https://gerrit.wikimedia.org/r/310773

gerritbot added a comment.Sep 15 2016, 10:31 AM

Change 310796 had a related patch set uploaded (by Elukey):
Remove mediawiki02 from deployment prep

https://gerrit.wikimedia.org/r/310796

gerritbot added a comment.Sep 15 2016, 11:43 AM

Change 310796 merged by Elukey:
Remove mediawiki02 from deployment prep

https://gerrit.wikimedia.org/r/310796

gerritbot added a comment.Sep 15 2016, 1:19 PM

Change 310818 had a related patch set uploaded (by Elukey):
Add mediawiki05 to deployment-prep

https://gerrit.wikimedia.org/r/310818

gerritbot added a comment.Sep 15 2016, 1:33 PM

Change 310818 merged by Elukey:
Add mediawiki05 to deployment-prep

https://gerrit.wikimedia.org/r/310818

elukey updated the task description. (Show Details)Sep 15 2016, 1:36 PM
Stashbot added a comment.Sep 15 2016, 2:44 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-15T14:44:08Z] <hashar> T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki05.deployment-prep.eqiad.wmflabs

Stashbot added a comment.Sep 15 2016, 2:45 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-15T14:45:00Z] <hashar> T144006 sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@mira02.deployment-prep.eqiad.wmflabs

Stashbot added a comment.Sep 15 2016, 3:05 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-15T15:05:04Z] <hashar> T144006 Applying class role::labs::lvm::srv to mira02 (it is out of disk space :D )

Stashbot added a comment.Sep 15 2016, 3:08 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-15T15:08:21Z] <hashar> T144006 Disabled Jenkins job beta-scap-eqiad. On mira02 rm -fR /srv/* . Applying puppet for role::labs::lvm::srv

zeljkofilipin subscribed.Sep 16 2016, 12:02 PM
• chasemp closed subtask T145611: Please raise quota for deployment-prep as Resolved.Sep 19 2016, 7:39 PM
elukey added a comment.Sep 20 2016, 9:26 AM

As @AlexMonk-WMF reported, we caused an issue when dealing with restbase configs: https://phabricator.wikimedia.org/T146053

As follow up would it be worth to write a page for Deployment-prep best practices? Ops maintains https://wikitech.wikimedia.org/wiki/Service_restarts, so we could either add a section in there or create a different page and link it in there. It would be great to know:

  1. general tribal knowledge about how things are handled in there (self-hosted puppet master, auto-rebase cron, etc..)
  2. impact of an outage - who are the users? Who should be notified about these maintenance work?
gerritbot added a comment.Sep 20 2016, 11:12 AM

Change 311681 had a related patch set uploaded (by Elukey):
Add jobrunner02 to deployment-prep

https://gerrit.wikimedia.org/r/311681

gerritbot added a comment.Sep 20 2016, 11:16 AM

Change 311681 merged by Elukey:
Add jobrunner02 to deployment-prep

https://gerrit.wikimedia.org/r/311681

• AlexMonk-WMF added a comment.Sep 20 2016, 12:00 PM
In T144006#2651679, @elukey wrote:

As @AlexMonk-WMF reported, we caused an issue when dealing with restbase configs: https://phabricator.wikimedia.org/T146053

As follow up would it be worth to write a page for Deployment-prep best practices?

It's not a deployment-prep specific thing, the same could happen in production.

elukey added a comment.Sep 20 2016, 12:03 PM
In T144006#2651889, @AlexMonk-WMF wrote:
In T144006#2651679, @elukey wrote:

As @AlexMonk-WMF reported, we caused an issue when dealing with restbase configs: https://phabricator.wikimedia.org/T146053

As follow up would it be worth to write a page for Deployment-prep best practices?

It's not a deployment-prep specific thing, the same could happen in production.

Yes ok you are right, I'll update also prod documentation, but this does not imply that deployment-prep shouldn't have its own documentation :)

gerritbot added a comment.Sep 20 2016, 2:40 PM

Change 311710 had a related patch set uploaded (by Muehlenhoff):
mira02 "reimaged" as deployment-mira02

https://gerrit.wikimedia.org/r/311710

gerritbot added a comment.Sep 20 2016, 2:43 PM

Change 311710 merged by Muehlenhoff:
mira02 "reimaged" as deployment-mira02

https://gerrit.wikimedia.org/r/311710

gerritbot added a comment.Sep 20 2016, 2:51 PM

Change 311717 had a related patch set uploaded (by Elukey):
Remove jobrunner01 from deployment-prep

https://gerrit.wikimedia.org/r/311717

Stashbot added a comment.Sep 20 2016, 6:33 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-20T18:33:31Z] <hashar> on deployment-mira02 ran sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mediawiki04.deployment-prep.eqiad.wmflabs per T144006

Stashbot added a comment.Sep 20 2016, 6:38 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-20T18:38:10Z] <hashar> on tin: sudo -u jenkins-deploy -H SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh mwdeploy@deployment-mira02.deployment-prep.eqiad.wmflabs - T144006

gerritbot added a comment.Sep 21 2016, 8:09 AM

Change 311717 merged by Elukey:
Remove jobrunner01 from deployment-prep

https://gerrit.wikimedia.org/r/311717

elukey updated the task description. (Show Details)Sep 21 2016, 8:11 AM
hashar updated the task description. (Show Details)Sep 26 2016, 10:36 AM
gerritbot added a comment.Sep 26 2016, 10:37 AM

Change 312654 had a related patch set uploaded (by Hashar):
beta: drop deployment-tin add deployment-tin02

https://gerrit.wikimedia.org/r/312654

hashar added a comment.Sep 26 2016, 10:38 AM

The deployment servers have been reimaged to Jessie:

  • deployment-mira
  • deployment-tin02

Last patch to land is https://gerrit.wikimedia.org/r/#/c/312654/

Stashbot added a comment.Sep 28 2016, 11:48 AM

Mentioned in SAL (#wikimedia-releng) [2016-09-28T11:48:28Z] <hashar> Deleting deployment-tin Trusty instance and recreate one with same hostname as Jessie; Meant to replace deployment-tin02 T144006

Stashbot added a comment.Sep 28 2016, 7:49 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-28T19:49:30Z] <hasharAway> Dropping deployment-tin02 , replacing it with deployment-tin which has been rebuild to Jessie T144006

hashar added a comment.Sep 28 2016, 8:47 PM

I have dropped deployment-tin02 it was confusing people and create a deployment-tin which is now the master https://gerrit.wikimedia.org/r/#/c/312654/

gerritbot added a comment.Oct 4 2016, 2:38 PM

Change 312654 merged by Elukey:
beta: update deployment-tin IP and make it master

https://gerrit.wikimedia.org/r/312654

hashar added a comment.Oct 4 2016, 5:34 PM

The deployment servers on beta cluster are now fully migrated to Jessie. We ended up keeping the same hostname and have:

  • deployment-tin.eqiad.wmflabs (primary, Jenkins slave)
  • deployment-mira.eqiad.wmflabs (secondary)

What is left is deployment-tmh01 which needs some packaging work for Jessie as I understood it.

fgiunchedi closed subtask T145609: Puppet sslcert::ca does not refresh the certificate symlinks when a .crt is updated as Resolved.Nov 29 2016, 11:56 PM
elukey removed elukey as the assignee of this task.Dec 14 2016, 11:05 AM
elukey added a project: User-Elukey.
elukey moved this task from Backlog to Ops Backlog on the User-Elukey board.Dec 14 2016, 11:08 AM
elukey moved this task from Ops Backlog to Stalled on the User-Elukey board.Dec 14 2016, 5:42 PM
elukey removed projects: User-Elukey, Patch-For-Review.May 2 2017, 4:25 PM
greg added a comment.Jul 7 2017, 11:39 PM
In T144006#2689359, @hashar wrote:

What is left is deployment-tmh01 which needs some packaging work for Jessie as I understood it.

That was Oct 2016 :)

This still is accurate, at least wrt tmh:

greg@x230  ~ % ssh deployment-tmh01.eqiad.wmflabs
Linux deployment-tmh01 3.13.0-121-generic #170-Ubuntu SMP Wed Jun 14 09:04:33 UTC 2017 x86_64
Ubuntu 14.04.5 LTS
deployment-tmh01 is mediawiki::videoscaler

Looks like there is recent activity on T145742, yay!

greg added a subtask: T145742: Migrate video scalers to jessie.Jul 7 2017, 11:40 PM
MoritzMuehlenhoff closed subtask T145742: Migrate video scalers to jessie as Resolved.Aug 29 2017, 11:44 AM
MoritzMuehlenhoff added a comment.Aug 29 2017, 11:47 AM

All the packaging work for jessie is complete (and the servers in production have been migrated). If deployment-tmh01 is still used it can be reimaged as well.

Liuxinyu970226 subscribed.Aug 29 2017, 1:04 PM
greg mentioned this in T174477: Reimage deployment-tmh01 with Debian Jessie.Aug 29 2017, 6:41 PM
greg created subtask T174477: Reimage deployment-tmh01 with Debian Jessie.
Jdforrester-WMF updated the task description. (Show Details)Sep 7 2017, 3:38 PM
EddieGP closed subtask T174477: Reimage deployment-tmh01 with Debian Jessie as Declined.Apr 4 2018, 10:16 PM
EddieGP added subtasks: T179371: Move deployment-prep redis instances to stretch, T184244: Puppet broken on deployment-mx due to systemd on trusty.Apr 4 2018, 10:25 PM
EddieGP subscribed.EditedApr 4 2018, 10:29 PM

There's 4 trusty instances left in deployment-prep:

  • deployment-tmh01 is to be deleted per T174477/T191293 (done)
  • deployment-redis0[12] are to be replaced by deployment-redis0[56] (stretch) per T179371
  • deployment-mx is to be replaced by deployment-mx02 (stretch) per T184244.
MoritzMuehlenhoff updated the task description. (Show Details)Apr 5 2018, 7:57 AM
EddieGP added a subscriber: akosiaris.Apr 6 2018, 7:51 PM

Also (labeled at "UNKNOWN" in openstack browser, but logging in there and looking at /etc/os-release) these are still trusty:

  • deployment-urldownloader
  • deployment-zotero01

Both were created by @akosiaris .

Joe subscribed.Apr 9 2018, 5:33 AM

I think this task is resolved as it's about the MediaWiki appservers and AFAICS they're all converted to jessie at least.

Joe closed this task as Resolved.Apr 9 2018, 5:34 AM
Liuxinyu970226 unsubscribed.Apr 9 2018, 10:04 AM
Krenair closed subtask T184244: Puppet broken on deployment-mx due to systemd on trusty as Resolved.Jul 11 2018, 8:49 PM
Krenair closed subtask T179371: Move deployment-prep redis instances to stretch as Resolved.Jul 13 2018, 8:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL