Page MenuHomePhabricator
Create Task
Maniphest T175288

setup/install/deploy deploy1001 as deployment server
Closed, ResolvedPublic
Actions

Description

This task will track the setup and deployment of system WMF4748 as deploy1001.eqiad.wmnet, for use as a deployment server in eqiad. Tin is out of warranty, and failing hardware has called for its replacement on parent task T174452.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -7mw-deployment: remove rsync for tin home dirs
operations/puppetproduction+9 -6decom and remove remnants of tin.eqiad.wmnet
operations/puppetproduction+8 -0deployment::server: add rsync for home dirs
operations/dnsmaster+2 -2switch deployment.[eqiad|codfw] from tin to deploy1001
operations/puppetproduction+3 -4install/cumin/scap: update/remove tin-related comments
operations/puppetproduction+0 -1remove tin from hosts kubernetes master is accessible to
operations/puppetproduction+1 -2scap: rm tin from masters,hosts, add deploy1001 to hosts
operations/puppetproduction+2 -2switch deployment_server from tin to deploy1001
operations/puppetproduction+3 -3scap: switch scap::deployment server to deploy1001
operations/puppetproduction+1 -0scap/dsh: add deploy1001 to scap masters
operations/puppetproduction+4 -0mariadb: grant deploy1001 access to labswiki
operations/puppetproduction+0 -2deploy1001: reinstall with stretch instead of jessie
operations/puppetproduction+2 -1mediawiki::deployment: don't use mysql module, fix stretch suppport
operations/puppetproduction+2 -0install_server: set deploy1001 to use jessie
operations/dnsmaster+2 -2Change cname for deployment.eqiad.wmnet and deployment.codfw.wmnet
operations/puppetproduction+5 -6switch deployment server from tin to deploy1001
operations/puppetproduction+2 -0add deploy1001 to scap masters and hosts in Hiera
operations/puppetproduction+8 -0deployment_server: allow rsyncing of /srv/ to new server
operations/puppetproduction+9 -1Restrict inclusion of full PHP packages on deployment servers to jessie
operations/puppetproduction+42 -1mediawik_deployment: stretch support for PHP packages
operations/puppetproduction+5 -2mediawiki::deployment: stretch support for php-readline package
operations/puppetproduction+1 -7add deployment_server role to deploy1001
operations/puppetproduction+2 -0network::constants: add deploy1001 as deployment server
operations/puppetproduction+3 -0tcpircbot: add deploy1001 as allowed host
operations/puppetproduction+1 -1netboot: temp remove deploy hosts for debugging
operations/puppetproduction+2 -0deploy1001: test with jessie installer
operations/puppetproduction+2 -0deploy1001: switch to use stretch installer
operations/puppetproduction+14 -5set deploy1001 isntall params
operations/dnsmaster+4 -0deploy1001 dns entries
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mar 27 2018, 10:27 PM

Change 422349 merged by Dzahn:
[operations/dns@master] Change cname for deployment.eqiad.wmnet and deployment.codfw.wmnet

https://gerrit.wikimedia.org/r/422349

Stashbot added a comment.Mar 27 2018, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-27T22:28:21Z] <mutante> DNS - switching deployment service name to deploy1001 (T175288)

Dzahn closed this task as Resolved.Mar 27 2018, 10:48 PM

There has been a deploy from it, and all the changes above, incl. DNS service name, Mukunda confirmed things looking good, announced to ops list, created wiki page on wikitech and pasted fingerprints.. It's up and running.

Dzahn updated the task description. (Show Details)Mar 27 2018, 10:48 PM
Dzahn added a subscriber: 20after4.
Dzahn mentioned this in T185275: replace tin (new hardware).Mar 27 2018, 10:52 PM
gerritbot added a comment.Mar 28 2018, 7:44 PM

Change 422479 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: set deploy1001 to use jessie

https://gerrit.wikimedia.org/r/422479

gerritbot added a comment.Mar 28 2018, 7:51 PM

Change 422479 merged by Dzahn:
[operations/puppet@production] install_server: set deploy1001 to use jessie

https://gerrit.wikimedia.org/r/422479

Stashbot added a comment.Mar 28 2018, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-28T19:54:29Z] <mutante> deploy1001 - schedule downtime for reinstall with jessie, reinstalling (T175288)

Dzahn reopened this task as Open.Mar 28 2018, 10:55 PM
Dzahn added a comment.Apr 10 2018, 6:15 PM

per the last ops meeting and joe's comments:

  • reinstall it one more time. back to stretch instead of jessie
gerritbot added a comment.Apr 10 2018, 6:17 PM

Change 425331 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deploy1001: reinstall with stretch instead of jessie

https://gerrit.wikimedia.org/r/425331

gerritbot added a comment.Apr 10 2018, 6:48 PM

Change 425331 merged by Dzahn:
[operations/puppet@production] deploy1001: reinstall with stretch instead of jessie

https://gerrit.wikimedia.org/r/425331

Stashbot added a comment.Apr 10 2018, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-10T20:30:47Z] <mutante> deploy1001 - reinstalled with jessie - re-adding to puppet (T175288)

Stashbot added a comment.Apr 10 2018, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-10T20:30:58Z] <mutante> deploy1001 - reinstalled with stretch - re-adding to puppet (T175288)

• mobrovac added a subscriber: • mobrovac.Apr 11 2018, 12:56 PM

It seems that the reimage is now blocking deployments, cf. T191972: Scap sync-file failing for deploy1001.eqiad.wmnet

Stashbot added a comment.Apr 11 2018, 6:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-11T18:11:47Z] <mutante> deploy1001 is back on stretch once again - it has been removed from scap hosts though (T175288 T185275)

Stashbot added a comment.Apr 19 2018, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-19T08:14:22Z] <ema> reboot deploy1001 and arm keyholder T175288

thcipriani mentioned this in T192561: Upgrade deployment-prep deployment servers to stretch.Apr 19 2018, 3:15 PM
Dzahn mentioned this in T174431: Upgrade mw* servers to Debian Stretch (using HHVM).May 4 2018, 11:29 PM
TerraCodes added a subscriber: TerraCodes.May 6 2018, 6:12 PM

Can this task be closed? (since everything in the task description is checked off)

Dzahn added a comment.May 7 2018, 5:21 AM

No, it can't be closed since it's not done and tin is still the deployment server. Issue is that checkbox(es) missing.

MoritzMuehlenhoff added a comment.May 14 2018, 11:04 AM

I suggest we do the following:

  • Pick a date/time frame of a few hours where no deployments are happening (or cancel existing ones)
  • We switch the deployment server to deploy1001 and test deployments using the PHP7 setup present there
  • If anything breaks, we revert to tin and fix whatever problem we found with PHP 7 and deploy1001 and re-attempt at a later stage
  • If everything works fine, we keep tin for a few weeks as a fallback and then decom it.
MoritzMuehlenhoff mentioned this in T191921: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5).May 14 2018, 11:06 AM
Joe added a subscriber: Joe.May 14 2018, 4:08 PM
In T175288#4204232, @MoritzMuehlenhoff wrote:

I suggest we do the following:

  • Pick a date/time frame of a few hours where no deployments are happening (or cancel existing ones)
  • We switch the deployment server to deploy1001 and test deployments using the PHP7 setup present there
  • If anything breaks, we revert to tin and fix whatever problem we found with PHP 7 and deploy1001 and re-attempt at a later stage
  • If everything works fine, we keep tin for a few weeks as a fallback and then decom it.

I would propose that if there are issues with PHP7 we just use HHVM on deploy1001 instead.

And, let's do this on friday, that leaves us until monday's SWAT (if any).

Dzahn added a comment.May 14 2018, 5:19 PM

The plan sounds good.

How do you guys feel about the maintenance server, terbium -> mwmaint1001 (T192092), should we do that first or after tin/deploy1001?

MoritzMuehlenhoff added a comment.May 14 2018, 5:28 PM
In T175288#4205272, @Dzahn wrote:

How do you guys feel about the maintenance server, terbium -> mwmaint1001 (T192092), should we do that first or after tin/deploy1001?

We should do tin first, replacing terbium is more complex.

MoritzMuehlenhoff added a comment.May 14 2018, 5:30 PM

And, let's do this on friday, that leaves us until monday's SWAT (if any).

How about Friday, 25th of May?

Dzahn added a comment.May 14 2018, 5:34 PM

@20after4 @greg We are trying to find a time to replace tin with deploy1001 (this time for real) and are thinking of Friday May 25th. Any thoughts on that? Maybe it would be interesting who is going to be the first deployer (mw and other) after that switch.

mmodell added a subscriber: mmodell.May 14 2018, 7:19 PM

@MoritzMuehlenhoff @Dzahn: The 25th should be fine.

Reedy removed a parent task: T174452: eqiad: replacement tin/deployment server.May 20 2018, 11:58 AM
Reedy added a subtask: T174452: eqiad: replacement tin/deployment server.
ArielGlenn mentioned this in T172165: Require either PHP 7.0+ or HHVM in MW 1.31.May 22 2018, 9:47 AM
Reedy mentioned this in T195393: Run all maintenance scripts on PHP7 or HHVM.May 23 2018, 6:09 PM
Dzahn updated the task description. (Show Details)May 23 2018, 10:50 PM
gerritbot added a comment.May 23 2018, 10:50 PM

Change 434821 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb: grant deploy1001 access to labswiki

https://gerrit.wikimedia.org/r/434821

Dzahn added a comment.May 23 2018, 10:57 PM

We'll need the database grants above added or that might block the switch. That being said, not sure why exactly the deployment server needs to talk to the 'labswiki" (wikitech) database. What would break without that?

gerritbot added a comment.May 24 2018, 2:41 PM

Change 434821 merged by Dzahn:
[operations/puppet@production] mariadb: grant deploy1001 access to labswiki

https://gerrit.wikimedia.org/r/434821

Dzahn closed subtask T193916: rename naos to deploy2001 and reinstall with stretch as Resolved.May 24 2018, 3:01 PM
Dzahn added a comment.May 25 2018, 5:41 PM
In T175288#4227190, @Dzahn wrote:

We'll need the database grants above added or that might block the switch

This is unblocked. All 3 DB grant related changes have been merged/deployed/tested. (thanks Manuel)

In T175288#4205550, @mmodell wrote:

@MoritzMuehlenhoff @Dzahn: The 25th should be fine.

We canceled today's planned migration. We are waiting because we need to be able to deploy at any time while T195520 is being resolved.

We will schedule a new date after that.

Krenair mentioned this in T195686: Move puppetmaster to Stretch.May 27 2018, 7:53 PM
Dzahn added a comment.May 30 2018, 6:57 PM

The new planned window for this migration is the upcoming Friday, June 1st. (with thcipriani hoping he gets the train back on track, so there won't be too many deploys)

Dzahn added a comment.May 30 2018, 7:03 PM

deployment-prep now also has a new instance using stretch with more disk space to match this (T192561#4243810)

Krinkle mentioned this in T176370: Migrate to PHP 7 in WMF production.Jun 1 2018, 1:56 AM
gerritbot added a comment.Jun 1 2018, 2:49 PM

Change 436814 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap/dsh: add deploy1001 to scap masters

https://gerrit.wikimedia.org/r/436814

gerritbot added a comment.Jun 1 2018, 2:50 PM

Change 436814 merged by Dzahn:
[operations/puppet@production] scap/dsh: add deploy1001 to scap masters

https://gerrit.wikimedia.org/r/436814

gerritbot added a comment.Jun 1 2018, 3:18 PM

Change 436816 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: switch scap::deployment server to deploy1001

https://gerrit.wikimedia.org/r/436816

Dzahn added a comment.Jun 1 2018, 3:20 PM
--- /etc/dsh/group/scap-masters	2018-05-24 14:25:47.608760286 +0000
+deploy1001.eqiad.wmnet

..

[deploy1001:~] $ scap pull
14:52:35 Copying to deploy1001.eqiad.wmnet from tin.eqiad.wmnet
14:52:35 Started rsync common

[deploy1001:~] $ scap pull-master tin.eqiad.wmnet
15:17:08 Copying to deploy1001.eqiad.wmnet from tin.eqiad.wmnet
15:17:08 Started rsync master
..
..

10:52 < mutante> !log deploy1001 - scap pull
11:05 < mutante> !log rsyncing /srv/mediawiki-staging to /srv/mediawiki-staging-before-backup/ on tin as a backup
11:12 < mutante> !log tin umask 022 && echo 'switching deploy servers' > /var/lock/scap-global-lock
11:17 < mutante> !log [deploy1001:~] $ scap pull-master tin.eqiad.wmnet
gerritbot added a comment.Jun 1 2018, 3:25 PM

Change 436818 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] switch deployment_server from tin to deploy1001

https://gerrit.wikimedia.org/r/436818

Dzahn added a comment.Jun 1 2018, 3:51 PM

11:41 < mutante> !log root@deploy1001:/srv/mediawiki-staging# find . -uid 996 -exec chown mwdeploy {} \;

11:47 < mutante> !log @deploy1001:/srv/deployment# find . -uid 997 -exec chown trebuchet {} \;

gerritbot added a comment.Jun 1 2018, 3:53 PM

Change 436816 merged by Dzahn:
[operations/puppet@production] scap: switch scap::deployment server to deploy1001

https://gerrit.wikimedia.org/r/436816

gerritbot added a comment.Jun 1 2018, 4:18 PM

Change 436818 merged by Dzahn:
[operations/puppet@production] switch deployment_server from tin to deploy1001

https://gerrit.wikimedia.org/r/436818

Stashbot added a comment.Jun 1 2018, 4:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-01T16:21:53Z] <mutante> deployment server has switched away from tin to deploy1001. set global scap lock on deploy1001, re-enabled puppet and ran puppet, disabled tin as deployment server (T175288)

Dzahn removed a project: Patch-For-Review.Jun 1 2018, 4:32 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jun 1 2018, 4:45 PM

Change 436827 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: remove tin from scap masters and hosts

https://gerrit.wikimedia.org/r/436827

gerritbot added a project: Patch-For-Review.Jun 1 2018, 4:45 PM
gerritbot added a comment.Jun 1 2018, 4:49 PM

Change 436827 merged by Dzahn:
[operations/puppet@production] scap: rm tin from masters,hosts, add deploy1001 to hosts

https://gerrit.wikimedia.org/r/436827

gerritbot added a comment.Jun 1 2018, 4:52 PM

Change 436830 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove tin from hosts kubernetes master is accessible to

https://gerrit.wikimedia.org/r/436830

gerritbot added a comment.Jun 1 2018, 4:54 PM

Change 436830 merged by Dzahn:
[operations/puppet@production] remove tin from hosts kubernetes master is accessible to

https://gerrit.wikimedia.org/r/436830

gerritbot added a comment.Jun 1 2018, 4:58 PM

Change 436831 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install/cumin/scap: update/remove tin-related comments

https://gerrit.wikimedia.org/r/436831

gerritbot added a comment.Jun 1 2018, 5:04 PM

Change 436831 merged by Dzahn:
[operations/puppet@production] install/cumin/scap: update/remove tin-related comments

https://gerrit.wikimedia.org/r/436831

Dzahn mentioned this in T196175: decom/reclaim tin.Jun 1 2018, 5:07 PM
Dzahn added a subtask: T196175: decom/reclaim tin.
Dzahn changed the status of subtask T196175: decom/reclaim tin from Open to Stalled.Jun 1 2018, 5:11 PM
Dzahn closed this task as Resolved.Jun 1 2018, 5:15 PM

deploy1001 is now the active deployment server. from here it should just be about removing tin. we will wait a grace period and then this continues on T185275 and finally T196175

gerritbot added a comment.Jun 1 2018, 5:26 PM

Change 436835 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] switch deployment.eqiad from tin to deploy1001

https://gerrit.wikimedia.org/r/436835

gerritbot added a comment.Jun 1 2018, 5:30 PM

Change 436835 merged by Dzahn:
[operations/dns@master] switch deployment.[eqiad|codfw] from tin to deploy1001

https://gerrit.wikimedia.org/r/436835

Jdforrester-WMF awarded a token.Jun 1 2018, 6:03 PM
Dzahn added a subtask: T185275: replace tin (new hardware).Jun 2 2018, 3:09 PM
gerritbot added a comment.Jun 2 2018, 3:10 PM

Change 436992 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] deployment::server: add rsync for home dirs

https://gerrit.wikimedia.org/r/436992

gerritbot added a comment.Jun 2 2018, 3:27 PM

Change 436992 merged by Dzahn:
[operations/puppet@production] deployment::server: add rsync for home dirs

https://gerrit.wikimedia.org/r/436992

gerritbot added a comment.Jun 7 2018, 1:53 PM

Change 420917 merged by Dzahn:
[operations/puppet@production] decom and remove remnants of tin.eqiad.wmnet

https://gerrit.wikimedia.org/r/420917

Dzahn closed subtask T185275: replace tin (new hardware) as Resolved.Jun 7 2018, 2:09 PM
Dzahn changed the status of subtask T196175: decom/reclaim tin from Stalled to Open.
Stashbot mentioned this in T196019: setup/install phab1002(WMF4727).Jun 11 2018, 10:52 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-11T10:52:48Z] <mutante> phab1002 - editing cached scap config /srv/deployment/phabricator/deployment-cache/.config to replace tin.eqiad with deploy1001.eqiad deployment server, run puppet. other options: run scap with --refresh-config, delet cached .config file (T196019) (T175288)

gerritbot added a comment.Jun 13 2018, 11:52 AM

Change 440100 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mw-deployment: remove rsync for tin home dirs

https://gerrit.wikimedia.org/r/440100

gerritbot added a comment.Jun 13 2018, 2:33 PM

Change 440100 merged by Dzahn:
[operations/puppet@production] mw-deployment: remove rsync for tin home dirs

https://gerrit.wikimedia.org/r/440100

Dzahn mentioned this in T197470: find a way to systematically update the deployment server name across all repos.Jun 15 2018, 2:48 PM
• Vvjjkkii reopened subtask T193916: rename naos to deploy2001 and reinstall with stretch as Open.Jul 1 2018, 1:11 AM
CommunityTechBot closed subtask T193916: rename naos to deploy2001 and reinstall with stretch as Resolved.Jul 2 2018, 1:45 PM
Cmjohnson closed subtask T196175: decom/reclaim tin as Resolved.Aug 7 2018, 9:09 PM
Stashbot mentioned this in T224247: upgrade and rename krypton & create its codfw equivalent.Aug 27 2019, 11:51 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T11:51:31Z] <mutante> miscweb1001 - manually remove tin.eqiad.wmnet (!) from /srv/iegreview/iegreview-cache/.config and replace with deploy1001 after first puppet run. still existing bug that tin is not fully removed (T224247, T175288, T197470)

RobH mentioned this in Unknown Object (Task).Oct 14 2020, 9:09 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL