Page MenuHomePhabricator
Create Task
Maniphest T266202

Move or recable labstore1004 to 10Gbps rack (if needed) and ethernet
Closed, ResolvedPublic
Actions

Description

To be done during [insert scheduled downtime]

  • - put system offline in all checks for maint window
  • - stop backups from (labs|cloud)tore1005
  • - failover to labstore1005/cloudstore1005 to prepare for shutdown
  • - rename/rebuild system from labstore1004 to cloudstore1004, preserving /srv storage
  • - relocate to 10G rack and update netbox
  • - enable PXE for 10G interfaces.
  • - update switch configuration for new primary 10G NIC
  • - remove old switch port info
  • - PXE boot and reimage system
  • - WMCS to fail system back
  • - reconnect DRBD
  • - once DRBD is caught up, re-enable backups

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -0partman: build a recipe to re-image nfs servers
operations/puppetproduction+11 -0cloud nfs: make sure the /srv/scratch dir is there and another fix
operations/puppetproduction+0 -2labstore: remove excess variable from hiera
operations/puppetproduction+25 -1cloud nfs: get primary interface from puppet and create mountpoints
operations/puppetproduction+5 -1labstore1004: move to 'insetup' role
operations/puppetproduction+1 -1cloud nfs: switch to the 10G interface for booting labstore1004
Customize query in gerrit

Related Objects
Search...

Event Timeline

Bstorm created this task.Oct 21 2020, 11:41 PM
Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.
Bstorm mentioned this in T266198: Move labstore1004 and labstore1005 to 10G Ethernet.
Cmjohnson moved this task from Backlog to Cloud Tasks on the ops-eqiad board.Oct 28 2020, 6:21 PM
bd808 edited projects, added cloud-services-team (Hardware); removed cloud-services-team (Kanban).Oct 29 2020, 5:26 PM
bd808 moved this task from Backlog to Racking / Decom on the cloud-services-team (Hardware) board.
Cmjohnson added a subscriber: Cmjohnson.Dec 10 2020, 4:27 PM

This server can stay in C2 and can be converted anytime.

gerritbot added a comment.Dec 10 2020, 11:02 PM

Change 647815 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] partman: build a recipe to re-image nfs servers

https://gerrit.wikimedia.org/r/647815

gerritbot added a project: Patch-For-Review.Dec 10 2020, 11:02 PM
gerritbot added a comment.Dec 17 2020, 3:59 PM

Change 647815 merged by Bstorm:
[operations/puppet@production] partman: build a recipe to re-image nfs servers

https://gerrit.wikimedia.org/r/647815

Maintenance_bot removed a project: Patch-For-Review.Dec 17 2020, 4:10 PM
gerritbot added a comment.Dec 22 2020, 3:49 PM

Change 651545 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: switch to the 10G interface for booting labstore1004

https://gerrit.wikimedia.org/r/651545

gerritbot added a project: Patch-For-Review.Dec 22 2020, 3:49 PM
Stashbot added a comment.Dec 22 2020, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-22T16:15:16Z] <bstorm> downtimed and stopped puppet on labstore1004 and labstore1005 for failover T266202

Bstorm added a comment.Dec 22 2020, 4:54 PM

Ok, that has gone very badly. Working to fix.

gerritbot added a comment.Dec 22 2020, 5:31 PM

Change 651564 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labstore1004: move to 'insetup' role

https://gerrit.wikimedia.org/r/651564

gerritbot added a comment.Dec 22 2020, 5:32 PM

Change 651564 merged by Bstorm:
[operations/puppet@production] labstore1004: move to 'insetup' role

https://gerrit.wikimedia.org/r/651564

gerritbot added a comment.Dec 22 2020, 5:33 PM

Change 651545 merged by Bstorm:
[operations/puppet@production] cloud nfs: switch to the 10G interface for booting labstore1004

https://gerrit.wikimedia.org/r/651545

gerritbot added a comment.Dec 22 2020, 5:56 PM

Change 651568 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: get primary interface from puppet and create mountpoints

https://gerrit.wikimedia.org/r/651568

gerritbot added a comment.Dec 22 2020, 6:48 PM

Change 651568 merged by Andrew Bogott:
[operations/puppet@production] cloud nfs: get primary interface from puppet and create mountpoints

https://gerrit.wikimedia.org/r/651568

gerritbot added a comment.Dec 22 2020, 6:57 PM

Change 651575 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] labstore: remove excess variable from hiera

https://gerrit.wikimedia.org/r/651575

gerritbot added a comment.Dec 22 2020, 6:59 PM

Change 651575 merged by Bstorm:
[operations/puppet@production] labstore: remove excess variable from hiera

https://gerrit.wikimedia.org/r/651575

gerritbot added a comment.Dec 22 2020, 7:50 PM

Change 651608 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: make sure the /srv/scratch dir is there and another fix

https://gerrit.wikimedia.org/r/651608

Bstorm updated the task description. (Show Details)Dec 22 2020, 7:52 PM
gerritbot added a comment.Dec 22 2020, 10:15 PM

Change 651608 merged by Bstorm:
[operations/puppet@production] cloud nfs: make sure the /srv/scratch dir is there and another fix

https://gerrit.wikimedia.org/r/651608

Andrew updated the task description. (Show Details)Dec 22 2020, 11:05 PM
Cmjohnson closed this task as Resolved.Jan 26 2021, 5:40 PM
Bstorm mentioned this in T283385: Upgrade labstore1004, labstore1005, cloudstore1008 and cloudstore1009 to Debian Buster.May 21 2021, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL