Page MenuHomePhabricator
Create Task
Maniphest T266199

Move labstore1005 to 10Gbps rack and ethernet
Closed, ResolvedPublic
Actions

Description

To be done during [insert scheduled downtime]

  • - put system offline in all checks for maint window
  • - stop puppet, replication and backups to codfw to take down server
  • - apply bios/ilo updates
  • - rebuild system from labstore1005 to cloudstore1005, preserving /srv storage
  • - relocate to 10G rack and update netbox
  • - enable PXE for 10G interfaces.
  • - update switch configuration for new primary 10G NIC
  • - connect crossover cable from labstore1004 to secondary 10G NIC
  • - remove old switch port info
  • - PXE boot and reimage system
  • - update netbox with new name and location
  • - reconnect DRBD
  • - update switch and physical labels with new name and location
  • - once DRBD is caught up, re-enable backups

Details

SubjectRepoBranchLines +/-
partman: build a recipe to re-image nfs serversoperations/puppetproduction+3 -0
cloud nfs: fix custom fact output while syncing drbdoperations/puppetproduction+1 -0
cloud nfs: fix broken file referenceoperations/puppetproduction+21 -1
cloud nfs: correct interfaces for 10G changeoperations/puppetproduction+2 -0
Change labstore1005 to role(insetup) so that we can log inoperations/puppetproduction+5 -1
cloud nfs: change primary interface of labstore1005 to 10Goperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm created this task.Oct 21 2020, 11:35 PM
• Bstorm updated the task description. (Show Details)
• Bstorm mentioned this in T266198: Move labstore1004 and labstore1005 to 10G Ethernet.
• Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.Oct 21 2020, 11:41 PM
• Marostegui moved this task from Backlog to Acknowledged on the SRE board.Oct 22 2020, 1:15 PM
• Cmjohnson moved this task from Backlog to Cloud Tasks on the ops-eqiad board.Oct 28 2020, 6:21 PM
bd808 edited projects, added cloud-services-team (Hardware); removed cloud-services-team (Kanban).Oct 29 2020, 5:26 PM
bd808 moved this task from Backlog to Racking / Decom on the cloud-services-team (Hardware) board.
• Cmjohnson subscribed.Dec 9 2020, 5:44 PM

@Bstorm Do we need to update both 1004 and 1005 to 10G at the same time? I can convert 1005 to 10G anytime.

• Bstorm added a comment.Dec 9 2020, 6:11 PM

@Cmjohnson not really at the same time, no. If the 1Gb crossover cable works after converting the primary interface to 10Gb, then that's pretty easy to do. We don't know for sure if it will work because we've had issues in the past having 10Gb and 1Gb ports enabled in the BIOS at the same time :)

The main thing here is just making sure we are ready to re-image it safely (preserving the data) and doing that part fairly quickly so that we can get replication back up (this is a replica of labstore1004). If we can move just the primary interface, and things work, then it can be done well before labstore1004. If that fails, then we'll need to figure out a 10Gb crossover cable for the two of them and do them at the same time.

• Cmjohnson added a comment.Dec 10 2020, 4:26 PM

@Bstorm I am sorry I confused which one was already in a 10G rack. I need to confirm that 1004 is in C2 and can stay and 1005 is in C5 and needs to move.

• Bstorm added a comment.Dec 10 2020, 5:29 PM

Ahhh, ok. That makes it a bit trickier. This is the one that should move first because it is the standby.

• Cmjohnson added a comment.Dec 10 2020, 5:46 PM

@Bstorm I just found a space for labstore1005. Let's schedule a move for Monday if that works for you, It will go to C4

1004 in C2
1005 in C4

• Bstorm added a subscriber: Andrew.Dec 10 2020, 6:14 PM

I don't know if the re-image is ready at this time (haven't synced up with @Andrew on that), so today would probably not have worked anyway. I can verify by Monday, I think :) Monday works for me.

gerritbot added a comment.Dec 10 2020, 11:02 PM

Change 647815 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] partman: build a recipe to re-image nfs servers

https://gerrit.wikimedia.org/r/647815

gerritbot added a project: Patch-For-Review.Dec 10 2020, 11:02 PM
gerritbot added a comment.Dec 17 2020, 3:59 PM

Change 647815 merged by Bstorm:
[operations/puppet@production] partman: build a recipe to re-image nfs servers

https://gerrit.wikimedia.org/r/647815

Maintenance_bot removed a project: Patch-For-Review.Dec 17 2020, 4:10 PM
Andrew updated the task description. (Show Details)Dec 18 2020, 3:05 PM
gerritbot added a comment.Dec 18 2020, 3:21 PM

Change 650513 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: change primary interface of labstore1005 to 10G

https://gerrit.wikimedia.org/r/650513

gerritbot added a project: Patch-For-Review.Dec 18 2020, 3:21 PM
gerritbot added a comment.Dec 18 2020, 3:28 PM

Change 650513 merged by Bstorm:
[operations/puppet@production] cloud nfs: change primary interface of labstore1005 to 10G

https://gerrit.wikimedia.org/r/650513

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2020, 4:10 PM
gerritbot added a comment.Dec 18 2020, 6:54 PM

Change 650556 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Change labstore1005 to role(insetup) so that we can log in

https://gerrit.wikimedia.org/r/650556

gerritbot added a project: Patch-For-Review.Dec 18 2020, 6:54 PM
gerritbot added a comment.Dec 18 2020, 7:00 PM

Change 650556 merged by Andrew Bogott:
[operations/puppet@production] Change labstore1005 to role(insetup) so that we can log in

https://gerrit.wikimedia.org/r/650556

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2020, 7:10 PM
gerritbot added a comment.Dec 18 2020, 7:19 PM

Change 650562 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: correct interfaces for 10G change

https://gerrit.wikimedia.org/r/650562

gerritbot added a project: Patch-For-Review.Dec 18 2020, 7:19 PM
gerritbot added a comment.Dec 18 2020, 7:22 PM

Change 650562 merged by Bstorm:
[operations/puppet@production] cloud nfs: correct interfaces for 10G change

https://gerrit.wikimedia.org/r/650562

gerritbot added a comment.Dec 18 2020, 7:39 PM

Change 650586 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: fix broken file reference

https://gerrit.wikimedia.org/r/650586

gerritbot added a comment.Dec 18 2020, 7:41 PM

Change 650586 merged by Bstorm:
[operations/puppet@production] cloud nfs: fix broken file reference

https://gerrit.wikimedia.org/r/650586

gerritbot added a comment.Dec 18 2020, 8:47 PM

Change 650599 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloud nfs: fix custom fact output while syncing drbd

https://gerrit.wikimedia.org/r/650599

gerritbot added a comment.Dec 18 2020, 8:55 PM

Change 650599 merged by Bstorm:
[operations/puppet@production] cloud nfs: fix custom fact output while syncing drbd

https://gerrit.wikimedia.org/r/650599

• Cmjohnson closed this task as Resolved.Jan 26 2021, 5:40 PM

This has been completed

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL