Page MenuHomePhabricator
Create Task
Maniphest T196560

(Need by: TBD) rack/setup/install LVS200[7-10]
Closed, ResolvedPublic
Actions

Description

This task will track the receiving, racking, setup, and installation of LVS200[7-10]

Racking proposal

serversRackNIC1NIC2NIC3NIC4
LVS2007A2asw-a2asw-b7asw-c7asw-d7
LVS2008B2asw-b2asw-a7asw-c2asw-d2
LVS2009C2asw-c2asw-a2asw-b2asw-d2
LVS2010D2asw-d2asw-a2asw-b2asw-c2

@BBlack since lvs2007 will be racked in the same rack as Lvs200[1-3] and lvs2008 in the same rack as lvs200[4-6] I can re-use the cables/fibers off one of the old LVS if we decommission one of them and for lvs2009 and lvs2010, I can pull new cables/fibers. Let me know what you think.

LVS2007:

  • - receive in system on procurement task T193820
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan) end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run - set to staged in netbox
  • - handoff for service implementation - set to active when performing its service

LVS2008:

  • - receive in system on procurement task T193820
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan) end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run - set to staged in netbox
  • - handoff for service implementation - set to active when performing its service

LVS2009:

  • - receive in system on procurement task T193820
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan) end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

LVS2010:

  • - receive in system on procurement task T193820
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan) end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

Wiring progress

serverNIC1NIC2NIC3NIC4
lvs2007completecompletecompletecompelet
lvs2008completecompletecompletecomplete
lvs2009completecompletecompletecomplete
lvs2010completecompletecompletecomplete

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT196560 (Need by: TBD) rack/setup/install LVS200[7-10]
 ResolvedPapaulT196946 switch port configuration for lvs200[7-10]
 ResolvedPapaulT205970 lvs2009/lvs2010 with no RAID configured

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Vgutierrez added a comment.Oct 15 2019, 3:27 AM

@Papaul you can proceed at will with lvs2009 and lvs2010 because they are not handling production traffic at the moment

Papaul added a comment.Oct 15 2019, 5:04 PM

@Vgutierrez firmware upgrade complete on both servers

gerritbot added a comment.Feb 25 2020, 7:57 AM

Change 574659 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Reimage lvs2010 as buster

https://gerrit.wikimedia.org/r/574659

gerritbot added a project: Patch-For-Review.Feb 25 2020, 7:57 AM
gerritbot added a comment.Feb 25 2020, 7:59 AM

Change 574659 merged by Vgutierrez:
[operations/puppet@production] install_server: Reimage lvs2010 as buster

https://gerrit.wikimedia.org/r/574659

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 8:10 AM
gerritbot added a comment.Feb 25 2020, 11:23 AM

Change 574714 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Rename ifaces for lvs2010

https://gerrit.wikimedia.org/r/574714

gerritbot added a project: Patch-For-Review.Feb 25 2020, 11:23 AM
gerritbot added a comment.Feb 25 2020, 11:32 AM

Change 574714 merged by Vgutierrez:
[operations/puppet@production] lvs: Rename ifaces for lvs2010

https://gerrit.wikimedia.org/r/574714

gerritbot added a comment.Feb 25 2020, 11:36 AM

Change 574715 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Set txqlen for the proper ifaces on lvs2010

https://gerrit.wikimedia.org/r/574715

gerritbot added a comment.Feb 25 2020, 11:39 AM

Change 574715 merged by Vgutierrez:
[operations/puppet@production] lvs: Set txqlen for the proper ifaces on lvs2010

https://gerrit.wikimedia.org/r/574715

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 12:10 PM
gerritbot added a comment.Feb 25 2020, 12:41 PM

Change 574737 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Set lvs2010 as a secondary LVS

https://gerrit.wikimedia.org/r/574737

gerritbot added a project: Patch-For-Review.Feb 25 2020, 12:42 PM
gerritbot added a comment.Feb 25 2020, 1:37 PM

Change 574737 merged by Vgutierrez:
[operations/puppet@production] lvs: Set lvs2010 as a secondary LVS

https://gerrit.wikimedia.org/r/574737

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 2:11 PM
gerritbot added a comment.Feb 25 2020, 2:17 PM

Change 574753 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Set BGP peers for lvs2010

https://gerrit.wikimedia.org/r/574753

gerritbot added a project: Patch-For-Review.Feb 25 2020, 2:17 PM
gerritbot added a comment.Feb 25 2020, 2:26 PM

Change 574753 merged by Vgutierrez:
[operations/puppet@production] lvs: Set BGP peers for lvs2010

https://gerrit.wikimedia.org/r/574753

Stashbot mentioned this in T245984: Upgrade load balancers to buster.Feb 25 2020, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T14:30:02Z] <vgutierrez> restart pybal with BGP enabled on lvs2010 - T245984 T196560

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 3:12 PM
gerritbot added a comment.Feb 25 2020, 3:15 PM

Change 574771 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Increase BGP MED on lvs2010 to 102

https://gerrit.wikimedia.org/r/574771

gerritbot added a project: Patch-For-Review.Feb 25 2020, 3:15 PM

Change 574772 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Set up lvs2009 as a low-traffic LVS

https://gerrit.wikimedia.org/r/574772

gerritbot added a comment.Feb 25 2020, 3:17 PM

Change 574771 merged by Vgutierrez:
[operations/puppet@production] lvs: Increase BGP MED on lvs2010 to 102

https://gerrit.wikimedia.org/r/574771

gerritbot added a comment.Feb 25 2020, 3:34 PM

Change 574772 merged by Vgutierrez:
[operations/puppet@production] lvs: Set up lvs2009 as a low-traffic LVS

https://gerrit.wikimedia.org/r/574772

gerritbot added a comment.Feb 25 2020, 4:07 PM

Change 574783 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Enable BGP in lvs2009

https://gerrit.wikimedia.org/r/574783

RobH unsubscribed.Feb 25 2020, 4:11 PM
gerritbot added a comment.Feb 25 2020, 4:23 PM

Change 574783 merged by Vgutierrez:
[operations/puppet@production] lvs: Enable BGP in lvs2009

https://gerrit.wikimedia.org/r/574783

Stashbot added a comment.Feb 25 2020, 4:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T16:25:06Z] <vgutierrez> enable BGP in lvs2009 - T196560 T245984

Vgutierrez added a comment.Feb 25 2020, 4:35 PM

@BBlack @Papaul lvs2009 and lvs2010 are now online as secondary load balancers and ready to take over lvs2003 and lvs2006 respectively.

This should be enough to unblock this task and be able to continue with lvs2007 and lvs2008

Papaul added a comment.Feb 25 2020, 5:40 PM

@Vgutierrez thanks for the update I will wait on @BBlack when he is done decommissioning lvs2003 and lvs2006 to proceed

wiki_willy renamed this task from rack/setup/install LVS200[7-10] to (Need by: TBD) rack/setup/install LVS200[7-10].Feb 26 2020, 1:42 AM
Vgutierrez added a comment.Feb 26 2020, 7:38 AM

@Papaul I think I can handle that as well. I'll let you know, the first one will be lvs2006

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2020, 8:10 AM
gerritbot added a comment.Feb 27 2020, 7:58 AM

Change 575203 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Replace lvs2006 with lvs2010

https://gerrit.wikimedia.org/r/575203

gerritbot added a project: Patch-For-Review.Feb 27 2020, 7:58 AM
gerritbot added a comment.Feb 27 2020, 10:43 AM

Change 575203 merged by Vgutierrez:
[operations/puppet@production] lvs: Replace lvs2006 with lvs2010

https://gerrit.wikimedia.org/r/575203

Stashbot added a comment.Feb 27 2020, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-27T10:54:30Z] <vgutierrez> replacing lvs2006 with lvs2010 - T196560 T245984

Stashbot added a comment.Feb 27 2020, 10:58 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-27T10:58:45Z] <vgutierrez> stop pybal on lvs2003 to let lvs2010 take the traffic for a little bit - T196560 T245984

Stashbot added a comment.Feb 27 2020, 11:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-27T11:02:56Z] <vgutierrez> start pybal on lvs2003 - T196560 T245984

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2020, 11:10 AM
Vgutierrez added a comment.Feb 27 2020, 11:59 AM

@Papaul lvs2006 is all yours, I've filed T246329

gerritbot added a comment.Feb 27 2020, 12:05 PM

Change 575224 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Replace lvs2003 with lvs2009

https://gerrit.wikimedia.org/r/575224

gerritbot added a project: Patch-For-Review.Feb 27 2020, 12:05 PM
gerritbot added a comment.Feb 27 2020, 12:11 PM

Change 575224 merged by Vgutierrez:
[operations/puppet@production] lvs: Replace lvs2003 with lvs2009

https://gerrit.wikimedia.org/r/575224

Stashbot added a comment.Feb 27 2020, 12:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-27T12:14:48Z] <vgutierrez> replace lvs2003 with lvs2009 - T196560 T245984 T246334

Stashbot mentioned this in T246334: decommission lvs2003.codfw.wmnet.Feb 27 2020, 12:14 PM
Vgutierrez added a comment.Feb 27 2020, 12:48 PM

@Papaul same for lvs2003: T246334

Regarding lvs2007 and lvs2008, please update the NICs FW to the same versions as you did for lvs2009 and lvs2010. Thanks!

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2020, 1:11 PM
Papaul updated the task description. (Show Details)Feb 27 2020, 5:52 PM
Papaul claimed this task.Feb 28 2020, 3:52 AM

Firmware upgrade on lvs2008
Before
BIOS Version 1.3.7
iDRAC Firmware Version 3.15.17.15

After
BIOS Version 2.4.8
iDRAC Firmware Version 4.00.00

gerritbot added a comment.Feb 28 2020, 4:16 PM

Change 575565 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address for lvs200[7-8]

https://gerrit.wikimedia.org/r/575565

gerritbot added a project: Patch-For-Review.Feb 28 2020, 4:16 PM
Papaul updated the task description. (Show Details)Feb 28 2020, 4:42 PM
gerritbot added a comment.Feb 28 2020, 7:34 PM

Change 575565 merged by Papaul:
[operations/puppet@production] DHCP: Add MAC address for lvs200[7-8]

https://gerrit.wikimedia.org/r/575565

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2020, 8:10 PM
Papaul added a comment.Feb 28 2020, 8:45 PM

@Vgutierrez lvs2008 is ready for service will can on lvs2007 on Monday

gerritbot added a comment.Mar 2 2020, 6:32 AM

Change 575826 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Add lvs2008 as a high-traffic2 load balancer

https://gerrit.wikimedia.org/r/575826

gerritbot added a project: Patch-For-Review.Mar 2 2020, 6:32 AM
gerritbot added a comment.Mar 2 2020, 6:43 AM

Change 575826 merged by Vgutierrez:
[operations/puppet@production] lvs: Add lvs2008 as a high-traffic2 load balancer

https://gerrit.wikimedia.org/r/575826

ops-monitoring-bot added a comment.Mar 2 2020, 6:50 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

lvs2008.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003020649_vgutierrez_222100_lvs2008_codfw_wmnet.log.

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2020, 7:10 AM
ops-monitoring-bot added a comment.Mar 2 2020, 7:19 AM

Completed auto-reimage of hosts:

['lvs2008.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Mar 2 2020, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-02T07:22:57Z] <vgutierrez> upgrading NICs FW on lvs2008 - T196560 T203194

Stashbot mentioned this in T203194: cp1075-90 - bnxt_en transmit hangs.Mar 2 2020, 7:22 AM
Vgutierrez added a comment.Mar 2 2020, 7:39 AM

@Papaul I had to upgrade the NIC FW on lvs2008

before
vgutierrez@lvs2008:~$ sudo -i ethtool -i ens2f0np0
driver: bnxt_en
version: 1.9.2
firmware-version: 20.6.151.0/pkg 20.06.05.11
after
vgutierrez@lvs2008:~$ sudo -i ethtool -i ens2f0np0
driver: bnxt_en
version: 1.9.2
firmware-version: 214.0.166.0/pkg 21.40.16.60

besides that, everything looks good on lvs2008.

gerritbot added a comment.Mar 2 2020, 2:01 PM

Change 576043 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Add missing mapping of lvs2008 as high-traffic2

https://gerrit.wikimedia.org/r/576043

gerritbot added a project: Patch-For-Review.Mar 2 2020, 2:01 PM
gerritbot added a comment.Mar 2 2020, 2:13 PM

Change 576043 merged by Vgutierrez:
[operations/puppet@production] lvs: Add missing mapping of lvs2008 as high-traffic2

https://gerrit.wikimedia.org/r/576043

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2020, 3:11 PM
Papaul updated the task description. (Show Details)Mar 2 2020, 3:31 PM
Papaul closed subtask T196946: switch port configuration for lvs200[7-10] as Resolved.Mar 2 2020, 4:09 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Mar 2 2020, 4:11 PM
Papaul closed this task as Resolved.Mar 2 2020, 6:08 PM
Papaul updated the task description. (Show Details)

@Vgutierrez lvs2007 is ready for service.

gerritbot added a comment.Mar 3 2020, 10:05 AM

Change 576287 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Test BGP in lvs2008

https://gerrit.wikimedia.org/r/576287

gerritbot added a project: Patch-For-Review.Mar 3 2020, 10:05 AM
gerritbot added a comment.Mar 3 2020, 10:13 AM

Change 576287 merged by Vgutierrez:
[operations/puppet@production] lvs: Test BGP in lvs2008

https://gerrit.wikimedia.org/r/576287

Stashbot added a comment.Mar 3 2020, 10:44 AM

Mentioned in SAL (#wikimedia-traffic) [2020-03-03T10:44:45Z] <vgutierrez> replace lvs2002 with lvs2008 - T196560

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 11:11 AM
gerritbot added a comment.Mar 3 2020, 1:25 PM

Change 576328 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Add lvs2007 as a high-traffic1 load balancer

https://gerrit.wikimedia.org/r/576328

gerritbot added a project: Patch-For-Review.Mar 3 2020, 1:25 PM
gerritbot added a comment.Mar 3 2020, 1:30 PM

Change 576328 merged by Vgutierrez:
[operations/puppet@production] lvs: Add lvs2007 as a high-traffic1 load balancer

https://gerrit.wikimedia.org/r/576328

ops-monitoring-bot added a comment.Mar 3 2020, 1:31 PM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

lvs2007.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003031331_vgutierrez_52584_lvs2007_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 3 2020, 2:01 PM

Completed auto-reimage of hosts:

['lvs2007.codfw.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 2:10 PM
gerritbot added a comment.Mar 3 2020, 2:18 PM

Change 576342 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] lvs: Test BGP in lvs2007

https://gerrit.wikimedia.org/r/576342

gerritbot added a project: Patch-For-Review.Mar 3 2020, 2:18 PM

Change 576342 merged by Vgutierrez:
[operations/puppet@production] lvs: Test BGP in lvs2007

https://gerrit.wikimedia.org/r/576342

Stashbot added a comment.Mar 3 2020, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-03T14:42:28Z] <vgutierrez> replace lvs2001 with lvs2007 - T196560

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 3:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL