Change Details

This task will track the full setup steps for our remote hands service in eqsin to finish the racking and setup to allow remote administration of the new ganeti nodes purchased on T222284. Please note this is not EQ SG3 remote hands, but a third party company we are trying out for our sporadic #ops-eqsin tasks. Each system will have nearly identical setup steps, but we'll provide the entire thing in a per server checklist, since each system will connect to different network ports and each cable will have a unique id number. All systems will need the full details, including every single cable ID number and port info for both ends (power, mgmt, and network). == Info to pass along to remote hands == Scheduling: We will need over 24 hours advance & the full name of the technician(s) going on-site. The names provided must exactly match your identification provided to Equinix Security at arrival to SG3. Ideally, work would take place starting at 8 or 9AM Singapore, as 9AM Singapore is 6PM Pacific time. @robh would be the ideal point of contact for #dc-ops (with @wiki_willy as fallback), as other team members are located in time zones less convenient to synchronization with an on-site in SG3. Our preferred days of the week are Monday, Tuesday, or Thursday. We don't like to do work in caching sites on Friday, since any outage that occurs interrupts the weekend for some of our staff. Wednesdays are non-preferred by @robh (but not a hard blocker.) Additional Tasks: <@robh will fill out this section with the other pending #ops-eqsin tasks for them to knock out while onsite with the below racking task.> Pre-staging: [] - Report back any inaccuracies in stock levels or directions below. [] - Locate the WMF asset tags somewhere in the rack. There should be a partial roll of them, they should have Property of Wikimedia, then an asset tag #. You will be placing a single one of these on each of the servers listed below and reporting back that number. [] - Locate 3 green cat5 patch cables to use for these systems idrac/ilom/mgmt connections. You will want to use the shortest cable that can effectively route from the servers to the switch with strain relief. [] - Locate 3 of the fs.com black DAC cables. These are cables iwth SFP+ transceivers permanently attached to their ends. There should be (3) 1M, (3) 2M, and (9) 3M spares in the racks. Please use the shortest cable that will still route effectively with strain relief. (I don't recall what would work best, but I expect it will be a 2M or 3M for these.) [] - Locate a label printer, as you will need to print number labels for the cables, and server hostname labels for the servers. @robh notes: The last cable ID used in eqsin was # 1089, so all of these will increment from there. [[ https://netbox.wikimedia.org/dcim/devices/2190/ | ganeti5001 ]]: [] - This system is located in rack 603 (06:040020:0603) : U13. [] - Pull out the 'luggage tag' on the front of the server. [] - Double check that the above listed system for ganeti5001 has a service tag of FLXS8X2. [] - Locate asset tags in rack, they should have Property of Wikimedia, then an asset tag #. [] - Place a single WMF asset tag on the luggage tag, ensuring it does not block any other information. Please look at the luggage tags of other systems in the rack as an example. [] - Report back the WMF asset tag number for ganeti5001. [] - apply hostname label of 'ganeti5001' to the front and the back of the chassis (see other servers in rack as examples.) [] - Place labels on both ends of the power cables, with the numbers 1090 & 1091. [] - Plug ganeti5001 PSU1 into PS1 (all the other server PSU1 go to PS1 as well) port 16 with cable 1090. [] - Plug ganeti5001 PSU2 into PS2 (all the other server PSU2 go to PS2 as well) port 16 with cable 1091. [] - Label and connect the shortest green patch cable you can use to effectively route from ganeti5001's dedicated idrac port to the next available port on msw1-esin with the number 1092. [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5001's eth0 port to asw-0603-eqsin (Juniper EX4600) port xe-0/0/9 with label number 1093. At this point, you may want to stop, and complete the initial label and wiring steps listed above for all three systems before proceeding to software setup. [] - With everything connected, attach the KVM (located in rack 604 (06:040020:0604) to the server to complete software setup for remote administration. The back of the KVM has the cables needed to attach to the back of one server at a time. [] - Boot up ganeti5001 and enter the BIOS setup. [] - Go to the serial redirection in bios and set/confirm the following: [] - Serial Redirection > Serial Communication set to: On with console redirection via COM2 [] - Serial Redirection > Serial Port Address set to Serial Device1=COM1,Serial Device2=COM2 [] - Serial Redirection > External Serial Connector: Serial Device 1 [] - Serial Redirection > Failsafe Baud Rage: 115200 [] - Serial Redirection > Remote Terminal Type: VT100/VT200 [] - Serial Redirection > Redirection after boot set to disabled [] - Save settings (esc, confirm save) back to the main setup screen (where it lists BIOS , IDRAC, and Periphials setup.) [] - Select iDRAC Settings and set/confirm the following: [] - iDRAC Setup > Network > Enable NIC is set to Enabled - should already be set to this, just a double-check [] - iDRAC Setup > Network > Nic Selection is set to Dedicated (iDRAC7 Enterprise only) - should be set to this already, if it won't change, it means the DRAC Enterprise License did not apply correctly during purchase. Please contact RobH to get this fixed. [] - iDRAC Setup > Network > Change IPv4 to Static (not DHCP assigned) [] - iDRAC Setup > Network > IPV4 Settings > IPv4 IP Address: [] - iDRAC Setup > Network > IPV4 Settings > Subnet Mask: 255.255.128.0 [] - iDRAC Setup > Network > IPV4 Settings > Gateway: 10.132.128.1 [] - iDRAC Setup > Network > IPV4 Settings > Don't bother setting dns servers. [] - iDRAC Setup > User Configuration > Set username to root (should already be this) [] - iDRAC Setup > User Configuration > Set password to asset tag # (the WMF#) (case sensitive). [] - iDRAC Setup > Exit and save all settings. [] - Report back to RobH so he can attempt to remotely connect to ganeti5001 to test the settings. [[ https://netbox.wikimedia.org/dcim/devices/2189/ | ganeti5002 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5002's eth0 port to asw-0604-eqsin (Junier EX4600) port xe-1/0/9 with label number <#>. [[ https://netbox.wikimedia.org/dcim/devices/2188/ | ganeti5003 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5003's eth0 port to asw-0603-eqsin (Junier EX4600) port xe-0/0/10 with label number <#>.

This task will track the full setup steps for our remote hands service in eqsin to finish the racking and setup to allow remote administration of the new ganeti nodes purchased on T222284. Please note this is not EQ SG3 remote hands, but a third party company we are trying out for our sporadic #ops-eqsin tasks. Each system will have nearly identical setup steps, but we'll provide the entire thing in a per server checklist, since each system will connect to different network ports and each cable will have a unique id number. All systems will need the full details, including every single cable ID number and port info for both ends (power, mgmt, and network). == Info to pass along to remote hands == Contact Info: <rob to fill this out with how to stay in contact while onsite in SG3> Scheduling: We will need over 24 hours advance & the full name of the technician(s) going on-site. The names provided must exactly match your identification provided to Equinix Security at arrival to SG3. Ideally, work would take place starting at 8 or 9AM Singapore, as 9AM Singapore is 6PM Pacific time. @robh would be the ideal point of contact for #dc-ops (with @wiki_willy as fallback), as other team members are located in time zones less convenient to synchronization with an on-site in SG3. Our preferred days of the week are Monday, Tuesday, or Thursday. We don't like to do work in caching sites on Friday, since any outage that occurs interrupts the weekend for some of our staff. Wednesdays are non-preferred by @robh (but not a hard blocker.) Additional Tasks: <@robh will fill out this section with the other pending #ops-eqsin tasks for them to knock out while onsite with the below racking task.> Pre-staging: [] - Report back any inaccuracies in stock levels or directions below. [] - Locate the WMF asset tags somewhere in the rack. There should be a partial roll of them, they should have Property of Wikimedia, then an asset tag #. You will be placing a single one of these on each of the servers listed below and reporting back that number. [] - Locate 3 green cat5 patch cables to use for these systems idrac/ilom/mgmt connections. You will want to use the shortest cable that can effectively route from the servers to the switch with strain relief. [] - Locate 3 of the fs.com black DAC cables. These are cables iwth SFP+ transceivers permanently attached to their ends. There should be (3) 1M, (3) 2M, and (9) 3M spares in the racks. Please use the shortest cable that will still route effectively with strain relief. (I don't recall what would work best, but I expect it will be a 2M or 3M for these.) [] - Locate a label printer, as you will need to print number labels for the cables, and server hostname labels for the servers. @robh notes: The last cable ID used in eqsin was # 1089, so all of these will increment from there. [[ https://netbox.wikimedia.org/dcim/devices/2190/ | ganeti5001 ]]: [] - This system is located in rack 603 (06:040020:0603) : U13. [] - Pull out the 'luggage tag' on the front of the server. [] - Double check that the above listed system for ganeti5001 has a service tag of FLXS8X2. [] - Locate asset tags in rack, they should have Property of Wikimedia, then an asset tag #. [] - Place a single WMF asset tag on the luggage tag, ensuring it does not block any other information. Please look at the luggage tags of other systems in the rack as an example. [] - Report back the WMF asset tag number for ganeti5001. [] - apply hostname label of 'ganeti5001' to the front and the back of the chassis (see other servers in rack as examples.) [] - Place labels on both ends of the power cables, with the numbers 1090 & 1091. [] - Plug ganeti5001 PSU1 into PS1 (all the other server PSU1 go to PS1 as well) port 16 with cable 1090. [] - Plug ganeti5001 PSU2 into PS2 (all the other server PSU2 go to PS2 as well) port 16 with cable 1091. [] - Label and connect the shortest green patch cable you can use to effectively route from ganeti5001's dedicated idrac port to the next available port on msw1-esin with the number 1092. [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5001's eth0 port to asw-0603-eqsin (Juniper EX4600) port xe-0/0/9 with label number 1093. At this point, you may want to stop, and complete the initial label and wiring steps listed above for all three systems before proceeding to software setup. [] - With everything connected, attach the KVM (located in rack 604 (06:040020:0604) to the server to complete software setup for remote administration. The back of the KVM has the cables needed to attach to the back of one server at a time. [] - Boot up ganeti5001 and enter the BIOS setup. [] - Go to the serial redirection in bios and set/confirm the following: [] - Serial Redirection > Serial Communication set to: On with console redirection via COM2 [] - Serial Redirection > Serial Port Address set to Serial Device1=COM1,Serial Device2=COM2 [] - Serial Redirection > External Serial Connector: Serial Device 1 [] - Serial Redirection > Failsafe Baud Rage: 115200 [] - Serial Redirection > Remote Terminal Type: VT100/VT200 [] - Serial Redirection > Redirection after boot set to disabled [] - Save settings (esc, confirm save) back to the main setup screen (where it lists BIOS , IDRAC, and Periphials setup.) [] - Select iDRAC Settings and set/confirm the following: [] - iDRAC Setup > Network > Enable NIC is set to Enabled - should already be set to this, just a double-check [] - iDRAC Setup > Network > Nic Selection is set to Dedicated (iDRAC7 Enterprise only) - should be set to this already, if it won't change, it means the DRAC Enterprise License did not apply correctly during purchase. Please contact RobH to get this fixed. [] - iDRAC Setup > Network > Change IPv4 to Static (not DHCP assigned) [] - iDRAC Setup > Network > IPV4 Settings > IPv4 IP Address: [] - iDRAC Setup > Network > IPV4 Settings > Subnet Mask: 255.255.128.0 [] - iDRAC Setup > Network > IPV4 Settings > Gateway: 10.132.128.1 [] - iDRAC Setup > Network > IPV4 Settings > Don't bother setting dns servers. [] - iDRAC Setup > User Configuration > Set username to root (should already be this) [] - iDRAC Setup > User Configuration > Set password to asset tag # (the WMF#) (case sensitive). [] - iDRAC Setup > Exit and save all settings. [] - Report back to RobH so he can attempt to remotely connect to ganeti5001 to test the settings. [[ https://netbox.wikimedia.org/dcim/devices/2189/ | ganeti5002 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5002's eth0 port to asw-0604-eqsin (Junier EX4600) port xe-1/0/9 with label number <#>. [[ https://netbox.wikimedia.org/dcim/devices/2188/ | ganeti5003 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5003's eth0 port to asw-0603-eqsin (Junier EX4600) port xe-0/0/10 with label number <#>.

This task will track the full setup steps for our remote hands service in eqsin to finish the racking and setup to allow remote administration of the new ganeti nodes purchased on T222284. Please note this is not EQ SG3 remote hands, but a third party company we are trying out for our sporadic #ops-eqsin tasks. Each system will have nearly identical setup steps, but we'll provide the entire thing in a per server checklist, since each system will connect to different network ports and each cable will have a unique id number. All systems will need the full details, including every single cable ID number and port info for both ends (power, mgmt, and network). == Info to pass along to remote hands == Contact Info: <rob to fill this out with how to stay in contact while onsite in SG3> Scheduling: We will need over 24 hours advance & the full name of the technician(s) going on-site. The names provided must exactly match your identification provided to Equinix Security at arrival to SG3. Ideally, work would take place starting at 8 or 9AM Singapore, as 9AM Singapore is 6PM Pacific time. @robh would be the ideal point of contact for #dc-ops (with @wiki_willy as fallback), as other team members are located in time zones less convenient to synchronization with an on-site in SG3. Our preferred days of the week are Monday, Tuesday, or Thursday. We don't like to do work in caching sites on Friday, since any outage that occurs interrupts the weekend for some of our staff. Wednesdays are non-preferred by @robh (but not a hard blocker.) Additional Tasks: <@robh will fill out this section with the other pending #ops-eqsin tasks for them to knock out while onsite with the below racking task.> Pre-staging: [] - Report back any inaccuracies in stock levels or directions below. [] - Locate the WMF asset tags somewhere in the rack. There should be a partial roll of them, they should have Property of Wikimedia, then an asset tag #. You will be placing a single one of these on each of the servers listed below and reporting back that number. [] - Locate 3 green cat5 patch cables to use for these systems idrac/ilom/mgmt connections. You will want to use the shortest cable that can effectively route from the servers to the switch with strain relief. [] - Locate 3 of the fs.com black DAC cables. These are cables iwth SFP+ transceivers permanently attached to their ends. There should be (3) 1M, (3) 2M, and (9) 3M spares in the racks. Please use the shortest cable that will still route effectively with strain relief. (I don't recall what would work best, but I expect it will be a 2M or 3M for these.) [] - Locate a label printer, as you will need to print number labels for the cables, and server hostname labels for the servers. @robh notes: The last cable ID used in eqsin was # 1089, so all of these will increment from there. [[ https://netbox.wikimedia.org/dcim/devices/2190/ | ganeti5001 ]]: [] - This system is located in rack 603 (06:040020:0603) : U13. [] - Pull out the 'luggage tag' on the front of the server. [] - Double check that the above listed system for ganeti5001 has a service tag of FLXS8X2. [] - Locate asset tags in rack, they should have Property of Wikimedia, then an asset tag #. [] - Place a single WMF asset tag on the luggage tag, ensuring it does not block any other information. Please look at the luggage tags of other systems in the rack as an example. [] - Report back the WMF asset tag number for ganeti5001. [] - apply hostname label of 'ganeti5001' to the front and the back of the chassis (see other servers in rack as examples.) [] - Place labels on both ends of the power cables, with the numbers 1090 & 1091. [] - Plug ganeti5001 PSU1 into PS1 (all the other server PSU1 go to PS1 as well) port 16 with cable 1090. [] - Plug ganeti5001 PSU2 into PS2 (all the other server PSU2 go to PS2 as well) port 16 with cable 1091. [] - Label and connect the shortest green patch cable you can use to effectively route from ganeti5001's dedicated idrac port to the next available port on msw1-esin with the number 1092. [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5001's eth0 port to asw-0603-eqsin (Juniper EX4600) port xe-0/0/9 with label number 1093. At this point, you may want to stop, and complete the initial label and wiring steps listed above for all three systems before proceeding to software setup. [] - With everything connected, attach the KVM (located in rack 604 (06:040020:0604) to the server to complete software setup for remote administration. The back of the KVM has the cables needed to attach to the back of one server at a time. [] - Boot up ganeti5001 and enter the BIOS setup. [] - Go to the serial redirection in bios and set/confirm the following: [] - Serial Redirection > Serial Communication set to: On with console redirection via COM2 [] - Serial Redirection > Serial Port Address set to Serial Device1=COM1,Serial Device2=COM2 [] - Serial Redirection > External Serial Connector: Serial Device 1 [] - Serial Redirection > Failsafe Baud Rage: 115200 [] - Serial Redirection > Remote Terminal Type: VT100/VT200 [] - Serial Redirection > Redirection after boot set to disabled [] - Save settings (esc, confirm save) back to the main setup screen (where it lists BIOS , IDRAC, and Periphials setup.) [] - Select iDRAC Settings and set/confirm the following: [] - iDRAC Setup > Network > Enable NIC is set to Enabled - should already be set to this, just a double-check [] - iDRAC Setup > Network > Nic Selection is set to Dedicated (iDRAC7 Enterprise only) - should be set to this already, if it won't change, it means the DRAC Enterprise License did not apply correctly during purchase. Please contact RobH to get this fixed. [] - iDRAC Setup > Network > Change IPv4 to Static (not DHCP assigned) [] - iDRAC Setup > Network > IPV4 Settings > IPv4 IP Address: [] - iDRAC Setup > Network > IPV4 Settings > Subnet Mask: 255.255.128.0 [] - iDRAC Setup > Network > IPV4 Settings > Gateway: 10.132.128.1 [] - iDRAC Setup > Network > IPV4 Settings > Don't bother setting dns servers. [] - iDRAC Setup > User Configuration > Set username to root (should already be this) [] - iDRAC Setup > User Configuration > Set password to asset tag # (the WMF#) (case sensitive). [] - iDRAC Setup > Exit and save all settings. [] - Report back to RobH so he can attempt to remotely connect to ganeti5001 to test the settings. [[ https://netbox.wikimedia.org/dcim/devices/2189/ | ganeti5002 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5002's eth0 port to asw-0604-eqsin (Junier EX4600) port xe-1/0/9 with label number <#>. [[ https://netbox.wikimedia.org/dcim/devices/2188/ | ganeti5003 ]]: [] - Label and connect the shortest black DAC cable you can use to effectively route from ganeti5003's eth0 port to asw-0603-eqsin (Junier EX4600) port xe-0/0/10 with label number <#>.