This task will track the racking, setup and configuration of all PDUs in row A. For all racks that are not network racks (A1 and A8) we shouldn't have any impact on servers going down. The only thing to keep in mind is that during the PDU replacing process the management switch in that particular rack will not be available.
If you are a service owner and think that you need to depool your server(s) during the maintenance window, please put a "YES" in the "List of Servers and network devices" table below.
Thanks
Schedule
| Rack | Date | Time | Comments | |
| A1 | Waiting for PDU's | |||
| A2 | June 21st | 9:30am CT/2:30 pm UTC | ~2hours to complete | |
| A3 | June 23rd | 9:30am CT/2:30 pmUTC | ~2 hours 15 minutes to complete | |
| A4 | June 30th | 9:30am CT/2:30pm UTC | ~ 1 hour 15 minutes to complete | |
| A5 | July 12th | 9:30am CT/2:30pm UTC | CY1 disconnected the whole rack by mistake | |
| A6 | July 14th | 9:30am CT/2:30pm UTC | ~ 1 hour 45 minutes | |
| A7 | August 2nd | 9:30am CT/2:30pm UTC | ||
| A8 | Waiting for PDUs | |||
Per PDU setup Checklist
ps1-a1-codfw/ps2-a1-codfw
send out a notification to notify everybody that the management network will not be available for the whole site. Send out another notification also to net-ops to let them know of the ongoing maintenance.
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A1
| Servers | Do you need to depool? |
| db2075 | No, host decommissioned. |
| db2136 | Yes |
| es2026 | Yes |
| gitlab2002 | |
| kubestage2001 | |
| mc2019 | |
| ml-serve2005 | |
| cr1-codfw | |
| mr1-codfw | |
| msw1-codfw | |
| scs-a1-codfw | |
| asw-a1-codfw | |
| msw-a1-codfw | |
| atlas-codfw | |
ps1-a2-codfw/ps2-a2-codfw
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A2
| Servers | Do you need to depool? |
| authdns2001 | |
| elastic2037 | |
| elastic2038 | |
| elastic2055 | |
| lvs2007 | |
| ms-be2028 | |
| ms-be2029 | |
| ms-be2040 | |
| ms-be2044 | |
| ms-be2051 | |
| thanos-fe2001 | |
| asw-a2-codfw | |
| msw-a2-codfw | |
ps1-a3-codfw/ps2-a3-codfw
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A3
| Servers | Do you need to depool? |
| db2089 | No, will be decommissioned before the date. |
| db2103 | Yes, master, needs downtime the whole chain |
| db2142 | Yes, x2 master |
| es2020 | Yes |
| mw2291 | |
| mw2292 | |
| mw2293 | |
| mw2294 | |
| mw2295 | |
| mw2296 | |
| mw2297 | |
| mw2298 | |
| mw2299 | |
| mw2300 | |
| mw2377 | |
| mw2378 | |
| mw2379 | |
| mw2380 | |
| mw2381 | |
| mw2382 | |
| mw2383 | |
| mw2384 | |
| mw2385 | |
| mw2386 | |
| mw2387 | |
| mw2388 | |
| mw2389 | |
| mw2390 | |
| mw2391 | |
| mw2392 | |
| mw2393 | |
| mw2394 | |
| mw2395 | |
| mw2396 | |
| mw2397 | |
| mw2398 | |
| mw2399 | |
| mw2400 | |
| mw2401 | |
| asw-a3-codfw | |
| msw-a3-codfw | |
ps1-a4-codfw/ps2-a4-codfw
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A4
| Servers | Do you need to depool? |
| asw-a4-codfw | |
| backup2002 | |
| backup2002-array1 | |
| backup2004 | |
| cp2027 | |
| cp2028 | |
| dbprov2001 | |
| ganeti2027 | |
| kafka-main2001 | |
| mc-gp2001 | |
| ms-be2060 | |
| ms-be2062 | |
| msw-a4-codfw | |
| mw2251 | |
| mw2252 | |
| mw2253 | |
| ores2001 | |
ps1-a5-codfw/ps2-a5-codfw===
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A5
| Servers | Do you need to depool? |
| asw-a5-codfw | |
| contint2001 | |
| db2079 | Yes (master but will be switchedover this weekT313798 |
| db2085 | No, host decommissioned |
| db2104 | Yes, s2 master, needs downtime |
| db2121 | Yes, s7 master, needs downtime |
| db2132 | Yes, m1 master, needs downtime |
| db2145 | Yes |
| elastic2025 | |
| ganeti2023 | |
| ganeti2024 | |
| graphite2003 | |
| kubernetes2018 | |
| logstash2001 | |
| maps2005 | |
| mc2020 | |
| ml-serve2001 | |
| msw-a5-codfw | |
| mw2402 | |
| mw2403 | |
| mw2404 | |
| mw2405 | |
| mw2406 | |
| mw2407 | |
| mw2408 | |
| mw2409 | |
| mw2410 | |
| mw2411 | |
| parse2001 | |
| parse2002 | |
| parse2003 | |
| pc2011 | |
| puppetmaster2001 | |
| wdqs2003 | |
ps1-a6-codfw/ps2-a6-codfw===
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A6
| Servers | Do you need to depool? |
ps1-a7-codfw/ps2-a7-codfw===
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A7
| Servers | Do you need to depool? |
| asw-a7-codfw | |
| cloudbackup2001 | Done |
| cp2029 | Done |
| cp2030 | Done |
| elastic2039 | |
| elastic2040 | |
| elastic2056 | |
| ganeti2028 | Powered down |
| ms-be2030 | make sure server is up before moving to rack B2/B4 |
| ms-be2045 | make sure server is up before moving to rack B2/B4 |
| ms-be2052 | make sure server is up before moving to rack B2/B4 |
| thanos-be2001 | |
ps1-a8-codfw/ps2-a8-codfw===
Send out a notification to net-ops to let them know of the ongoing maintenance.
- - receive in new PDUs on T303460
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox
- - Downtime the old PDU in Icinga
- - Run the "Move devices attributes" to move all settings from old PDU to new PDU
- - Login to the master PDU and do the configuration
- - Make sure Icinga is seeing the new PDU
List of Servers and network devices in rack A8
| Servers | Do you need to depool? |