@colewhite @herron hello Any update on this? Thanks

Thank you

@BCornwall hello just wanted to follow up on this to know when do you think you or your team will be done with the re-image. Our goal is to have all the re-image done before the new switches arrived which is some times close to the end of this month. Thanks.

@cmooney thank you, yes move-vlan flag cookbook will also work we need to test that. I don't think we have done any in POP sites.

Please see below steps before re-imaging a node into the new vlan

• Netbox

1- Search for the node in Netbox
2- Click "interface" and find the interface(nic) that has both the IPV4 and IPV6 update the IP addresses by clicking on each IP address and Edit
Example:
if the IPV4 address is 10.132.0.16/24 change it to 10.132.1.16/24 (0 to 1)
if the IPV6 address is 2001:df2:e500:101:10:132:0:16/64 change it to 2001:df2:e500:102:10:132:1:16/64 (101 to 102 and 0 to 1)
4- Setup the switch interface to the new VLAN.

• Search for asw-0604-eqsin
• click on "interface"
• Find the interface of the server you are working on
• click on the interface (xe-1/0/x)
• click on Edit on the top right corner
• Navigate to "Untagged VLAN" and change it to 512 if public1-604 or 522 if private1-604

5- Run homer on the switch (sudo homer asw1-eqsin* commit "change vlan for cp5032")
6- Run the netbox dns cookbook

@BCornwall re-image done on cp5032. The node is now on the new private1-604-eqsin vlan. The DHCP issue I was having, was that ae1.522 was not in the dhcp relay so i made a patch to add both ae1.512 and ae1.522

@BCornwall thank you will do that after lunch doing some onsite work

@BCornwall hello can you please provide me with one CP node in rack 604 that i can use later on today to test the re-image on the new subnet?
https://netbox.wikimedia.org/dcim/racks/78/
Thanks

VRRP is up on cr2-eqsin

cr2-eqsin> show interfaces terse | match "ae1.512|ae1.522"    
et-0/0/1.512            up    up   aenet    --> ae1.512
et-0/0/1.522            up    up   aenet    --> ae1.522
ae1.512                 up    up   inet     103.102.166.33/27
ae1.522                 up    up   inet     10.132.1.1/24

cr2-eqsin> show vrrp brief | match "ae1.512|ae1.522" 
ae1.512       up              3   master   Active      A  0.315 lcl    103.102.166.34 
ae1.512       up              3   master   Active      A  0.146 lcl    2001:df2:e500:2::2
ae1.522       up              4   master   Active      A  0.172 lcl    10.132.1.2     
ae1.522       up              4   master   Active      A  0.706 lcl    2001:df2:e500:102::2

cr2-eqsin> show vrrp summary | match "ae1.512|ae1.522"
ae1.512       up              3   master          Active    lcl    103.102.166.34     
ae1.512       up              3   master          Active    lcl    2001:df2:e500:2::2 
ae1.522       up              4   master          Active    lcl    10.132.1.2         
ae1.522       up              4   master          Active    lcl    2001:df2:e500:102::2

@cmooney yes we will change the VLAN-id and rename the VLAN for rack 0603 during the switch migration. so it will be 511 and 521. see https://phabricator.wikimedia.org/T418439 for the irb interface creation.

Email back from Nokia team

The target release is still being considered. I’ll let you know once we have more information.

Both switches are now set to offline. The only step left is for onsite to remove all the cables and unrack it for recycle

I sent a follow up email on this and Engineer said he will get back with me

All 3 routers are now up to date.

@Jclark-ctr partman fixed 1015 is done you can install 1016. thanks

@ayounsi no ETA was given to me but yes i can can follow up with them.

The ULSFO switch refresh is complete. Good to close this task.

The last BGP session between cr3 and asw1-23 is now up, We ca now close this task. Thanks to all that did help on this project.

This is complete. thanks to @Jhancock.wm and @Jgreen

@ssingh i think it will be best to depool the site since this will be my first time doing the draining process I will like to be on the safe side.

@ssingh and team now that we are done with the switch refresh and everything is stable in ulsfo and after we connect the missing link between cr3 and asw1-23 We will like to schedule a 3 hours downtime for the JUNOS upgrade on the core routers next week May 20th at 9:45am CT 10:45am EST. Please let us know if this time and date works for you.
Thanks.

@ayounsi we can close this task.

@Jhancock.wm when you have time can you please look and see if there are any bad disks on this server? Thanks

All the servers in rack 23 are online and ready for re-image. I tested the re-image on cp4038 and completed with no issues after @ayounsi fixed the DHCP issue. The list of servers above are still on 10G because the 1m DAC 25G were too short to use. We will be ordering some 2m 25G cable for the replacement. As for now the migration on DC-ops and Netops side is complete.

@RobH see below the list of node still on 10G DAC that We will need to move to 25G DAC. Can you please order 7x2m 25G DAC? Thank you
A:papaul@asw1-23-ulsfo# show interface brief
+---------------------+------------------------------------+------------------------------------+------------------------------------+------------------------------------+------------------------------------+

+=====================+====================================+====================================+====================================+====================================+====================================+

We can close this

All the servers in rack 22 are connected to the new switch and all the link are up I just tested cp4037 but all others should be online.
We had 2 issues :
1- The 1m MTP cable order to the switch/router connections was too short so we didn't make the connection from asw1-23 ethernet-1/55 to cr3 et-0/0/2.@RobH
has put in a order for purchases some 2M
2- The 1m 25G DAC cables where to short for the server/switch connection so we used most of the 2M 25G DAC for rack 22. I asked them to provide with the count of cable left onsite to see if we can use some 1M the servers close to the switch in rack 23 and for the others servers with can keep them at 10G and order more 2m 25G DAC
What left?

• rack 23 servers migration to new switches
• some cable id's that i am still waiting for
• move oob to ge-0/0/7
• DNS name for IPV6 in Netbox

ok thank you.

@cmooney please see below for all the DNS names for IPV6 needed. Thanks

Yes I can take care of that.

@RobH Remote hands instructions are ready @ https://docs.google.com/document/d/1EW6hxHCQjXPy1PXQWluwOTnCl_AHddI34iOYHdJuvek/edit?tab=t.0
Please review and let me know if all good and i can open a ticket and submit just the second and third stage steps. Thanks

@ssingh important note:
The public subnet mask for servers in rack 103.02.22 will be changing for /28 to /27 so will will have to manually change the subnet mask of dns4003 (198.35.26.8/28) which is the only host on public VLAN in that rack.

@ssingh hello just wanted to let you and your team that we have decided to do the switch refresh starting May 4th to May 6th ( 3 days)

• First day : onsite work and validation
• second day: configuration and testing and some servers re-image
• Third day: complete servers re-image

@Jclark-ctr Netbox is no longer reporting errors on this server , once @jcrespo done putting the server back in production you can resolve this task.

@jcrespo I am done all yours. This server is power on. Thank you

Please see below for the the steps on how to use sum to update chassis , board and product information on SuperMicro servers

- Navigate to the directory when sum is. I just have it in my home directory 
cd sum_2.15.0_Linux_x86_64_20251104/sum_2.15.0_Linux_x86_64/
-  export the DMI info to a file
sudo ./sum -c GetDmiInfo --file /home/pt1979/dmi_info.txt
- open the file and change all the parts needed save the file 
- Push the changes to the server 
sudo ./sum -c ChangeDmiInfo --file /home/pt1979/dmi_info.txt
After the step below you will be asked to reboot the server

Planing on doing the mr router on April 16th at 10am CT

@jcrespo we can do this next week Wednesday April 15th at 10am CT . Thank you.

@jcrespo hello We have an issue with the serial number on this sever and we have some update from SM on how to fix it but we will have to take the server offline for that. Is there a date and time we can work on this?

BGP is up and OSPF removed

@VRiley-WMF any update on this: I will like to move the OOB tomorrow during the maintenance window. Thanks

All the BGP sessions are up

mr1-eqiad# run show bgp summary group Production 
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 1 Peers: 4 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       2          0          0          0          0          0
inet6.0              
                       1          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
208.80.154.204        14907          9          8       0       0        2:28 Establ
  inet.0: 0/1/1/0
208.80.154.206        14907          3          3       0       0          12 Establ
  inet.0: 0/1/1/0
2620:0:861:fe04::1       14907          8          8       0       0        2:24 Establ
  inet6.0: 0/1/1/0
2620:0:861:fe05::1       14907          2          3       0       0           7 Establ
  inet6.0: 0/0/0/0

@VRiley-WMF @Jclark-ctr when you are next onsite, can you please look for 1 QFX-SFP-1GE-T and plug it into mr1-eqiad ge-0/0/7? Thank you

@SLyngshede-WMF thank you very much.

@Jclark-ctr
For wikikube-worker 1371 wrong serial number in Netbox it was S497720X5834979 after the 5 is it not a 8 but a B S497720X5B34979
For backup1012 (WMF10495) S480845X4915849' is the serial number showing on the outside and 'S480845X3505676' is the serial number puppetdb is getting from puppet facter which is the serial number on the main board so the serial number that puppetdb is proving is always the right one so update Netbox with the serial number from puppetdb. Update the BMC and see if this fix the issue because BIOS is reporting

System Information
	Manufacturer: Supermicro
	Product Name: SSG-620P-E1CR24H
	Version: 0123456789
	Serial Number: S480845X3505676
	UUID: 707e3200-2ede-11ee-8000-905a0800adda
	Wake-up Type: Power Switch
	SKU Number: To be filled by O.E.M.
	Family: Family

and the BMC is reporting

S480845X4915849

if updating the BMC doesn't fix the issue please open a ticket and cc me. Thanks

@ayounsi please see below for the BGP config to setup BGP and remove OSFP between the mr router and the core routers. I will send out gerrit patch later today and merge it when ready to make the changes. Thanks

### mr1-eqiad

@ayounsi yes I can setup for next week since this week we have the DC switch over.