@cmooney thanks for bringing this up and finding the issue. I didn't really think about the mgmt vlan until now reading your comment. What you say about having the firewall to hang the MGMT vlan makes sense in this scenario. If we have to go this path that means we will have to have a long fiber run for 1G from rack C8 where the other fmsw is to rack A1 where the patch panel is in DH7 then run another fiber in DH5 from E3 where the patch panel is to F5 with the second fmsw is to connect both fmsw's . This s a quick fix for this issues no big changes on our side or frack side.
I did a tutorial similar to this where I setup irb interfaces on the core switches and keeping the gateway on the firewall and using it as the virtual-gateway-address on the irb interface. See link below and maybe this it too much complex (DON'T LOOK AT THE EVPN LOL). We can also discuss more during the next meeting. Thank you
https://papaulgigitech.com/index.php?title=Juniper_Collapsed_Spine_with_EVPN

@herron complete closing the task. Thank you.

@herron Hello the default partman recipe for mwlog is not working with new servers so to install mwlog1003 I created a new line for it in the preseed.ymal file.

'mwlog1003':
   - partman/standard.cfg
   - partman/raid10-4dev.cfg

The out put of mdadm -D /dev/md0 is the same as for mwlog1002. Please let me know if all looks good on the server so I can remove the above changes from the pressed.yaml file so that the server defaults to "partman/custom/reuse-lvm-root-4dev.cfg"
Thanks

@Clement_Goubert thanks

@Clement_Goubert hello can you or someone on your team please add these servers to site.pp with the insetup role?
Thanks

Phase 1 of ULSFO migration which was changing the loopback addresses of cr1,cr4 ,mr1 and the IP address of the link between cr3 and cr4 was done today and complete with out an issue. Thanks to all that did help in the process.
Phase 2 will be to have the new switches racked and cable so we can have the configuration done and test out for phase 3 which will be the migration of the servers to the new switches.

I think the quick fix here is for us to go with your option (2) exclude any interface called "mgmt0" for the time being and when Nokia comes up with the upgrade patch for this each re-add mgmt0.

@ssingh Hello and Happy New year I just wanted to check with you once again if it is now safe to resume the loopback IP changes on the core routers in ulsfo. If yes , next week Jan 14the at 10am CT works for you and your team?

I sent a Follow up email to Nokia to ask if there were any updates.

Configuration done

lsw1-a3-codfw# run show route receive-protocol bgp 10.192.254.15

I am still waiting for Nokia to get back in touch with me.

Add interface ge-0/0/[44-47}

[edit interfaces]
+   ge-0/0/44 {
+       description DISABLED;
+       disable;
+   }
+   ge-0/0/45 {
+       description DISABLED;
+       disable;
+   }
+   ge-0/0/46 {
+       description DISABLED;
+       disable;
+   }
+   ge-0/0/47 {
+       description DISABLED;
+       disable;
+   }

Set interface ge-0/0/[44-47] to 1G

lsw1-a3-codfw# show | compare 
[edit chassis fpc 0 pic 0]
+     port 44 {
+         speed 1G;
+     }

IP addresses to use will be

IPV4
mr1-codfw = ge-0/0/4 10.192.254.15/31
lsw1-a3-codfw = ge-0/0/47 10.192.254.14/31

@Jhancock.wm thank you for the update. WE can resolve this task for now if it does happen again we can reopen.

We are seeing the same error on lswtest-d8 in eqiad

in-error-packets 2466

Ticket 05304338 has been submitted with Nokia

I took a quick look at this before getting the support ticket going on.
On lsw1-e2-codfw we have

Frame length statistics for mgmt0
----------------------------------------------------------------------------------------------------------------------
  Frame length(Octets)     Rx        Tx  
  64 bytes               5002621   2685  
  65-127 bytes           2263275   233924
  128-255 bytes          2256117   466154
  256-511 bytes          1340118   541089
  512-1023 bytes         63809     94005 
  1024-1518 bytes        1178      739944
  1519+ bytes            0         0

on lsw1-c2-eqaid we have

Frame length statistics for mgmt0
--------------------------------------------------------------------------------------------------------
  Frame length(Octets)   Rx   Tx
  64 bytes               0    0 
  65-127 bytes           0    0 
  128-255 bytes          0    0 
  256-511 bytes          0    0 
  512-1023 bytes         0    0 
  1024-1518 bytes        0    0 
  1519+ bytes            0    0

IS it possible to upgrade one of the eqiad Nokia switch to version 25 and see if we do have the same issue also.
Side node:
arp on lsw1-e2-codfw is seeing both cr1 and cr2 re0and re1 or on lsw1-c2-eqiad it is seeing only cr2

|
|
|
arp
|
|neighbor 10.193.0.10 {
                    link-layer-address XXXX
                    origin dynamic
                    expiration-time "2025-12-17T08:44:13.553Z (3 hours from now)"
                    datapath-programming {
                        status success
                    }
                }
                neighbor 10.193.0.11 {
                    link-layer-address XXXX
                    origin dynamic
                    expiration-time "2025-12-17T08:44:06.973Z (3 hours from now)"
                    datapath-programming {
                        status success
                    }
                }
                neighbor 10.193.0.12 {
                    link-layer-address XXXX
                    origin dynamic
                    expiration-time "2025-12-17T08:44:12.890Z (3 hours from now)"
                    datapath-programming {
                        status success
                    }
                }
                neighbor 10.193.0.13 {
                    link-layer-address XXXXX
                    origin dynamic
                    expiration-time "2025-12-17T08:44:09.799Z (3 hours from now)"
                    datapath-programming {
                        status success
                    }

@ayounsi assigned back to you since you are working on it. thanks

@ayounsi what else needs to be done here?

@ssingh yes we have to depool the site, yes 10 AM CT

@ssingh We are planning on doing the first phase(loopback IP change on core routers and management router) of the ULSFO refresh next week Dec 09th at 10:00am. Please let me know if this work for you an your team.

@ayounsi @cmooney thanks for the feedback.

@ayounsi @cmooney please see below the steps to replace the loopback IPs on cr3/4-ulsfo and mr1-ulsfo If all this looks good, I will setup a maintenance with traffic for December 9th at 11am CT. Thanks

@RobH I update the task description with all the connections that we need for phase 1 in December. Please don't forget the Cable ID's. Please let me know if you have any questions. Thanks

@ayounsi sretest1005 is the same as 2004 see below. what you can maybe check is the redfish /IDRAC version on sretest2004 and 1005

@ayounsi for the feed back i will work on it

I think a am wrong on the public vlan for rack 22. We will not be re-imaging the servers in that rack with public vlan just changing the network mask from /28 to /27

Both switches in drmrs are now running Junos: 23.4R2-S5.8. @cmooney i am sending the task to you since you wanted to do the cloud switches.

@ayounsi Please see below the steps to disable LLDP in the BIOS for Dell servers.

@ayounsi @cmooney on the other QFX5120-48Y in magru we are running version 22.2R3.S3.18 or right now the recommande version for that model is 23.4R2-S5. Do you want me to do 23.4R2-S5 or stick to 22.2R3-S7?

I took a look at xe-1/0/8 as you mentioned it was cp5002 and i saw dns5004 and just to realized that this task has been open since 2020 5 years ago so now on port xe-1/0/8 we have dns5004.

papaul@asw1-eqsin> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
[----]
xe-1/0/8           -                   84:16:0c:5d:9c:70   NIC 1/10/25Gb SFP+ DA Broadcom Adv. Dual 25Gb Ethernet fw_version:AFW_218.0.219.9
[---]

papaul@asw1-eqsin> show lldp neighbors interface xe-1/0/8  
LLDP Neighbor Information:
Local Information:
Index: 734 Time to live: 120 Time mark: Mon Nov 17 21:42:59 2025 Age: 7 secs 
Local Interface    : xe-1/0/8
Parent Interface   : -
Local Port ID      : 559
Ageout Count       : 0

@cmooney @ayouns I update the task with all the IPV4 and IPV6 addresses for the links, irb's and loopbacks. Please review and let me know if there is anything I need to change or add.

@ayounsi yes I can look into it. Thanks.

After swapping both PEM 2 and 3

re0.cr1-codfw> show chassis environment pem    
PEM 0 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        58          1             58       2      
PEM 1 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        58          32            1856     90     
PEM 2 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        58          0             0        0      
PEM 3 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        58          2             116      5

re0.cr2-codfw> show chassis environment pem    
PEM 0 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        59          0             0        0      
PEM 1 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        60          13            780      38     
PEM 2 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        57          0             0        0      
PEM 3 status:
  State                      Online
  Temperature                             OK                                      
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                        55          0             0        0

@ssingh @Vgutierrez planning on doing this on Nov 19th @10:am CT. Thank you

Bother firewalls are not running Junos: 23.4R2-S5.5. Thanks to @Jgreen and @Dwisehaupt.
Closing this task now

@cmooney thanks for the feedback we can clarify this tomorrow during the meeting and have all ready and run it by @ayounsi when he is back.

@cmooney i update all the IP's to match the other POP sites. I will be re-running the configuration and validation sometimes this week in my lab and post back the result. I update also the irb interfaces configuation. I will update also the ip addresses of the link to eqsin and codfw later in the description.

@Dwisehaupt yes Wednesday 11/5 is ok with me. Let us do 10:00am CT. Thank you.

@Dwisehaupt hello yes we can do this during the maintenance windows in November. Any day you prefer for that week? Thank you

We still have an ongoing email section going on with Juniper on this to understanding why in Eqiad the power is balance on all PEM's and not in codfw. Please see below for the last update we had from Juniper. Thanks.

@cmooney thanks for the feedback, I will upgrade the diagram to match the 100G links between the core routers and the switches and the type of transceivers needed.

@elukey no problem

@ssingh thanks for the update. I am planning on doing it before Thanksgiving any day during the week of November 17th works for me. Let me know if that woks for you and I can get back with you on the exact day and time.

While trying to use the firmware upgrade cookbook with "sudo cookbook sre.hardware.upgrade-firmware ms-be2078 --new" i get the error below so i have to to run the cookbook by passing the flag for each component
"sudo cookbook sre.hardware.upgrade-firmware ms-be2078 -c bios --new " this works only for the BIOS and when doing the same for the IDRAC i get the second error below.
Is it possible please to look into the code and see why this is failing? In the main time i was able to manually upgrade the IDRAC. Thanks

@elukey i think the next step will be to try to install the OS without setting up the boot disk and let the OS take care of it. maybe this is one of the many cases where it is not possible to set out the boot disk before the OS install
Thanks.

@elukey on can you please provide me with one of the node that is working like you said so i can check what is different from this node and the one that is not working?

@elukey @MatthewVernon thank you that was very helpful information. Now I can answer you question
"In UEFI Boot Mode, fixed media (see Hard Disk items in the earlier section) may or may not be added to the
boot sequence. Unlike legacy Boot Mode, in UEFI Boot Mode, the OS has the ability to add to and modify the
boot sequence"

@ssingh @Vgutierrez hello just checking in to see if you have a day and time for this for drmrs.
Thanks

can you please provide me with some context here on what we are trying to do, The only thing I see in the task is we are testing UEFI mode on the node.
1- Are we moving from Debain 11 to Debian 12?
2- What partman recipe are we using for testing?