Both Junos 22.2R3-Sx and Junos 22.4R3 are latest recommended. fyi, I went with 22.4R3 in magru.

Mentioned in SAL (#wikimedia-operations) [2024-06-17T15:49:59Z] <topranks> rebooting cr2-eqdfw to upgrade JunOS T364092

In T364092#9766653, @ayounsi wrote:

Both Junos 22.2R3-Sx and Junos 22.4R3 are latest recommended. fyi, I went with 22.4R3 in magru.

Doh, I went with 22.2R3 on cr2-eqdfw should have checked this previously. We can use 22.4R3 on the rest and revisit eqdfw if we think it's worth it.

There has been a spike of CPU usage on cr1-eqiad (with no impact), not sure if just a coincidence.

A few more reasons to upgrade in {T376986}.

There will be some maintenance in magru sometime next week and the site will be de-pool we can take advantage of this maintenance window to upgrade the router there from 22.4R3 to 23.4R2

cr1-eqiad is stated for Nov 13 but note that T376737 is also scheduled for that period (Nov 13, 8 CT) and it might make tricky for both magru and eqiad to be depooled. Can we please move this to the next day perhaps, or even better, give a buffer of two days?

I see there is a maintenance planned for codfw now, and that the plan is to depool the datacenter. Does this mean we're doing a datacenter switchover?

Because otherwise, we can't really depool codfw from being in the path of a request. We can of course just do the upgrade, and then do a switchover if we lose redundancy, but we should be aware of the risk.

@ssingh thanks i forgot about the 13th I update the dates.

Upgrades should follow the standard process

The standard process docs are outdated I fear.

Depool site (optional)
(optional) if codfw, drain mw traffic sudo cookbook sre.mediawiki.route-traffic primary

codfw will be the primary during that set of dates, it should NOT be depooled.

In T364092#10296937, @Joe wrote:

I see there is a maintenance planned for codfw now, and that the plan is to depool the datacenter. Does this mean we're doing a datacenter switchover?

Because otherwise, we can't really depool codfw from being in the path of a request. We can of course just do the upgrade, and then do a switchover if we lose redundancy, but we should be aware of the risk.

@Joe thank you for letting us know.

Thanks @akosiaris @Joe we can hold back on codfw for now and work on eqiad. when we switch back to eqiad we can schedule the upgrade for codfw.

In T364092#10296980, @Papaul wrote:

Thanks @akosiaris @Joe we can hold back on codfw for now and work on eqiad. when we switch back to eqiad we can schedule the upgrade for codfw.

Makes sense to me. Thanks for accommodating us. For what is worth, we are moving back to eqiad on Wednesday, 19 March 2025

In T364092#10296958, @akosiaris wrote:

codfw will be the primary during that set of dates, it should NOT be depooled.

Agreed. It should also be possible for us to do the core router upgrade without depooling the site, but it makes the steps a little trickier and adds some risk.

So overall agree it's best to wait until we switch back to eqiad, I don't think the urgency of the router upgrades requires us to do it before then.

In T364092#10296958, @akosiaris wrote:

Upgrades should follow the standard process

The standard process docs are outdated I fear.

Depool site (optional)
(optional) if codfw, drain mw traffic sudo cookbook sre.mediawiki.route-traffic primary

codfw will be the primary during that set of dates, it should NOT be depooled.

Wiki updated.

Makes sense to me. Thanks for accommodating us. For what is worth, we are moving back to eqiad on Wednesday, 19 March 2025

Is there a tracking task or is it too soon :) We could then add it as sub-task.

It's indeed not an emergency so we can easily wait for the next switchover.

@Papaul I added a 8th point in the "Cleanup" section of https://wikitech.wikimedia.org/wiki/Juniper_router_upgrade#Cleanup
I noticed that some interfaces were missing from LibreNMS billing feature, and I suspect it's because they got their internal ID changed with the latest upgrades. Something to keep an eye on.

@ayounsi thanks for the information

Mentioned in SAL (#wikimedia-operations) [2024-11-14T15:36:05Z] <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqiad [reason: junos upgrade, T364092]

Mentioned in SAL (#wikimedia-operations) [2024-11-14T15:36:21Z] <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqiad [reason: junos upgrade, T364092]

Mentioned in SAL (#wikimedia-operations) [2024-11-14T19:37:25Z] <sukhe@cumin1002> START - Cookbook sre.dns.admin DNS admin: pool site eqiad [reason: junos upgrade done, T364092]

Mentioned in SAL (#wikimedia-operations) [2024-11-14T19:37:29Z] <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqiad [reason: junos upgrade done, T364092]