Page MenuHomePhabricator
Create Task
Maniphest T348977

Upgrade EVPN switches Eqiad row E-F to JunOS 22.2
Closed, ResolvedPublic
Actions

Description

Currently we are running JunOS 20.4R3 on our EVPN switches in Eqiad rows E and F.

There are several improvements and bugfixes in more recent releases that we need to upgrade to overcome, detailed in T306421, T358488 and T365204, as well as some security fixes. So we need to upgrade the devices in these rows over the next while.

We are using the below master gsheet to co-ordinate with other SRE teams on depool and similar actions that are needed to enable this work:

https://docs.google.com/spreadsheets/d/1pLPpzGBmdExXxQ_0_eGXpO0VlUU5oPKZy-_KViMSwuM

Current proposed schedule as follows:

DeviceDateTasksTeamsNotes
ssw1-e1-eqiadThur Jun 5thT366361Traffic
ssw1-f1-eqiadThur Jun 5thT366361Traffic
lsw1-f5-eqiadTue Jun 11thT365982Data Platform, Search, Data Persistence
lsw1-f6-eqiadThur Jun 13thT365983Data Platform, Data Persistence
lsw1-f7-eqiadTue Jun 18thT365984Data Platform, Data Persistence
lsw1-e6-eqiadTue Jun 20thT365987Data Platform, Search, Data Persistenceswapped with T365986
lsw1-e5-eqiadThur Jun 25thT365986Data Platform, Search, Data Persistence, Service Opsswapped with T365987
lsw1-e7-eqiadThur Jun 27thT365988Data Platform, Search, Data Persistence
lsw1-e2-eqiadWed Jul 3rdT365994Data Platform, Search, Data Persistence, Machine Learning, Observability, Service Ops
lsw1-e3-eqiadTue Jul 9thT365995Data Platform, Search, Data Persistence, Machine Learning, Observability, Service Ops
lsw1-e1-eqiadWed Jul 10thT365993Data Platform, Search, Data Persistence, Machine Learning, Observability, Service OpsPostponed from Jul 2nd
lsw1-f1-eqiadThu Jul 11thT365996Data Platform, Search, Data Persistence, Machine Learning, Observability
lsw1-f2-eqiadTue Jul 16thT365997Data Platform, Search, Data Persistence, Machine Learning, Observability, Service Ops
lsw1-f3-eqiadTue Jul 23rdT365998Data Platform, Search, Data Persistence, Machine Learning, Service Ops

Details

SubjectRepoBranchLines +/-
Facilitate a role swap from an-mariadb1001 to an-mariadb1002operations/deployment-chartsmaster+6 -6
Configure analytics_meta MariaDB clients to connect to an-mariadb1002operations/puppetproduction+7 -4
Change IP for new Eqiad switches to MGMT IPoperations/puppetproduction+6 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
cmooney added a comment.Dec 20 2023, 6:25 PM

it seems this limitation does not apply to 22.2 which we are using in codfw.

An update on this. It seems that we do have this bug in 22.2, but we don't always trigger it which is why I thought it had been fixed.

On a new switch, which has never had an IRB interface configured, pinging between the lo0.5000 interfaces in the routing-instance (and thus VXLAN encap) works just fine. If you subsequently configure an IRB interface on the box, without an UP port in the associated vlan (thus irb int is state up/down), pings between the loopback begin to fail. Further if you remove the IRB interface and vlan completely, once this state has been entered, pings continue to fail.

It is possible this is the exact same in 20.4 but we just didn't notice that things are ok if an IRB has never been configured on the box.

Anyway we should still upgrade, but it looks like we are stuck with this annoying bug.

cmooney updated the task description. (Show Details)Dec 20 2023, 6:35 PM
cmooney added a comment.EditedJan 25 2024, 3:50 PM

Just an update here, the restriction still exists however I think I know how I went wrong.

In order for the irb interface to be "up" the associated vlan needs to also be "up". However the vlan will come up if there are any MAC addresses in it's table it seems.

Specifically in the VXLAN/EVPN context it means the irb interface will be in state "up" if it belongs to a stretched-vlan, and any other port in that vlan is up across the network. This is probably something we can take advantage of to ensure we've an "up" irb on otherwise empty switches, and ensure we can ping/monitor the VRF loopbacks.

cmooney@lsw1-b5-codfw> show interfaces descriptions | match irb.2002 
irb.2002        up    up   Subnet public1-b-codfw

No local ports are up but it knows remote MACs:

cmooney@lsw1-b5-codfw> show ethernet-switching table vlan-id 2002 

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 17 entries, 17 learned
Routing instance : default-switch
   Vlan                MAC                 MAC      Logical                SVLBNH/      Active
   name                address             flags    interface              VENH Index   source
   public1-b-codfw     14:23:f2:4d:cd:60   DR       vtep.32772                          10.192.252.11                 
   public1-b-codfw     14:23:f2:4e:fc:00   DR       vtep.32771                          10.192.252.4                  
   public1-b-codfw     14:23:f2:4f:0d:f1   DR       vtep.32769                          10.192.252.2                  
   public1-b-codfw     14:23:f2:4f:30:21   DR       vtep.32770                          10.192.252.1                  
   public1-b-codfw     64:87:88:f2:6d:b0   DR       vtep.32770                          10.192.252.1                  
   public1-b-codfw     a8:d0:e5:e3:81:b0   DR       vtep.32769                          10.192.252.2                  
   public1-b-codfw     aa:00:00:0e:a4:81   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:26:f2:33   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:31:50:4c   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:53:f8:01   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:71:23:8b   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:99:be:5a   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     aa:00:00:d2:18:cf   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     b0:4f:13:b9:89:6c   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     b0:4f:13:bb:e6:08   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     b0:4f:13:bc:e8:c2   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02 
   public1-b-codfw     c4:5a:b1:a3:b6:f7   DR       esi.1827               1826         00:00:00:00:01:02:00:00:00:02
cmooney raised the priority of this task from Low to Medium.Feb 26 2024, 7:23 PM

Actually a different need to upgrade has now become clear, relating to the issue detailed in T358488

The solution to that requires the switch inserting sub-option 5 into the DHCP option 82 information, with the forwarding-options dhcp-relay relay-option-82 link-selection statement, however this was only introduced in 21.2R1.

Right now we've no public vlan stretched in eqiad so there is no massive urgency, but another reason to get it done.

cmooney mentioned this in T365204: Problem re-imaging hosts on row-wide vlan on EVPN switches.May 20 2024, 1:55 PM
cmooney added a comment.EditedMay 23 2024, 9:28 AM

We also now have the issue from T365204 that we can resolve with an upgrade of JunOS. Not essential in eqiad but still I think we need to stop procrastinating here.

I'd propose to tackle the devices in the following order, starting with the Spines as they are route-reflectors, so I think it's best they not be behind any of the clients in terms of OS release. Also to do those we only need to co-ordinate with Traffic (for the LVS links that land there), rather than multiple SRE teams. After that tackle the racks that most recently came online, as they have only a few devices, but devices are being added all the time so best do them when they're as empty as possible.

<moved to task description>

cmooney updated the task description. (Show Details)May 23 2024, 3:19 PM
cmooney updated the task description. (Show Details)May 23 2024, 4:26 PM
ABran-WMF updated the task description. (Show Details)May 27 2024, 11:47 AM
ABran-WMF updated the task description. (Show Details)
ABran-WMF updated the task description. (Show Details)May 27 2024, 11:51 AM
ABran-WMF updated the task description. (Show Details)May 27 2024, 11:58 AM
ABran-WMF updated the task description. (Show Details)May 27 2024, 12:00 PM
ABran-WMF updated the task description. (Show Details)May 27 2024, 1:12 PM
ABran-WMF updated the task description. (Show Details)May 27 2024, 1:23 PM
MatthewVernon subscribed.May 28 2024, 10:18 AM

@cmooney can you give me a time estimate for when you're going to be doing these, please? I'd like to put notes in my calendar.

cmooney added a subscriber: ABran-WMF.May 31 2024, 2:19 PM

@ABran-WMF thanks for creating all the tasks! Really appreciated, I did not expect to come back and see that :)

In T348977#9837047, @MatthewVernon wrote:

@cmooney can you give me a time estimate for when you're going to be doing these, please? I'd like to put notes in my calendar.

Hi @MatthewVernon sorry for the late response I was on leave earlier in the week.

Thinking about it I'd probably aim to start the upgrade at 15:00 UTC (so 4pm our time UK/Ireland) to maximise coverage in terms of people on both sides of the Atlantic. I can add entries in the SRE maintenance calendar for them, but if that time doesn't work for you let me know.

cmooney added a comment.May 31 2024, 2:27 PM
This comment was removed by cmooney.
cmooney added a subtask: T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3.May 31 2024, 3:58 PM
cmooney updated the task description. (Show Details)
cmooney updated the task description. (Show Details)May 31 2024, 4:31 PM
cmooney updated the task description. (Show Details)Jun 4 2024, 6:19 PM
cmooney closed subtask T366361: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 as Resolved.Jun 6 2024, 5:50 PM
cmooney updated the task description. (Show Details)Jun 10 2024, 12:33 PM
bking subscribed.Jun 10 2024, 1:30 PM
Stashbot added a comment.Jun 10 2024, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-10T13:57:30Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1107 for T348977 - bking@cumin2002

Stashbot added a comment.Jun 10 2024, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-10T13:57:34Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic1107 for T348977 - bking@cumin2002

Stashbot added a comment.Jun 10 2024, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-10T13:57:47Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1107.eqiad.wmnet for T348977 - bking@cumin2002

Stashbot added a comment.Jun 10 2024, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-10T13:57:50Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1107.eqiad.wmnet for T348977 - bking@cumin2002

cmooney closed subtask T365982: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad as Resolved.Jun 13 2024, 1:01 PM
cmooney closed subtask T365983: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad as Resolved.Jun 13 2024, 3:55 PM
ABran-WMF updated the task description. (Show Details)Jun 20 2024, 8:30 AM
cmooney closed subtask T365984: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad as Resolved.Jun 20 2024, 11:40 AM
Stashbot added a comment.Jun 20 2024, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T19:18:45Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1105 for T348977 - bking@cumin2002

Stashbot added a comment.Jun 20 2024, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T19:18:50Z] <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic1105 for T348977 - bking@cumin2002

Stashbot added a comment.Jun 20 2024, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T19:18:54Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1105* for T348977 - bking@cumin2002

Stashbot added a comment.Jun 20 2024, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T19:18:58Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1105* for T348977 - bking@cumin2002

Stashbot added a comment.Jun 20 2024, 8:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T20:53:02Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on elastic1105.eqiad.wmnet with reason: T348977

Stashbot added a comment.Jun 20 2024, 8:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T20:53:17Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on elastic1105.eqiad.wmnet with reason: T348977

cmooney closed subtask T365987: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad as Resolved.Jun 21 2024, 10:41 PM
cmooney closed subtask T365986: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad as Resolved.Jun 25 2024, 4:36 PM
cmooney closed subtask T365988: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad as Resolved.Jun 28 2024, 9:11 AM
Stashbot added a comment.Jul 1 2024, 9:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-01T21:58:43Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1089-1090,1104].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 1 2024, 9:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-01T21:58:50Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1089-1090,1104].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 1 2024, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-01T21:59:17Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1089*,elastic1090*,elastic1104* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 1 2024, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-01T21:59:20Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1089*,elastic1090*,elastic1104* for T348977 - bking@cumin2002

BTullis mentioned this in T365503: Upgrade mariadb on analytics_meta from 10.4 to 10.6.Jul 2 2024, 10:42 AM
cmooney updated the task description. (Show Details)Jul 2 2024, 1:58 PM
Stashbot added a comment.Jul 3 2024, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-03T13:56:45Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1091*,elastic1092* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 3 2024, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-03T13:56:48Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1091*,elastic1092* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 3 2024, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-03T13:59:12Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1091-1092].eqiad.wmnet,wdqs[1018,1020].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 3 2024, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-03T13:59:21Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1091-1092].eqiad.wmnet,wdqs[1018,1020].eqiad.wmnet with reason: T348977

cmooney closed subtask T365994: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad as Resolved.Jul 3 2024, 5:14 PM
• ayounsi moved this task from Backlog to This quarter on the netops board.Jul 8 2024, 7:17 AM
cmooney updated the task description. (Show Details)Jul 8 2024, 4:06 PM
Stashbot added a comment.Jul 8 2024, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-08T21:05:17Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1093-1095].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 8 2024, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-08T21:05:34Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1093-1095].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 8 2024, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-08T21:05:46Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic109[3-5]* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 8 2024, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-08T21:05:57Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic109[3-5]* for T348977 - bking@cumin2002

gerritbot added a comment.Jul 9 2024, 9:40 AM

Change #1052932 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure analytics_meta MariaDB clients to connect to an-mariadb1002

https://gerrit.wikimedia.org/r/1052932

gerritbot added a project: Patch-For-Review.Jul 9 2024, 9:40 AM
gerritbot added a comment.Jul 9 2024, 10:03 AM

Change #1052937 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Facilitate a role swap from an-mariadb1001 to an-mariadb1002

https://gerrit.wikimedia.org/r/1052937

Stashbot added a comment.Jul 9 2024, 12:19 PM

Mentioned in SAL (#wikimedia-analytics) [2024-07-09T12:19:41Z] <btullis> failing over hive and presto to the standby coordinator for T348977

gerritbot added a comment.Jul 10 2024, 10:20 AM

Change #1052932 merged by Btullis:

[operations/puppet@production] Configure analytics_meta MariaDB clients to connect to an-mariadb1002

https://gerrit.wikimedia.org/r/1052932

gerritbot added a comment.Jul 10 2024, 10:27 AM

Change #1052937 merged by Btullis:

[operations/deployment-charts@master] Facilitate a role swap from an-mariadb1001 to an-mariadb1002

https://gerrit.wikimedia.org/r/1052937

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2024, 10:30 AM
cmooney closed subtask T365995: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad as Resolved.Jul 10 2024, 12:39 PM
Stashbot added a comment.Jul 10 2024, 9:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T21:03:53Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1096-1097,1106].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 10 2024, 9:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T21:04:11Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1096-1097,1106].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 10 2024, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T21:10:26Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1096*,elastic1097*,elastic1106* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 10 2024, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T21:10:29Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1096*,elastic1097*,elastic1106* for T348977 - bking@cumin2002

cmooney closed subtask T365993: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad as Resolved.Jul 11 2024, 10:18 AM
cmooney closed subtask T365996: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad as Resolved.Jul 12 2024, 11:37 AM
Stashbot added a comment.Jul 15 2024, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-15T19:23:51Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1098-1099].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 15 2024, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-15T19:24:07Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1098-1099].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 15 2024, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-15T19:24:52Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic109[8-9]* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 15 2024, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-15T19:24:55Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic109[8-9]* for T348977 - bking@cumin2002

cmooney closed subtask T365997: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad as Resolved.Jul 17 2024, 11:08 AM
Stashbot added a comment.Jul 18 2024, 12:35 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-18T00:35:01Z] <ryankemper@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic110[0-2]* for row maint - ryankemper@cumin2002 - T348977

Stashbot added a comment.Jul 18 2024, 12:35 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-18T00:35:07Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic110[0-2]* for row maint - ryankemper@cumin2002 - T348977

Stashbot added a comment.Jul 22 2024, 10:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-22T22:34:52Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 22 2024, 10:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-22T22:35:10Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T348977

Stashbot added a comment.Jul 22 2024, 10:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-22T22:36:53Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic110[0-2]* for T348977 - bking@cumin2002

Stashbot added a comment.Jul 22 2024, 10:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-22T22:37:02Z] <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic110[0-2]* for T348977 - bking@cumin2002

cmooney closed this task as Resolved.Jul 23 2024, 4:23 PM
cmooney closed subtask T365998: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits