Page MenuHomePhabricator
Create Task
Maniphest T365993

Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad
Closed, ResolvedPublic
Actions

Description

Postponed until Wed 2024-07-10, 15:00 UTC
on this rack: https://netbox.wikimedia.org/dcim/racks/81/

  • backup1010
  • db1190 - s4
  • dbproxy1026 - m3
  • ms-be1068
  • ms-fe1012
  • an-coord1003
  • an-mariadb1001
  • an-presto1006
  • an-presto1007
  • an-worker1147
  • an-worker1153
  • an-worker1142
  • cephosd1001
  • dumpsdata1006
  • kafka-jumbo1010
  • stat1010
  • elastic1104
  • elastic1089
  • elastic1090
  • dse-k8s-worker1005
  • ml-cache1001
  • logstash1036
  • kubernetes1059
  • lvs1013
  • lvs1014
  • lvs1015

Teams Involved: Data Persistence, Data Platform, Machine Learning, Search, Observability, Service Ops, Traffic

Expected outage: 15-30 minutes

Please use the below sheet to detail any actions that are required in advance of the work:

https://docs.google.com/spreadsheets/d/1pLPpzGBmdExXxQ_0_eGXpO0VlUU5oPKZy-_KViMSwuM

Details

Other Assignee
MatthewVernon
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fail over hive and presto services to the standby coordinatoroperations/dnsmaster+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

ABran-WMF created this task.May 27 2024, 1:07 PM
ABran-WMF updated Other Assignee, added: MatthewVernon.
ABran-WMF mentioned this in T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2.May 27 2024, 1:12 PM
ABran-WMF added projects: Data-Persistence, DBA.May 27 2024, 3:02 PM
ABran-WMF moved this task from Triage to Ready on the DBA board.May 28 2024, 7:48 AM
ABran-WMF added a project: SRE-swift-storage.May 28 2024, 10:23 AM
MatthewVernon updated the task description. (Show Details)May 28 2024, 10:40 AM
MatthewVernon subscribed.

I'm on annual leave this day, so someone else will have to handle the ms frontend, which needs depooling beforehand and repooling afterwards. Once we know the planned time, we can find someone suitable.

cmooney claimed this task.Jun 5 2024, 7:05 PM
cmooney triaged this task as Medium priority.
cmooney updated the task description. (Show Details)
cmooney updated the task description. (Show Details)
MatthewVernon added a subscriber: Eevans.Jun 6 2024, 12:42 PM

@Eevans would you be OK to handle this as well, please? It's a bit more involved as you'll need to run sudo depool on the ms-fe node beforehand (and then sudo pool afterwards) as well as a quick cluster health check for the backend node.

Eevans added a comment.Jun 6 2024, 4:33 PM
In T365993#9867211, @MatthewVernon wrote:

@Eevans would you be OK to handle this as well, please? It's a bit more involved as you'll need to run sudo depool on the ms-fe node beforehand (and then sudo pool afterwards) as well as a quick cluster health check for the backend node.

Sure.

jcrespo subscribed.Jun 12 2024, 9:28 AM

backup1010 is in intermittent usage to support mediabackups disk space, but mostly idle at the time, so unless it situtation changes by july and finally gets pooled for bacula, it will require no action.

ABran-WMF updated the task description. (Show Details)Jun 18 2024, 8:09 AM
jcrespo added a comment.Jul 1 2024, 1:51 PM

No action will be needed for backup1010 in the end.

cmooney updated the task description. (Show Details)Jul 2 2024, 1:59 PM
ayounsi moved this task from Backlog to This quarter on the netops board.Jul 8 2024, 7:24 AM
akosiaris subscribed.Jul 8 2024, 4:35 PM
gerritbot added a comment.Jul 9 2024, 12:17 PM

Change #1052965 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/dns@master] Fail over hive and presto services to the standby coordinator

https://gerrit.wikimedia.org/r/1052965

gerritbot added a project: Patch-For-Review.Jul 9 2024, 12:17 PM

Change #1052965 merged by Btullis:

[operations/dns@master] Fail over hive and presto services to the standby coordinator

https://gerrit.wikimedia.org/r/1052965

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2024, 7:59 PM
Stashbot added a comment.Jul 10 2024, 2:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:02:24Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T365993 - depool db1190 - s4', diff saved to https://phabricator.wikimedia.org/P66129 and previous config saved to /var/cache/conftool/dbconfig/20240710-140224-arnaudb.json

Stashbot added a comment.Jul 10 2024, 2:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:02:56Z] <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on backup1010.eqiad.wmnet,db1190.eqiad.wmnet,dbproxy1026.eqiad.wmnet with reason: T365993

Stashbot added a comment.Jul 10 2024, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:03:11Z] <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1010.eqiad.wmnet,db1190.eqiad.wmnet,dbproxy1026.eqiad.wmnet with reason: T365993

ops-monitoring-bot added a comment.Jul 10 2024, 2:08 PM

Icinga downtime and Alertmanager silence (ID=9ca0faf1-4b9d-4345-9bb8-9c7153e17163) set by cmooney@cumin1002 for 1:30:00 on 1 host(s) and their services with reason: prep JunOS upgrade lsw1-e1-eqiad

lsw1-e1-eqiad.mgmt
Stashbot added a comment.Jul 10 2024, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:34:10Z] <cmooney@cumin1002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1104,elastic1089,elastic1090 for ban elastic nodes before switch upgrade rack E1 - cmooney@cumin1002 - T365993

Stashbot added a comment.Jul 10 2024, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:34:13Z] <cmooney@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic1104,elastic1089,elastic1090 for ban elastic nodes before switch upgrade rack E1 - cmooney@cumin1002 - T365993

Stashbot added a comment.Jul 10 2024, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:55:46Z] <cmooney@cumin1002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1104*,elastic1089*,elastic1090* for T365993 - cmooney@cumin1002

Stashbot added a comment.Jul 10 2024, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T14:56:01Z] <cmooney@cumin1002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1104*,elastic1089*,elastic1090* for T365993 - cmooney@cumin1002

ops-monitoring-bot added a comment.Jul 10 2024, 3:23 PM

Icinga downtime and Alertmanager silence (ID=5386f05e-734c-49b0-a4c5-1acbef4c187a) set by cmooney@cumin1002 for 0:30:00 on 4 host(s) and their services with reason: JunOS upgrade lsw1-e1-eqiad

lsw1-e1-eqiad,lsw1-e1-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt
ops-monitoring-bot added a comment.Jul 10 2024, 3:24 PM

Icinga downtime and Alertmanager silence (ID=9475b2b6-bc5f-41f8-97d1-970eb62b38bc) set by cmooney@cumin1002 for 0:30:00 on 26 host(s) and their services with reason: JunOS upgrade lsw1-e1-eqiad

an-coord1003.eqiad.wmnet,an-mariadb1001.eqiad.wmnet,an-presto[1006-1007].eqiad.wmnet,an-worker[1142,1147,1153].eqiad.wmnet,backup1010.eqiad.wmnet,cephosd1001.eqiad.wmnet,db1190.eqiad.wmnet,dbproxy1026.eqiad.wmnet,dse-k8s-worker1005.eqiad.wmnet,dumpsdata1006.eqiad.wmnet,elastic[1089-1090,1104].eqiad.wmnet,kafka-jumbo1010.eqiad.wmnet,kubernetes1059.eqiad.wmnet,logstash1036.eqiad.wmnet,lvs[1013-1015].eqiad.wmnet,ml-cache1001.eqiad.wmnet,ms-be1068.eqiad.wmnet,ms-fe1012.eqiad.wmnet,stat1010.eqiad.wmnet
Stashbot added a comment.Jul 10 2024, 3:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T15:24:16Z] <topranks> rebooting lsw1-e1-eqiad to install updated JunOS version T365993

cmooney added a comment.Jul 10 2024, 3:45 PM

Switch upgraded successfully and all hosts back online/pinging. Thanks everyone for the assistance!

Stashbot added a comment.Jul 10 2024, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T15:46:15Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 5%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66146 and previous config saved to /var/cache/conftool/dbconfig/20240710-154615-arnaudb.json

ABran-WMF added a comment.Jul 10 2024, 3:46 PM

db1190 repooling
dbproxy reloaded

everything looks OK

Eevans added a comment.Jul 10 2024, 3:52 PM

ms-fe1012 repooled, and everything looks good.

Stashbot added a comment.Jul 10 2024, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T16:01:31Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 10%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66149 and previous config saved to /var/cache/conftool/dbconfig/20240710-160120-arnaudb.json

Stashbot added a comment.Jul 10 2024, 4:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T16:16:26Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 25%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66152 and previous config saved to /var/cache/conftool/dbconfig/20240710-161626-arnaudb.json

BTullis mentioned this in T369737: Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic.Jul 10 2024, 4:21 PM
Stashbot added a comment.Jul 10 2024, 4:31 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T16:31:32Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 50%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66157 and previous config saved to /var/cache/conftool/dbconfig/20240710-163131-arnaudb.json

Stashbot added a comment.Jul 10 2024, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T16:46:38Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 75%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66160 and previous config saved to /var/cache/conftool/dbconfig/20240710-164637-arnaudb.json

Stashbot added a comment.Jul 10 2024, 5:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-10T17:01:43Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1190 (re)pooling @ 100%: post T365993 repool', diff saved to https://phabricator.wikimedia.org/P66162 and previous config saved to /var/cache/conftool/dbconfig/20240710-170143-arnaudb.json

cmooney closed this task as Resolved.Jul 11 2024, 10:18 AM
Stashbot added a comment.Sep 6 2024, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-06T14:28:16Z] <akosiaris> repool kubernetes1059 T365993

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits