Page MenuHomePhabricator
Create Task
Maniphest T363214

kafka-main100[6789] and kafka-main1010 implementation tracking
Closed, ResolvedPublic
Actions

Description

This task has been created by DC Ops for serviceops implementation tracking (per serviceops request when filing racking tasks.)

Once racking task T363212 has been completed, this task can be taken by service ops for implementation. Please note this task is not monitored by DC ops and any questions should be directed to the racking task.

The process for replacing brokers is described in https://wikitech.wikimedia.org/wiki/Kafka/Administration#Hardware_replace_a_broker

Before replacing the first broker here, give traffic a headsup so they can monitor for high latency of cache purges (purged). We saw that happening during T363210: kafka-main200[6789] and kafka-main2010 implementation tracking but I think this is not an issue anymore with the transfer.py approach.

  • kafka-main1006
  • kafka-main1007
  • kafka-main1008
  • kafka-main1009
  • kafka-main1010

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Apr 23 2024, 7:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2024, 7:32 PM
RobH unsubscribed.Apr 23 2024, 7:32 PM
JMeybohm claimed this task.Jun 28 2024, 9:40 AM
JMeybohm changed the task status from Open to Stalled.Jun 28 2024, 11:36 AM
JMeybohm mentioned this in T368714: kafka-main replacement nodes don't fit kafka-main (storage wise).
JMeybohm added a parent task: T368714: kafka-main replacement nodes don't fit kafka-main (storage wise).
akosiaris mentioned this in Unknown Object (Task).Jul 9 2024, 9:17 AM
akosiaris mentioned this in Unknown Object (Task).
JMeybohm added a parent task: T373189: Establish a proper process for replacing kafka nodes.Aug 23 2024, 12:46 PM
JMeybohm updated the task description. (Show Details)Sep 12 2024, 9:26 AM
JMeybohm updated the task description. (Show Details)Sep 13 2024, 9:43 AM
dcausse subscribed.Oct 4 2024, 8:40 AM
akosiaris edited subscribers, added: jijiki; removed: dcausse.Oct 17 2024, 12:55 PM
akosiaris added subscribers: dcausse, akosiaris.

Mistakenly removed @dcausse, re-adding.

jijiki triaged this task as Medium priority.Oct 17 2024, 1:00 PM
jijiki moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
JMeybohm reassigned this task from JMeybohm to jijiki.Oct 18 2024, 8:56 AM
JMeybohm subscribed.
jijiki changed the task status from Stalled to In Progress.Oct 18 2024, 12:04 PM
gerritbot added a comment.Nov 11 2024, 4:29 PM

Change #1089822 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] Add replacement kafka nodes to kafka_brokers_main on eqiad

https://gerrit.wikimedia.org/r/1089822

gerritbot added a project: Patch-For-Review.Nov 11 2024, 4:29 PM
gerritbot added a comment.Nov 19 2024, 10:57 AM

Change #1089822 merged by Effie Mouzeli:

[operations/puppet@production] Add replacement kafka nodes to kafka_brokers_main on eqiad

https://gerrit.wikimedia.org/r/1089822

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2024, 11:31 AM
Stashbot added a comment.Nov 20 2024, 10:22 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-20T10:22:46Z] <effie> removing leadership from kafka-main1001 - T363214

ops-monitoring-bot added a comment.Nov 20 2024, 10:33 AM

Icinga downtime and Alertmanager silence (ID=1cb13fea-2d29-4686-a824-95c9972431a0) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1001,1006].eqiad.wmnet
gerritbot added a comment.Nov 20 2024, 1:14 PM

Change #1093330 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kafka-main: Replace kafka-main1001 with kafka-main1006

https://gerrit.wikimedia.org/r/1093330

gerritbot added a project: Patch-For-Review.Nov 20 2024, 1:14 PM
gerritbot added a comment.Nov 20 2024, 1:19 PM

Change #1093330 merged by Effie Mouzeli:

[operations/puppet@production] kafka-main: Replace kafka-main1001 with kafka-main1006

https://gerrit.wikimedia.org/r/1093330

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2024, 1:31 PM
gerritbot added a comment.Nov 20 2024, 1:57 PM

Change #1093337 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1006

https://gerrit.wikimedia.org/r/1093337

gerritbot added a project: Patch-For-Review.Nov 20 2024, 1:57 PM
jijiki updated the task description. (Show Details)Nov 20 2024, 4:19 PM
gerritbot added a comment.Nov 20 2024, 4:19 PM

Change #1093337 merged by jenkins-bot:

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1006

https://gerrit.wikimedia.org/r/1093337

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2024, 4:30 PM
ops-monitoring-bot added a comment.Nov 21 2024, 2:43 PM

Icinga downtime and Alertmanager silence (ID=9f9d188a-551c-412a-8d68-ca67db96a150) set by jynus@cumin1002 for 7 days, 0:00:00 on 1 host(s) and their services with reason: Per claime's recommendation

kafka-main1001.eqiad.wmnet
Stashbot added a comment.Nov 27 2024, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-27T12:38:13Z] <effie> start replacing kafka-main1002 with kafka-main1007 - T363214

ops-monitoring-bot added a comment.Nov 27 2024, 12:56 PM

Icinga downtime and Alertmanager silence (ID=2081928b-ccb6-4f65-91fa-39f067b89fb2) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1002,1007].eqiad.wmnet
gerritbot added a comment.Nov 27 2024, 3:49 PM

Change #1098548 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kafka-main: Replace kafka-main1002 with kafka-main1007

https://gerrit.wikimedia.org/r/1098548

gerritbot added a project: Patch-For-Review.Nov 27 2024, 3:49 PM
gerritbot added a comment.Nov 27 2024, 4:04 PM

Change #1098548 merged by Effie Mouzeli:

[operations/puppet@production] kafka-main: Replace kafka-main1002 with kafka-main1007

https://gerrit.wikimedia.org/r/1098548

Stashbot added a comment.Nov 27 2024, 4:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-27T16:12:48Z] <effie> roll restarting kafka-main brokers - T363214

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 4:32 PM
gerritbot added a comment.Nov 27 2024, 4:41 PM

Change #1098559 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1007

https://gerrit.wikimedia.org/r/1098559

gerritbot added a project: Patch-For-Review.Nov 27 2024, 4:41 PM
gerritbot added a comment.Nov 27 2024, 5:16 PM

Change #1098559 merged by jenkins-bot:

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1007

https://gerrit.wikimedia.org/r/1098559

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 5:31 PM
jijiki updated the task description. (Show Details)Nov 27 2024, 5:51 PM
Stashbot added a comment.Dec 2 2024, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-02T13:31:37Z] <effie> repacing kafka-main1003 in production with kafka-main1008 - T363214

ops-monitoring-bot added a comment.Dec 2 2024, 1:37 PM

Icinga downtime and Alertmanager silence (ID=1b8e1077-6d61-4aa8-9dd3-51831260ac7d) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1002,1007].eqiad.wmnet
ops-monitoring-bot added a comment.Dec 2 2024, 1:51 PM

Icinga downtime and Alertmanager silence (ID=e69f875e-bd0c-45ff-b223-fbcb65b96846) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1003,1008].eqiad.wmnet
gerritbot added a comment.Dec 2 2024, 2:33 PM

Change #1099707 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kafka-main: Replace kafka-main1003 with kafka-main1008

https://gerrit.wikimedia.org/r/1099707

gerritbot added a project: Patch-For-Review.Dec 2 2024, 2:33 PM
gerritbot added a comment.Dec 2 2024, 5:38 PM

Change #1099707 merged by Effie Mouzeli:

[operations/puppet@production] kafka-main: Replace kafka-main1003 with kafka-main1008

https://gerrit.wikimedia.org/r/1099707

gerritbot added a comment.Dec 2 2024, 6:24 PM

Change #1099763 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1008

https://gerrit.wikimedia.org/r/1099763

gerritbot added a comment.Dec 2 2024, 6:30 PM

Change #1099763 merged by jenkins-bot:

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1008

https://gerrit.wikimedia.org/r/1099763

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2024, 7:30 PM
jijiki updated the task description. (Show Details)Dec 3 2024, 12:32 PM
ops-monitoring-bot added a comment.Dec 4 2024, 12:54 PM

Icinga downtime and Alertmanager silence (ID=1e93a56a-8513-43f0-ac1d-4277501c6e39) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1004,1009].eqiad.wmnet
gerritbot added a comment.Dec 4 2024, 1:05 PM

Change #1100447 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kafka-main: Replace kafka-main1004 with kafka-main1009

https://gerrit.wikimedia.org/r/1100447

gerritbot added a project: Patch-For-Review.Dec 4 2024, 1:05 PM
gerritbot added a comment.Dec 4 2024, 1:23 PM

Change #1100452 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1009 Replacing kafka-main1004

https://gerrit.wikimedia.org/r/1100452

gerritbot added a comment.Dec 4 2024, 3:14 PM

Change #1100447 merged by Effie Mouzeli:

[operations/puppet@production] kafka-main: Replace kafka-main1004 with kafka-main1009

https://gerrit.wikimedia.org/r/1100447

gerritbot added a comment.Dec 4 2024, 4:22 PM

Change #1100452 merged by jenkins-bot:

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1009 Replacing kafka-main1004

https://gerrit.wikimedia.org/r/1100452

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2024, 4:31 PM
jijiki updated the task description. (Show Details)Dec 4 2024, 4:53 PM
gerritbot added a comment.Dec 5 2024, 12:58 PM

Change #1100806 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kafka-main: Replace kafka-main1005 with kafka-main1010

https://gerrit.wikimedia.org/r/1100806

gerritbot added a project: Patch-For-Review.Dec 5 2024, 12:58 PM

Change #1100807 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] common.yaml: remove firewall rules for kafka-main100[1-5]

https://gerrit.wikimedia.org/r/1100807

ops-monitoring-bot added a comment.Dec 5 2024, 1:08 PM

Icinga downtime and Alertmanager silence (ID=96be0251-35a2-4695-9aaa-8a46e08acd28) set by jiji@cumin1002 for 1 day, 0:00:00 on 2 host(s) and their services with reason: Hardware refresh

kafka-main[1005,1010].eqiad.wmnet
gerritbot added a comment.Dec 5 2024, 3:20 PM

Change #1100827 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1010 Replacing kafka-main1005

https://gerrit.wikimedia.org/r/1100827

gerritbot added a comment.Dec 5 2024, 3:34 PM

Change #1100806 merged by Effie Mouzeli:

[operations/puppet@production] kafka-main: Replace kafka-main1005 with kafka-main1010

https://gerrit.wikimedia.org/r/1100806

gerritbot added a comment.Dec 5 2024, 4:07 PM

Change #1100827 merged by jenkins-bot:

[operations/deployment-charts@master] Update various kafka-main connection strings for kafka-main1010 Replacing kafka-main1005

https://gerrit.wikimedia.org/r/1100827

jijiki updated the task description. (Show Details)Dec 5 2024, 4:31 PM
jijiki closed this task as Resolved.Dec 5 2024, 4:33 PM

All servers have been replaced with the newer ones, decommission task has been filed: T381593. I also made some edits to the very helpful documentation Janis wrote.

dcausse mentioned this in T374628: Investigate why rdf-streaming-updater is unable to recover after replacing kafka-main@codfw nodes.Dec 5 2024, 5:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits