Page MenuHomePhabricator
Create Task
Maniphest T279342

Migrate colocated kafka-logging brokers to dedicated kafka-logging hosts
Closed, ResolvedPublic
Actions

Description

Kafka-logging hardware is online now, time to migrate the brokers colocated on the logstash nodes to to dedicated hardware.

  • logstash1010 -> kafka-logging1001
  • logstash1011 -> kafka-logging1002
  • logstash1012 -> kafka-logging1003
  • logstash2001 -> kafka-logging2001
  • logstash2002 -> kafka-logging2002
  • logstash2003 -> kafka-logging2003

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

herron triaged this task as Medium priority.Apr 5 2021, 7:40 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 5 2021, 7:40 PM
herron added a project: observability.Apr 5 2021, 7:40 PM
gerritbot added a comment.Apr 5 2021, 7:40 PM

Change 677009 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrate broker logstash1010 to kafka-logging1001

https://gerrit.wikimedia.org/r/677009

gerritbot added a project: Patch-For-Review.Apr 5 2021, 7:40 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Apr 6 2021, 3:13 PM
Stashbot added a comment.Apr 13 2021, 3:26 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-13T15:26:15Z] <herron> migrating kafka-logging broker logstash1010 to kafka-logging1001 T279342

gerritbot added a comment.Apr 13 2021, 3:40 PM

Change 677009 merged by Herron:

[operations/puppet@production] kafka-logging: migrate broker logstash1010 to kafka-logging1001

https://gerrit.wikimedia.org/r/677009

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2021, 4:10 PM
herron updated the task description. (Show Details)Apr 13 2021, 7:12 PM
gerritbot added a comment.Apr 14 2021, 6:23 PM

Change 679411 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrate broker logstash1011 to kafka-logging1002

https://gerrit.wikimedia.org/r/679411

gerritbot added a project: Patch-For-Review.Apr 14 2021, 6:23 PM
Stashbot added a comment.Apr 14 2021, 7:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T19:42:04Z] <herron> migrating kafka-logging broker logstash1011 to kafka-logging1002 T279342

gerritbot added a comment.Apr 14 2021, 7:45 PM

Change 679411 merged by Herron:

[operations/puppet@production] kafka-logging: migrate broker logstash1011 to kafka-logging1002

https://gerrit.wikimedia.org/r/679411

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2021, 8:10 PM
gerritbot added a comment.Apr 15 2021, 9:59 AM

Change 679740 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/homer/public@master] Add kafka-logging100{2,3} IPs to the kafka term of analytics filters

https://gerrit.wikimedia.org/r/679740

gerritbot added a project: Patch-For-Review.Apr 15 2021, 9:59 AM
elukey added subscribers: dcaro, aborrero, elukey.Apr 15 2021, 10:05 AM

@herron the move to the new brokers is causing some troubles to analytics and wmcs, since the IPs are not listed in the VLAN firewall filters. For example:

https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=20&orgId=1&refresh=5m&from=now-24h&to=now&var-server=cloudvirt1012&var-datasource=eqiad%20prometheus%2Fops&var-cluster=wmcs

I added 1001 yesterday, and filed https://gerrit.wikimedia.org/r/679740 today to add 1002/1003, but we should have done it before starting :D

Pinging also @aborrero @dcaro, they will probably make a similar change to homer_public.

elukey added a comment.Apr 15 2021, 10:07 AM

Also for some reason the AAAA records are not present in our Auth DNS servers, but I see it listed in Netbox..

gerritbot added a comment.Apr 15 2021, 10:18 AM

Change 679740 merged by Elukey:

[operations/homer/public@master] Add kafka-logging100{2,3} IPs to the kafka term of analytics filters

https://gerrit.wikimedia.org/r/679740

elukey mentioned this in rOHPU2a1f9ed8f9a6: Add kafka-logging100{2,3} IPs to the kafka term of analytics filters.Apr 15 2021, 10:21 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 15 2021, 11:10 AM
herron added a comment.Apr 15 2021, 2:27 PM

Hey @elukey thanks for taking care of that! Was not aware that needed to happen ahead of migrating brokers, TIL. Have added that to the migration process as an early step. I'll upload a similar patch to cover the codfw broker migrations.

With regard to AAAA records, this is intentional for now. There is T271138 which includes enabling IPv6 on the logging cluster. Planning to tackle that independent from the broker migrations.

herron updated the task description. (Show Details)Apr 15 2021, 2:28 PM
gerritbot added a comment.Apr 15 2021, 3:37 PM

Change 679862 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/homer/public@master] cr/firewall: add kafka-logging servers to labs-in filters

https://gerrit.wikimedia.org/r/679862

gerritbot added a project: Patch-For-Review.Apr 15 2021, 3:37 PM
gerritbot added a comment.Apr 15 2021, 4:31 PM

Change 679875 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrake broker logstash1012 to kafka-logging1003

https://gerrit.wikimedia.org/r/679875

Stashbot added a comment.Apr 15 2021, 8:03 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-15T20:03:34Z] <herron> migrating kafka-logging broker logstash1012 to kafka-logging1003 T279342

gerritbot added a comment.Apr 15 2021, 8:09 PM

Change 679875 merged by Herron:

[operations/puppet@production] kafka-logging: migrake broker logstash1012 to kafka-logging1003

https://gerrit.wikimedia.org/r/679875

gerritbot added a comment.Apr 16 2021, 10:41 AM

Change 679862 merged by jenkins-bot:

[operations/homer/public@master] cr/firewall: add kafka-logging servers to labs-in filters

https://gerrit.wikimedia.org/r/679862

jenkins-bot mentioned this in rOHPU6695c39e7ff1: cr/firewall: add kafka-logging servers to labs-in filters.Apr 16 2021, 10:44 AM
Stashbot added a comment.Apr 16 2021, 10:44 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-16T10:44:48Z] <arturo> merging homer change to cr-eqiad (T279342)

aborrero added a subscriber: ayounsi.Apr 16 2021, 10:45 AM

Thanks @elukey and @ayounsi, all is apparently done on the WMCS side. Ping me if something else is needed.

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2021, 11:10 AM
Ottomata mentioned this in T253058: DRY kafka broker declaration in helmfiles.Apr 16 2021, 3:55 PM
gerritbot added a comment.Apr 16 2021, 4:05 PM

Change 680380 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-logging-external - update networkpolicy with new kafka brokers

https://gerrit.wikimedia.org/r/680380

gerritbot added a project: Patch-For-Review.Apr 16 2021, 4:05 PM
gerritbot added a comment.Apr 16 2021, 4:08 PM

Change 680380 merged by Ottomata:

[operations/deployment-charts@master] eventgate-logging-external - update networkpolicy with new kafka brokers

https://gerrit.wikimedia.org/r/680380

herron updated the task description. (Show Details)Apr 16 2021, 4:11 PM
fgiunchedi mentioned this in T279457: Multiple host down alerts from rack C2.Apr 20 2021, 1:56 PM
herron mentioned this in T281266: Decommission old ELK5 Logstash cluster.Apr 27 2021, 3:30 PM
gerritbot added a comment.Apr 27 2021, 5:31 PM

Change 683012 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrate logstash2001 broker to kafka-logging2001

https://gerrit.wikimedia.org/r/683012

gerritbot added a comment.Apr 27 2021, 5:31 PM

Change 683013 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrate logstash2002 broker to kafka-logging2002

https://gerrit.wikimedia.org/r/683013

gerritbot added a comment.Apr 27 2021, 5:31 PM

Change 683014 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafka-logging: migrate logstash2003 broker to kafka-logging2003

https://gerrit.wikimedia.org/r/683014

gerritbot added a comment.Apr 27 2021, 7:42 PM

Change 683047 had a related patch set uploaded (by Herron; author: Herron):

[operations/deployment-charts@master] eventgate-logging-external: add new codfw kafka-logging hosts to network policy

https://gerrit.wikimedia.org/r/683047

gerritbot added a comment.Apr 27 2021, 7:50 PM

Change 683050 had a related patch set uploaded (by Herron; author: Herron):

[operations/homer/public@master] add kafka-logging200[123] to kafka term

https://gerrit.wikimedia.org/r/683050

herron added a comment.EditedMay 3 2021, 10:08 PM

Reimaging the eqiad kafka-logging hosts and configuring them with raid50 layout, this will give us 5T usable (as opposed to the current 3T raid10) per host.

  • kafka-logging1001
  • kafka-logging1002
  • kafka-logging1003
gerritbot added a comment.May 4 2021, 8:21 PM

Change 685090 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] logstash101[012]: prep for reimaging

https://gerrit.wikimedia.org/r/685090

gerritbot added a comment.May 11 2021, 2:29 PM

Change 685090 merged by Herron:

[operations/puppet@production] logstash101[012]: prep for reimaging

https://gerrit.wikimedia.org/r/685090

gerritbot added a comment.May 13 2021, 3:18 PM

Change 683050 merged by jenkins-bot:

[operations/homer/public@master] add kafka-logging200[123] to kafka term

https://gerrit.wikimedia.org/r/683050

jenkins-bot mentioned this in rOHPUb0ea55dd5903: add kafka-logging200[123] to kafka term.May 13 2021, 3:23 PM
gerritbot added a comment.May 13 2021, 8:07 PM

Change 683047 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-logging-external: add codfw kafka-logging hosts

https://gerrit.wikimedia.org/r/683047

Ottomata mentioned this in T282454: Switch kafka/Hadoop away from java::security.May 19 2021, 8:01 PM
elukey added a comment.Jun 3 2021, 6:58 AM

The kafka-logging100[1-3] hosts have /srv partitions slowly filling up, mostly due to two topics:

elukey@kafka-logging1001:/srv/kafka/data$ sudo du -hs -- * | sort -h | tail -n 20
24G	rsyslog-info-4
24G	rsyslog-info-5
37G	udp_localhost-info-2
37G	udp_localhost-info-5
38G	udp_localhost-info-0
38G	udp_localhost-info-1
38G	udp_localhost-info-3
38G	udp_localhost-info-4
279G	rsyslog-notice-0
279G	rsyslog-notice-1
279G	rsyslog-notice-2
279G	rsyslog-notice-3
279G	rsyslog-notice-4
279G	rsyslog-notice-5
388G	udp_localhost-warning-1
388G	udp_localhost-warning-3
388G	udp_localhost-warning-4
388G	udp_localhost-warning-5
389G	udp_localhost-warning-0
389G	udp_localhost-warning-2

I briefly checked udp_localhost-warning and there is definitely some spammy log from mediawiki ongoing, but maybe it is needed for debugging purposes. Maybe we could lower the retention for some topics?

gerritbot added a comment.Jun 11 2021, 1:50 PM

Change 699418 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/cookbooks@master] sre/kafka/* update kafka cluster choices

https://gerrit.wikimedia.org/r/699418

gerritbot added a comment.Jun 14 2021, 1:36 PM

Change 699418 merged by Ottomata:

[operations/cookbooks@master] sre/kafka/* update kafka cluster choices

https://gerrit.wikimedia.org/r/699418

gerritbot added a comment.Jun 24 2021, 4:54 PM

Change 683012 merged by Herron:

[operations/puppet@production] kafka-logging: migrate logstash2001 broker to kafka-logging2001

https://gerrit.wikimedia.org/r/683012

gerritbot added a comment.Jun 24 2021, 5:15 PM

Change 701440 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] add kafka-logging200[123] to kafka_brokers_logging

https://gerrit.wikimedia.org/r/701440

gerritbot added a comment.Jun 24 2021, 5:18 PM

Change 701440 merged by Herron:

[operations/puppet@production] add kafka-logging200[123] to kafka_brokers_logging

https://gerrit.wikimedia.org/r/701440

gerritbot added a comment.Jun 24 2021, 5:52 PM

Change 683013 merged by Herron:

[operations/puppet@production] kafka-logging: migrate logstash2002 broker to kafka-logging2002

https://gerrit.wikimedia.org/r/683013

gerritbot added a comment.Jun 24 2021, 6:28 PM

Change 683014 merged by Herron:

[operations/puppet@production] kafka-logging: migrate logstash2003 broker to kafka-logging2003

https://gerrit.wikimedia.org/r/683014

herron closed this task as Resolved.Jun 24 2021, 6:51 PM
herron claimed this task.
herron updated the task description. (Show Details)
gerritbot added a comment.Apr 5 2022, 3:43 PM

Change 777375 had a related patch set uploaded (by Herron; author: Herron):

[operations/cookbooks@master] sre.kafka.reboot-workers: add logging-codfw targets

https://gerrit.wikimedia.org/r/777375

gerritbot added a comment.Apr 6 2022, 4:32 PM

Change 777375 merged by jenkins-bot:

[operations/cookbooks@master] sre.kafka.reboot-workers: add logging-codfw targets

https://gerrit.wikimedia.org/r/777375

Ottomata mentioned this in rCCKB448cfb2784f1: sre/kafka/* update kafka cluster choices.Dec 14 2022, 3:28 PM
herron mentioned this in rCCKB88b168f32f4e: sre.kafka.reboot-workers: add logging-codfw targets.Dec 14 2022, 3:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL