Page MenuHomePhabricator
Create Task
Maniphest T106581

Build 0.8.2.1 Kafka package and upgrade Kafka brokers
Closed, ResolvedPublic
Actions

Description

This is now a blocker for our eventlogging on Kafka project.

Details

SubjectRepoBranchLines +/-
Decom analytics1021 as a Kafka brokeroperations/puppetproduction+2 -12
Repuppetize kafka1018 as a brokeroperations/puppetproduction+4 -4
Rename A record for analytics1018 -> kafka1018operations/dnsmaster+2 -2
Rename analytics1018 -> kafka1018 in linux-host-entriesoperations/puppetproduction+7 -7
Rename analytics1022 -> kafka1022operations/puppetproduction+3 -3
Return expire of kafka1022 PTR to 1Hoperations/dnsmaster+1 -1
Temporarily set expire of PTR for kafka1022 to 5 min so I can reinstall asapoperations/dnsmaster+1 -1
Rename analytics1022 to kafka1022operations/dnsmaster+10 -8
Rename analytics1022 -> kafka1022operations/puppetproduction+7 -7
Update camus property files with names of new brokersanalytics/refinerymaster+2 -2
Puppetize systemd override for Kafka LimitNOFILEoperations/puppet/kafkamaster+47 -4
Don't use partman for analytics kafka jessie reinstall, do this part manuallyoperations/puppetproduction+1 -1
Use kafka1012 as hostname in Kafka cluster configoperations/puppetproduction+1 -1
Puppetize kafka1012 as kafka broker in analytics Kafka clusteroperations/puppetproduction+12 -0
Rename analytics1012 to kafka1012operations/dnsmaster+4 -4
Rename analytics1012 to kafka1012, site.pp puppetization coming in separate commitoperations/puppetproduction+8 -8
Update alerts and jmx for Kafka 0.8.2+operations/puppetproduction+19 -261
Update JMX metrics names for Kafka 0.8.2+operations/puppet/kafkamaster+363 -139
Better alias name for All topic metrics from kafkaoperations/puppetproduction+2 -2
Override kafka jmxtrans metrics to test new config for version 0.8.2.1operations/puppetproduction+26 -0
Rename analytics1013,1014,1020 to kafka1013,1014,1020operations/puppetproduction+39 -25
Updates and fixes for 0.8.2.1-2 releaseoperations/debs/kafkadebian+104 -74
Remove analytics1012 1013 1020 from list of kafka brokers in site.ppoperations/puppetproduction+1 -1
Remove newly provisioned kafka nodes from clusteroperations/puppetproduction+3 -3
Provision analytics1014 and analytics1020 as kafka brokersoperations/puppetproduction+6 -5
Provisioning analytics1013 as Kafka broker in analytics clusteroperations/puppetproduction+4 -4
Removing analytics1013,1014 and 1018 from hadoop worker list in site.ppoperations/puppetproduction+1 -1
Preparing to reinstall and expand Kafka cluster on Jessie at Kafka 0.8.2.1operations/puppetproduction+15 -1
Recreate debianization on top of 0.8.2.1 tagoperations/debs/kafkadebian+1 K -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ottomata closed subtask T98161: Build Kafka 0.8.1.1 package for Jessie and upgrade Brokers to Jessie. as Declined.Jul 31 2015, 2:09 PM
Ottomata added a parent task: T92602: Secure inter-datacenter web request log (Kafka) traffic.
Ottomata added a subtask: T90640: Audit hyperthreading on analytics nodes..Jul 31 2015, 2:23 PM
Ottomata added a comment.Jul 31 2015, 9:02 PM

analytics1046-1049 are online as of today! I have started the decommission process of analytics1013,1014 and 1020. These nodes will become kafka brokers soon. Hopefully by Monday they will have their blocks replicated elsewhere.

gerritbot added a comment.Aug 3 2015, 3:01 PM

Change 228826 had a related patch set uploaded (by Ottomata):
Preparing to reinstall and expand Kafka cluster on Jessie at Kafka 0.8.2.1

https://gerrit.wikimedia.org/r/228826

Ottomata mentioned this in rODBEaef854d1ef05: Debianize 0.8.2.1 tag.Aug 3 2015, 3:14 PM
gerritbot added a comment.Aug 3 2015, 3:20 PM

Change 228826 merged by Ottomata:
Preparing to reinstall and expand Kafka cluster on Jessie at Kafka 0.8.2.1

https://gerrit.wikimedia.org/r/228826

Ottomata mentioned this in rOPUP5e0d10afc822: Preparing to reinstall and expand Kafka cluster on Jessie at Kafka 0.8.2.1.Aug 3 2015, 3:20 PM
gerritbot added a comment.Aug 3 2015, 3:23 PM

Change 228832 had a related patch set uploaded (by Ottomata):
Removing analytics1013,1014 and 1018 from hadoop worker list in site.pp

https://gerrit.wikimedia.org/r/228832

gerritbot added a comment.Aug 3 2015, 3:24 PM

Change 228832 merged by Ottomata:
Removing analytics1013,1014 and 1018 from hadoop worker list in site.pp

https://gerrit.wikimedia.org/r/228832

Ottomata mentioned this in rOPUP0cff68bb11c1: Removing analytics1013,1014 and 1018 from hadoop worker list in site.pp.Aug 3 2015, 3:24 PM
gerritbot added a comment.Aug 3 2015, 4:28 PM

Change 228847 had a related patch set uploaded (by Ottomata):
Provisioning analytics1013 as Kafka broker in analytics cluster

https://gerrit.wikimedia.org/r/228847

gerritbot added a comment.Aug 3 2015, 4:33 PM

Change 228847 merged by Ottomata:
Provisioning analytics1013 as Kafka broker in analytics cluster

https://gerrit.wikimedia.org/r/228847

Ottomata mentioned this in rOPUP2dfd816de1e6: Provisioning analytics1013 as Kafka broker in analytics cluster.Aug 3 2015, 4:33 PM
gerritbot added a comment.Aug 3 2015, 4:42 PM

Change 228851 had a related patch set uploaded (by Ottomata):
Provision analytics1014 and analytics1020 as kafka brokers

https://gerrit.wikimedia.org/r/228851

gerritbot added a comment.Aug 3 2015, 4:43 PM

Change 228851 merged by Ottomata:
Provision analytics1014 and analytics1020 as kafka brokers

https://gerrit.wikimedia.org/r/228851

Ottomata mentioned this in rOPUP8b2068fdcf85: Provision analytics1014 and analytics1020 as kafka brokers.Aug 3 2015, 4:44 PM
gerritbot added a comment.Aug 3 2015, 10:42 PM

Change 229012 had a related patch set uploaded (by Ottomata):
Remove newly provisioned kafka nodes from cluster

https://gerrit.wikimedia.org/r/229012

gerritbot added a comment.Aug 3 2015, 10:52 PM

Change 229012 merged by Ottomata:
Remove newly provisioned kafka nodes from cluster

https://gerrit.wikimedia.org/r/229012

Ottomata mentioned this in rOPUPb8664b53189a: Remove newly provisioned kafka nodes from cluster.Aug 3 2015, 10:52 PM
gerritbot added a comment.Aug 4 2015, 12:02 AM

Change 229035 had a related patch set uploaded (by Ottomata):
Remove analytics1012 1013 1020 from list of kafka brokers in site.pp

https://gerrit.wikimedia.org/r/229035

gerritbot added a comment.Aug 4 2015, 12:02 AM

Change 229035 merged by Ottomata:
Remove analytics1012 1013 1020 from list of kafka brokers in site.pp

https://gerrit.wikimedia.org/r/229035

Ottomata mentioned this in rOPUPac7478154a43: Remove analytics1012 1013 1020 from list of kafka brokers in site.pp.Aug 4 2015, 12:02 AM
Ottomata added a comment.Aug 4 2015, 4:14 PM

Oof, had some problems yesterday :(

Incident documentation here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150803-Kafka

gerritbot added a comment.Aug 4 2015, 7:29 PM

Change 229193 had a related patch set uploaded (by Ottomata):
Updates and fixes for 0.8.2.1-2 release

https://gerrit.wikimedia.org/r/229193

Ottomata added a comment.Aug 5 2015, 8:43 PM

Phew, ok, Joseph and I tested this migration again in labs:
https://etherpad.wikimedia.org/p/kafka_0.8.2.1_migration_labs

This all went smoothly. I've made a migration plan based on this process here:
https://etherpad.wikimedia.org/p/kafka_0.8.2.1_migration2

Alex is reviewing my latest packaging patch:
https://gerrit.wikimedia.org/r/#/c/229193/

Once we get that settled, I can rebuild and publish the .debs in our apt. I'll then test this package in labs one more time, and then I'll feel comfortable trying the upgrade again.

gerritbot added a comment.Aug 6 2015, 6:16 PM

Change 229193 merged by Ottomata:
Updates and fixes for 0.8.2.1-2 release

https://gerrit.wikimedia.org/r/229193

Ottomata mentioned this in rODBEa63cb8805fbd: Updates and fixes for 0.8.2.1-2 release.Aug 6 2015, 6:54 PM
gerritbot added a comment.Aug 6 2015, 9:28 PM

Change 229961 had a related patch set uploaded (by Ottomata):
Rename analytics1013,1014,1020 to kafka1013,1014,1020

https://gerrit.wikimedia.org/r/229961

gerritbot added a comment.Aug 6 2015, 9:38 PM

Change 229961 merged by Ottomata:
Rename analytics1013,1014,1020 to kafka1013,1014,1020

https://gerrit.wikimedia.org/r/229961

Ottomata mentioned this in rOPUP35b6a4e92e7e: Rename analytics1013,1014,1020 to kafka1013,1014,1020.Aug 6 2015, 9:39 PM
gerritbot added a comment.Aug 10 2015, 4:52 PM

Change 230576 had a related patch set uploaded (by Ottomata):
Override kafka jmxtrans metrics to test new config for version 0.8.2.1

https://gerrit.wikimedia.org/r/230576

gerritbot added a comment.Aug 10 2015, 4:56 PM

Change 230576 merged by Ottomata:
Override kafka jmxtrans metrics to test new config for version 0.8.2.1

https://gerrit.wikimedia.org/r/230576

Ottomata mentioned this in rOPUPd7867d2a8413: Override kafka jmxtrans metrics to test new config for version 0.8.2.1.Aug 10 2015, 4:56 PM
gerritbot added a comment.Aug 10 2015, 5:11 PM

Change 230577 had a related patch set uploaded (by Ottomata):
Better alias name for All topic metrics from kafka

https://gerrit.wikimedia.org/r/230577

gerritbot added a comment.Aug 10 2015, 5:13 PM

Change 230577 merged by Ottomata:
Better alias name for All topic metrics from kafka

https://gerrit.wikimedia.org/r/230577

Ottomata mentioned this in rOPUP8ca612e3b369: Better alias name for All topic metrics from kafka.Aug 10 2015, 5:13 PM
Ottomata renamed this task from Build new latest stable (0.8.2.1?) Kafka package and upgrade Kafka brokers to Build 0.8.2.1 Kafka package and upgrade Kafka brokers.Aug 12 2015, 2:13 PM
Ottomata added a comment.EditedAug 12 2015, 2:18 PM

Phew, after much difficulty, the 4 original Precise brokers are now running 0.8.2.1. There was a bug in the version of snappy that 0.8.2.1 needs that caused us much headache.

Immediate clean up TODOs:

  • Make an awesome grafana dashboard: http://grafana.wikimedia.org/#/dashboard/db/kafka
  • Clean up jmxtrans metrics, adapt kafka jmxtrans class, etc.
  • Audit Kafka alerts and make sure they are based on the correct metric names.
  • Write incident report and document data loss
  • Make sure Oozie jobs are running and normal (@JAllemandou is doing this).
  • Repackage Kafka for Precise, Trusty and Jessie with snappy 1.1.1.7 fix.

I won't be doing the next steps of this migration until the above is done.

gerritbot added a comment.Aug 12 2015, 3:21 PM

Change 231021 had a related patch set uploaded (by Ottomata):
Update JMX metrics names for Kafka 0.8.2

https://gerrit.wikimedia.org/r/231021

gerritbot added a comment.Aug 12 2015, 4:04 PM

Change 231021 merged by Ottomata:
Update JMX metrics names for Kafka 0.8.2

https://gerrit.wikimedia.org/r/231021

gerritbot added a comment.Aug 12 2015, 4:05 PM

Change 231028 had a related patch set uploaded (by Ottomata):
Update alerts and jmx for Kafka 0.8.2

https://gerrit.wikimedia.org/r/231028

gerritbot added a comment.Aug 12 2015, 4:09 PM

Change 231028 merged by Ottomata:
Update alerts and jmx for Kafka 0.8.2

https://gerrit.wikimedia.org/r/231028

Ottomata mentioned this in rOPUPbaf71f5c0a9d: Update alerts and jmx for Kafka 0.8.2+.Aug 12 2015, 4:09 PM
Ottomata mentioned this in rOPKAba99cba34dfc: Update JMX metrics names for Kafka 0.8.2+.Aug 12 2015, 6:31 PM
JAllemandou added a comment.Aug 12 2015, 7:19 PM
In T106581#1531644, @Ottomata wrote:

Phew, after much difficulty, the 4 original Precise brokers are now running 0.8.2.1. There was a bug in the version of snappy that 0.8.2.1 needs that caused us much headache.

Immediate clean up TODOs:

  • Make an awesome grafana dashboard: http://grafana.wikimedia.org/#/dashboard/db/kafka
  • Clean up jmxtrans metrics, adapt kafka jmxtrans class, etc.
  • Audit Kafka alerts and make sure they are based on the correct metric names.
  • Write incident report and document data loss
  • Make sure Oozie jobs are running and normal (@JAllemandou is doing this).
  • Repackage Kafka for Precise, Trusty and Jessie with snappy 1.1.1.7 fix.

I won't be doing the next steps of this migration until the above is done.

My bit seems ok :)

gerritbot added a comment.Aug 17 2015, 7:01 PM

Change 232097 had a related patch set uploaded (by Ottomata):
Don't use partman for analytics kafka jessie reinstall, do this part manually

https://gerrit.wikimedia.org/r/232097

gerritbot added a comment.Aug 17 2015, 7:02 PM

Change 232098 had a related patch set uploaded (by Ottomata):
Rename analytics1012 to kafka1012

https://gerrit.wikimedia.org/r/232098

gerritbot added a comment.Aug 17 2015, 7:04 PM

Change 232097 merged by Ottomata:
Don't use partman for analytics kafka jessie reinstall, do this part manually

https://gerrit.wikimedia.org/r/232097

Ottomata mentioned this in rOPUP5c357d7a21fc: Don't use partman for analytics kafka jessie reinstall, do this part manually.Aug 17 2015, 7:04 PM
gerritbot added a comment.Aug 17 2015, 7:11 PM

Change 232136 had a related patch set uploaded (by Ottomata):
Rename analytics1012 to kafka1012, site.pp puppetization coming in separate commit

https://gerrit.wikimedia.org/r/232136

gerritbot added a comment.Aug 17 2015, 7:13 PM

Change 232136 merged by Ottomata:
Rename analytics1012 to kafka1012, site.pp puppetization coming in separate commit

https://gerrit.wikimedia.org/r/232136

Ottomata mentioned this in rOPUPad7c8803b298: Rename analytics1012 to kafka1012, site.pp puppetization coming in separate….Aug 17 2015, 7:14 PM
gerritbot added a comment.Aug 17 2015, 7:23 PM

Change 232098 merged by Ottomata:
Rename analytics1012 to kafka1012

https://gerrit.wikimedia.org/r/232098

Ottomata mentioned this in rODNSc7db862f2e1e: Rename analytics1012 to kafka1012.Aug 17 2015, 7:23 PM
gerritbot added a comment.Aug 17 2015, 8:58 PM

Change 232202 had a related patch set uploaded (by Ottomata):
Puppetize kafka1012 as kafka broker in analytics Kafka cluster

https://gerrit.wikimedia.org/r/232202

gerritbot added a comment.Aug 17 2015, 9:01 PM

Change 232202 merged by Ottomata:
Puppetize kafka1012 as kafka broker in analytics Kafka cluster

https://gerrit.wikimedia.org/r/232202

Ottomata mentioned this in rOPUP319db37b3d9d: Puppetize kafka1012 as kafka broker in analytics Kafka cluster.Aug 17 2015, 9:02 PM
gerritbot added a comment.Aug 17 2015, 9:04 PM

Change 232203 had a related patch set uploaded (by Ottomata):
Use kafka1012 as hostname in Kafka cluster config

https://gerrit.wikimedia.org/r/232203

gerritbot added a comment.Aug 17 2015, 9:05 PM

Change 232203 merged by Ottomata:
Use kafka1012 as hostname in Kafka cluster config

https://gerrit.wikimedia.org/r/232203

Ottomata mentioned this in rOPUPb3fe7e1666b0: Use kafka1012 as hostname in Kafka cluster config.Aug 17 2015, 9:05 PM
gerritbot added a comment.Aug 18 2015, 6:56 PM

Change 232319 had a related patch set uploaded (by Ottomata):
Puppetize systemd override for Kafka LimitNOFILE

https://gerrit.wikimedia.org/r/232319

gerritbot added a comment.Aug 18 2015, 7:54 PM

Change 232319 merged by Ottomata:
Puppetize systemd override for Kafka LimitNOFILE

https://gerrit.wikimedia.org/r/232319

Ottomata mentioned this in rOPKAb75128da2755: Puppetize systemd override for Kafka LimitNOFILE.Aug 18 2015, 8:11 PM
gerritbot added a comment.Aug 19 2015, 5:42 PM

Change 232534 had a related patch set uploaded (by Ottomata):
Rename analytics1022 to kafka1022

https://gerrit.wikimedia.org/r/232534

gerritbot added a comment.Aug 19 2015, 5:46 PM

Change 232535 had a related patch set uploaded (by Ottomata):
Update camus property files with names of new brokers

https://gerrit.wikimedia.org/r/232535

gerritbot added a comment.Aug 19 2015, 5:46 PM

Change 232535 merged by Ottomata:
Update camus property files with names of new brokers

https://gerrit.wikimedia.org/r/232535

Ottomata mentioned this in rANREd1c175f77ecc: Update camus property files with names of new brokers.Aug 19 2015, 5:48 PM
gerritbot added a comment.Aug 19 2015, 6:29 PM

Change 232542 had a related patch set uploaded (by Ottomata):
Rename analytics1022 -> kafka1022

https://gerrit.wikimedia.org/r/232542

gerritbot added a comment.Aug 19 2015, 6:29 PM

Change 232542 merged by Ottomata:
Rename analytics1022 -> kafka1022

https://gerrit.wikimedia.org/r/232542

Ottomata mentioned this in rOPUPa51ed9040a63: Rename analytics1022 -> kafka1022.Aug 19 2015, 6:29 PM
gerritbot added a comment.Aug 19 2015, 6:33 PM

Change 232534 merged by Ottomata:
Rename analytics1022 to kafka1022

https://gerrit.wikimedia.org/r/232534

Ottomata mentioned this in rODNS9e15aa5de7d8: Rename analytics1022 to kafka1022.Aug 19 2015, 6:35 PM
gerritbot added a comment.Aug 19 2015, 6:42 PM

Change 232557 had a related patch set uploaded (by Ottomata):
Rename analytics1022 -> kafka1022

https://gerrit.wikimedia.org/r/232557

gerritbot added a comment.Aug 19 2015, 6:47 PM

Change 232559 had a related patch set uploaded (by Ottomata):
Temporarily set expire of PTR for kafka1022 to 5 min so I can reinstall asap

https://gerrit.wikimedia.org/r/232559

gerritbot added a comment.Aug 19 2015, 6:47 PM

Change 232559 merged by Ottomata:
Temporarily set expire of PTR for kafka1022 to 5 min so I can reinstall asap

https://gerrit.wikimedia.org/r/232559

Ottomata mentioned this in rODNSefb4710330fe: Temporarily set expire of PTR for kafka1022 to 5 min so I can reinstall asap.Aug 19 2015, 6:47 PM
gerritbot added a comment.Aug 19 2015, 6:49 PM

Change 232560 had a related patch set uploaded (by Ottomata):
Return expire of kafka1022 PTR to 1H

https://gerrit.wikimedia.org/r/232560

gerritbot added a comment.Aug 19 2015, 7:06 PM

Change 232560 merged by Ottomata:
Return expire of kafka1022 PTR to 1H

https://gerrit.wikimedia.org/r/232560

Ottomata mentioned this in rODNS88cd420e7cf4: Return expire of kafka1022 PTR to 1H.Aug 19 2015, 7:06 PM
gerritbot added a comment.Aug 19 2015, 8:16 PM

Change 232557 merged by Ottomata:
Rename analytics1022 -> kafka1022

https://gerrit.wikimedia.org/r/232557

Ottomata mentioned this in rOPUP639f89a0b5de: Rename analytics1022 -> kafka1022.Aug 19 2015, 8:16 PM
gerritbot added a comment.Aug 20 2015, 5:29 PM

Change 232769 had a related patch set uploaded (by Ottomata):
Rename analytics1018 -> kafka1018 in linux-host-entries

https://gerrit.wikimedia.org/r/232769

gerritbot added a comment.Aug 20 2015, 5:31 PM

Change 232769 merged by Ottomata:
Rename analytics1018 -> kafka1018 in linux-host-entries

https://gerrit.wikimedia.org/r/232769

Ottomata mentioned this in rOPUPa7a1f69bed14: Rename analytics1018 -> kafka1018 in linux-host-entries.Aug 20 2015, 5:31 PM
gerritbot added a comment.Aug 20 2015, 6:10 PM

Change 232774 had a related patch set uploaded (by Ottomata):
Rename A record for analytics1018 -> kafka1018

https://gerrit.wikimedia.org/r/232774

gerritbot added a comment.Aug 20 2015, 6:11 PM

Change 232774 merged by Ottomata:
Rename A record for analytics1018 -> kafka1018

https://gerrit.wikimedia.org/r/232774

Ottomata mentioned this in rODNS37f5827403fa: Rename A record for analytics1018 -> kafka1018.Aug 20 2015, 6:11 PM
gerritbot added a comment.Aug 20 2015, 6:19 PM

Change 232776 had a related patch set uploaded (by Ottomata):
Repuppetize kafka1018 as a broker

https://gerrit.wikimedia.org/r/232776

gerritbot added a comment.Aug 20 2015, 6:20 PM

Change 232776 merged by Ottomata:
Repuppetize kafka1018 as a broker

https://gerrit.wikimedia.org/r/232776

Ottomata mentioned this in rOPUPadc7313afc55: Repuppetize kafka1018 as a broker.Aug 20 2015, 6:21 PM
gerritbot added a comment.Aug 27 2015, 1:12 PM

Change 234265 had a related patch set uploaded (by Ottomata):
Decom analytics1021 as a Kafka broker

https://gerrit.wikimedia.org/r/234265

gerritbot added a comment.Aug 27 2015, 1:14 PM

Change 234265 merged by Ottomata:
Decom analytics1021 as a Kafka broker

https://gerrit.wikimedia.org/r/234265

Ottomata mentioned this in rOPUP738294950870: Decom analytics1021 as a Kafka broker.Aug 27 2015, 1:14 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Aug 28 2015, 3:34 PM
• kevinator closed this task as Resolved.Aug 29 2015, 12:19 AM
• kevinator subscribed.
Ottomata closed subtask T90640: Audit hyperthreading on analytics nodes. as Resolved.Jul 27 2016, 2:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL