Page MenuHomePhabricator
Create Task
Maniphest T138265

Upgrade kafka main clusters to 0.9
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Just the work necessary to do the actual upgrade (plan & execute). codfw already done. (blocked on event bus work).

Details

SubjectRepoBranchLines +/-
Remove now unused kafka moduleoperations/puppetproduction+0 -71
Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9operations/puppetproduction+0 -2
Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9operations/puppetproduction+29 -87
Add error_output to eventlogging service and make eventbus write EventErrors to log fileoperations/puppetproduction+57 -34
Finalize main-codfw Kafka upgradeoperations/puppetproduction+10 -14
Upgrade Kafka main-codfw to 0.9operations/puppetproduction+96 -49
Stop kafka mirror maker on kafka100[12], it is not doing anything anywayoperations/puppetproduction+1 -4
Customize query in gerrit

Related Objects

Mentioned In
rOPUPea2329c6c0d9: Remove now unused kafka module
rOPUP5d412d2e337e: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9
rOPUP236f30a76529: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9
rOPUPf3e97ee237db: Remove now unused kafka module
rOPUP9a2dc83eb621: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9
rOPUP9ae819e4cdac: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9
rOPUP408243f7b410: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9
rOPUPbd9d086c1e84: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9
rOPUP7f5396ec0c7b: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUPa4c99a022d92: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP3c57558e6ea8: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUPfb95e441fab6: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP0139eb3f4b53: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP3430bde1f207: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP9d18edc1e08d: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP7a23f4475ac9: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUPc51dd9af7b65: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUP80cd11d55513: Add error_output to eventlogging service and make eventbus write EventErrors…
rOPUPd7cfa88b9631: Finalize main-codfw Kafka upgrade
rOPUPdc4b2b6971a9: Finalize main-codfw Kafka upgrade
rOPUPe329aecc77e8: Upgrade Kafka main-codfw to 0.9
rOPUP5ee562102732: Finalize main-codfw Kafka upgrade
rOPUP430f3fb1fba9: Upgrade Kafka main-codfw to 0.9
rOPUP74131f1b71fd: Stop kafka mirror maker on kafka100[12], it is not doing anything anyway
rOPUPfb2802045a34: Stop kafka mirror maker on kafka100[12], it is not doing anything anyway

Event Timeline

• Nuria created this task.Jun 20 2016, 8:55 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 20 2016, 8:55 PM
Milimetric triaged this task as Medium priority.Jul 7 2016, 5:32 PM
Milimetric moved this task from Incoming to Dashiki on the Analytics board.
gerritbot subscribed.Jul 15 2016, 2:21 PM

Change 299149 had a related patch set uploaded (by Ottomata):
Stop kafka mirror maker on kafka100[12], it is not doing anything anyway

https://gerrit.wikimedia.org/r/299149

gerritbot added a project: Patch-For-Review.Jul 15 2016, 2:21 PM
gerritbot added a comment.Jul 15 2016, 2:25 PM

Change 299149 merged by Ottomata:
Stop kafka mirror maker on kafka100[12], it is not doing anything anyway

https://gerrit.wikimedia.org/r/299149

Ottomata mentioned this in rOPUPfb2802045a34: Stop kafka mirror maker on kafka100[12], it is not doing anything anyway.Jul 15 2016, 2:26 PM
Ottomata mentioned this in rOPUP74131f1b71fd: Stop kafka mirror maker on kafka100[12], it is not doing anything anyway.
Ottomata renamed this task from Upgrade Kafka (non-analytics cluster) to Upgrade kafka main clusters.Jul 22 2016, 1:04 PM
Ottomata renamed this task from Upgrade kafka main clusters to Upgrade kafka main clusters to 0.9.Jul 25 2016, 1:46 PM
Ottomata claimed this task.
Ottomata edited projects, added Analytics-Kanban, Event-Platform; removed Analytics.
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)

I'd like to do main-codfw this week. Will coordinate with services on this.

Ottomata mentioned this in rOPUP430f3fb1fba9: Upgrade Kafka main-codfw to 0.9.Jul 25 2016, 1:57 PM
Ottomata added a subscriber: Eevans.Jul 25 2016, 2:21 PM

@Eevans let's sync up on IRC today about this.

gerritbot added a comment.Jul 25 2016, 4:19 PM

Change 300896 had a related patch set uploaded (by Ottomata):
Finalize main-codfw Kafka upgrade

https://gerrit.wikimedia.org/r/300896

Ottomata mentioned this in rOPUP5ee562102732: Finalize main-codfw Kafka upgrade.Jul 25 2016, 4:22 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 25 2016, 5:46 PM
gerritbot added a comment.Jul 25 2016, 6:30 PM

Change 300867 merged by Ottomata:
Upgrade Kafka main-codfw to 0.9

https://gerrit.wikimedia.org/r/300867

Ottomata mentioned this in rOPUPe329aecc77e8: Upgrade Kafka main-codfw to 0.9.Jul 25 2016, 6:32 PM
Ottomata mentioned this in rOPUPdc4b2b6971a9: Finalize main-codfw Kafka upgrade.Jul 25 2016, 6:46 PM
Ottomata mentioned this in rOPUPd7cfa88b9631: Finalize main-codfw Kafka upgrade.
gerritbot added a comment.Jul 25 2016, 6:47 PM

Change 300896 merged by Ottomata:
Finalize main-codfw Kafka upgrade

https://gerrit.wikimedia.org/r/300896

Ottomata added a comment.Jul 25 2016, 8:10 PM

codfw has been upgraded to 0.9.

We found a bug in the version of kafka-python we are using for eventbus. To work around this for this deploy, before we upgrade in eqiad I'd like to merge and deploy https://gerrit.wikimedia.org/r/#/c/300944/. This will allow us to log events that did not produce properly, and reproduce them after the upgrade.

Ottomata moved this task from In Progress to Paused on the Analytics-Kanban board.Jul 28 2016, 4:18 PM
Milimetric updated the task description. (Show Details)Jul 28 2016, 4:57 PM
Milimetric set the point value for this task to 5.
gerritbot added a comment.Aug 10 2016, 2:14 PM

Change 304029 had a related patch set uploaded (by Ottomata):
Add error_output to eventlogging service and make eventbus write EventErrors to log file

https://gerrit.wikimedia.org/r/304029

Ottomata mentioned this in rOPUP80cd11d55513: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:19 PM
Ottomata mentioned this in rOPUPc51dd9af7b65: Add error_output to eventlogging service and make eventbus write EventErrors….
Ottomata mentioned this in rOPUP7a23f4475ac9: Add error_output to eventlogging service and make eventbus write EventErrors….
Ottomata mentioned this in rOPUP9d18edc1e08d: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:25 PM
Ottomata mentioned this in rOPUP3430bde1f207: Add error_output to eventlogging service and make eventbus write EventErrors….
Ottomata mentioned this in rOPUP0139eb3f4b53: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:30 PM
Ottomata mentioned this in rOPUPfb95e441fab6: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:35 PM
Ottomata mentioned this in rOPUP3c57558e6ea8: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:41 PM
Ottomata mentioned this in rOPUPa4c99a022d92: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 2:51 PM
Ottomata mentioned this in rOPUP7f5396ec0c7b: Add error_output to eventlogging service and make eventbus write EventErrors….Aug 10 2016, 4:52 PM
gerritbot added a comment.Aug 10 2016, 5:04 PM

Change 304029 merged by Ottomata:
Add error_output to eventlogging service and make eventbus write EventErrors to log file

https://gerrit.wikimedia.org/r/304029

Ottomata moved this task from Paused to In Progress on the Analytics-Kanban board.Aug 15 2016, 2:06 PM
gerritbot added a comment.Aug 15 2016, 2:17 PM

Change 304821 had a related patch set uploaded (by Ottomata):
Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9

https://gerrit.wikimedia.org/r/304821

gerritbot added a comment.Aug 15 2016, 2:17 PM

Change 304822 had a related patch set uploaded (by Ottomata):
Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9

https://gerrit.wikimedia.org/r/304822

Ottomata mentioned this in rOPUPbd9d086c1e84: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9.Aug 15 2016, 2:20 PM
Ottomata mentioned this in rOPUP408243f7b410: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9.
Ottomata mentioned this in rOPUP9ae819e4cdac: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9.Aug 15 2016, 2:29 PM
Ottomata mentioned this in rOPUP9a2dc83eb621: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9.
gerritbot added a comment.Aug 15 2016, 2:37 PM

Change 304827 had a related patch set uploaded (by Ottomata):
Remove now unused kafka module

https://gerrit.wikimedia.org/r/304827

Ottomata added a subscriber: • Pchelolo.Aug 15 2016, 2:40 PM

@Pchelolo we are ready to go for main-eqiad! Ping me when you are online so you can babysit change-prop.

Ottomata mentioned this in rOPUPf3e97ee237db: Remove now unused kafka module.Aug 15 2016, 2:43 PM
• mobrovac moved this task from Backlog to In Progress Before Value Streams Kickoff (August 15th) on the Event-Platform board.Aug 15 2016, 3:47 PM
gerritbot added a comment.Aug 15 2016, 6:15 PM

Change 304821 merged by Ottomata:
Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9

https://gerrit.wikimedia.org/r/304821

Ottomata mentioned this in rOPUP236f30a76529: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9.Aug 15 2016, 6:18 PM
Ottomata mentioned this in rOPUP5d412d2e337e: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9.Aug 15 2016, 6:22 PM
gerritbot added a comment.Aug 15 2016, 6:28 PM

Change 304822 merged by Ottomata:
Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9

https://gerrit.wikimedia.org/r/304822

gerritbot added a comment.Aug 15 2016, 8:17 PM

Change 304827 merged by Ottomata:
Remove now unused kafka module

https://gerrit.wikimedia.org/r/304827

Ottomata mentioned this in rOPUPea2329c6c0d9: Remove now unused kafka module.Aug 15 2016, 8:17 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Aug 15 2016, 8:17 PM

Done.

eventlogging-service-eventbus did drop a few events during broker restarts. GAHHHHH. We captured these events in a file and then replayed them manually after the upgrade was done.

I will try again to replicate this in labs.

• Nuria closed this task as Resolved.Aug 16 2016, 8:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL