Delete stale topics from main Kafka clusters
Closed, ResolvedPublic

Description

Now that we've deprecated many old EventBus topic names, we should delete them from main-eqiad Kafka clusters, so we don't have to think about managing them in the future.

Ottomata created this task.Oct 31 2016, 4:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 31 2016, 4:54 PM
Milimetric triaged this task as Low priority.
elukey added a subscriber: elukey.Dec 1 2016, 2:51 PM
Ottomata added a comment.EditedDec 1 2016, 3:00 PM

Topics to delete:

codfw.change-prop.retry.change-prop.backlinks.continue
eqiad.change-prop.retry.change-prop.transcludes.continue

codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
codfw.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
codfw.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.change-prop.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts


eqiad.mediawiki.resource-change
eqiad.resource-change
resource_change

change-prop.retry.mediawiki.page_delete
change-prop.retry.mediawiki.page_move
change-prop.retry.mediawiki.page_restore
change-prop.retry.mediawiki.revision_create
change-prop.retry.mediawiki.revision_visibility_set
change-prop.retry.mediawiki.user_block
change-prop.retry.resource_change
eqiad.change-prop.tanscludes.continue
elukey
eqiad.mediawiki.page-edit
eqiad.mediawiki.page_delete
eqiad.mediawiki.page_move
eqiad.mediawiki.page_restore
eqiad.mediawiki.resource_change
eqiad.mediawiki.resourcechange
eqiad.mediawiki.revision_create
eqiad.mediawiki.revision_visibility_set
eqiad.mediawiki.user_block
eqiad.revision_create
eqiaed.mediawiki.resource_change
mediawiki.page_delete
mediawiki.page_edit
mediawiki.page_move
mediawiki.page_restore
mediawiki.repage_move
mediawiki.revision_create
mediawiki.revision_visibility_set
mediawiki.user_block

test
test-1codfw.change-prop.backlinks.continue
test1
test2
test2-1

All of these topics had 0 log data stored when I looked. This list was reviewed by otto and petr on 2018-01-03.

Ottomata moved this task from Backlog to Kafka on the Analytics-Cluster board.Jul 11 2017, 3:10 PM
Ottomata edited projects, added Analytics-Kanban; removed Analytics.Jan 3 2018, 8:31 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-03T21:01:20Z] <ottomata> deleting stale topics from main kafka clusters: T149594

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Jan 3 2018, 9:30 PM
Ottomata claimed this task.

Done (mostly!) there are still some weird change-prop retry topics, but for the most part things are much better. We've deleted the older non DC prefixed eventbus topics, as well as some ones with typos, which were the ones I was most excited to get rid of.

Nuria closed this task as Resolved.Feb 12 2018, 3:56 PM