Maniphest T199510

Clean up leftover topics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• mobrovac
	Jul 13 2018, 8:00 AM

Project Tags

Referenced Files

None

Subscribers

Description

Following the Kafka outage which created a lot of topics, there are still some that need cleaning up:

eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.enqueue
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.securePollPopulateVoterList
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ThumbnailRender
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchLinksUpdatePrioritized
eqiad.change-prop.retry.changeprop.retry.change-prop.transcludes.resource-change
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.sendMail
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.RecordLintJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchCheckerJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageSubmitJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalRenameUserJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.GlobalUserPageLocalJobSubmitJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchIncomingLinkCount
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.recentChangesUpdate
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.BounceHandlerJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.translationNotificationJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.renameUser
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageServerSideJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cdnPurge
eqiad.change-prop.retry.changeprop.retry.mediawiki.revision-visibility-change
eqiad.change-prop.retry.changeprop.retry.change-prop.wikidata.resource-change
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CentralAuthCreateLocalAccountJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.synchroniseThreadArticleData
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.webVideoTranscode
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.categoryMembershipChange
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.EchoNotificationJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.updateBetaFeaturesUserCounts
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinksDynamic
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-properties-change
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.--domain
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TTMServerMessageUpdateJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.clearUserWatchlist
eqiad.change-prop.retry.changeprop.retry.resource_change
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.UpdateRepoOnDelete
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.wikibase-addUsagesForPage
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinksPrioritized
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslationsUpdateJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalSharedHelpPageCacheUpdateJob
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-create
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.flaggedrevs_CacheUpdate
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.AssembleUploadChunks
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.PublishStashedFile
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.constraintsTableUpdate
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.BounceHandlerNotificationJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslatablePageMoveJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchElasticaWrite
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchDeleteArchive
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CentralAuthUnattachUserJob
eqiad.change-prop.retry.cpjobqueue.retry.change-prop.partitioned.mediawiki.job.refreshLinks
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-move
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinks
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.enotifNotify
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslateRenderJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.globalUsageCachePurge
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetUploadMediafileJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchJobChecker
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MessageGroupStatesUpdaterJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetUploadMetadataJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CognateCacheUpdateJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.crosswikiSuppressUser
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalGlobalUserPageCacheUpdateJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchLinksUpdate
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.HTMLCacheUpdate
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CognateLocalJobSubmitJob
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-delete
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchMassIndex
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.deleteLinks
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.htmlCacheUpdate
eqiad.change-prop.retry.changeprop.retry.change-prop.backlinks.resource-change
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MessageIndexRebuildJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalPageMoveJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.wikibase-InjectRCRecords
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchOtherIndex
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.EchoNotificationDeleteJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.userGroupExpiry
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LoginNotifyChecks
eqiad.change-prop.retry.changeprop.retry.mediawiki.revision-create
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchDeletePages
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ChangeNotification
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.UpdateRepoOnMove
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-undelete
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslateDeleteJob
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.webVideoTranscodePrioritized

Event Timeline

• mobrovac created this task.Jul 13 2018, 8:00 AM

• mobrovac triaged this task as Medium priority.

Restricted Application added a project: Analytics. · View Herald TranscriptJul 13 2018, 8:00 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

elukey added a project: User-Elukey.Jul 13 2018, 8:01 AM

Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Jul 18 2018, 7:21 PM

Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.Jul 19 2018, 3:41 PM

Milimetric moved this task from Operational Excellence to Kafka Work on the Analytics board.

Just to be sure, here's the len of all the topics:

cat topics | while read line; do echo $line" "${#line}; done | sort -n -k 2
eqiad.change-prop.retry.changeprop.retry.resource_change 56
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-move 60
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-create 62
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-delete 62
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.enqueue 62
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.--domain 63
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cdnPurge 63
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.sendMail 63
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-undelete 64
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.renameUser 65
eqiad.change-prop.retry.changeprop.retry.mediawiki.revision-create 66
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.deleteLinks 66
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.enotifNotify 67
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinks 67
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.RecordLintJob 68
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageJob 69
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.HTMLCacheUpdate 70
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ThumbnailRender 70
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.htmlCacheUpdate 70
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.userGroupExpiry 70
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.BounceHandlerJob 71
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalPageMoveJob 71
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.UpdateRepoOnMove 71
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LoginNotifyChecks 72
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob 72
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.webVideoTranscode 72
eqiad.change-prop.retry.changeprop.retry.mediawiki.page-properties-change 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.ChangeNotification 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalRenameUserJob 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.PublishStashedFile 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslateDeleteJob 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslateRenderJob 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.UpdateRepoOnDelete 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.clearUserWatchlist 73
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.EchoNotificationJob 74
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.recentChangesUpdate 74
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinksDynamic 74
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.AssembleUploadChunks 75
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageSubmitJob 75
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CognateCacheUpdateJob 76
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslationsUpdateJob 76
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchMassIndex 76
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.crosswikiSuppressUser 76
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.globalUsageCachePurge 76
eqiad.change-prop.retry.changeprop.retry.change-prop.wikidata.resource-change 77
eqiad.change-prop.retry.changeprop.retry.mediawiki.revision-visibility-change 77
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MessageIndexRebuildJob 77
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchCheckerJob 77
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchJobChecker 77
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchOtherIndex 77
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.constraintsTableUpdate 77
eqiad.change-prop.retry.changeprop.retry.change-prop.backlinks.resource-change 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TranslatablePageMoveJob 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchDeletePages 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchLinksUpdate 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.flaggedrevs_CacheUpdate 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.refreshLinksPrioritized 78
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CognateLocalJobSubmitJob 79
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MassMessageServerSideJob 79
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.categoryMembershipChange 79
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.wikibase-InjectRCRecords 79
eqiad.change-prop.retry.changeprop.retry.change-prop.transcludes.resource-change 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.EchoNotificationDeleteJob 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.TTMServerMessageUpdateJob 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchDeleteArchive 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchElasticaWrite 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.wikibase-addUsagesForPage 80
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CentralAuthUnattachUserJob 81
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetUploadMetadataJob 81
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.translationNotificationJob 81
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetUploadMediafileJob 82
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.securePollPopulateVoterList 82
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.BounceHandlerNotificationJob 83
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.MessageGroupStatesUpdaterJob 83
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.synchroniseThreadArticleData 83
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.updateBetaFeaturesUserCounts 83
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.webVideoTranscodePrioritized 83
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchIncomingLinkCount 84
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.GlobalUserPageLocalJobSubmitJob 86
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.CentralAuthCreateLocalAccountJob 87
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalGlobalUserPageCacheUpdateJob 88
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.LocalSharedHelpPageCacheUpdateJob 88
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob 88
eqiad.change-prop.retry.cpjobqueue.retry.mediawiki.job.cirrusSearchLinksUpdatePrioritized 89
eqiad.change-prop.retry.cpjobqueue.retry.change-prop.partitioned.mediawiki.job.refreshLinks 91

As far as I can see from https://issues.apache.org/jira/browse/KAFKA-4893 the max "safe" topic len is 209 chars (to avoid hitting the bug), so we should be good to delete.

Mentioned in SAL (#wikimedia-operations) [2018-07-23T14:53:22Z] <elukey> delete empty/not-used/wrongly-created topics in Kafka main-eqiad - T199510

All topics deleted!

• mobrovac edited projects, added: Services (done); removed: Services (next).Jul 23 2018, 8:27 PM

Krinkle edited projects, added: Sustainability (Incident Followup); removed: Wikimedia-Incident.Apr 28 2020, 9:50 PM

Clean up leftover topicsClosed, ResolvedPublicActions

Description

Event Timeline

Clean up leftover topics
Closed, ResolvedPublic
Actions