Page MenuHomePhabricator
Create Task
Maniphest T288825

Rebalance kafka partitions in main-{eqiad,codfw} clusters
Closed, ResolvedPublic
Actions

Description

This task is about rebalancing the kafka partitions in main eqiad and codfw after the additional 2 brokers per cluster added some time ago. Kafka doesn't do it by itself, and for Kafka Jumbo we used a special tool that DataDog created called topicmappr. The idea is to:

  1. use topicmappr to generate a plan (namely json files) representing the moves (like topic X partition 1 from broker 1 to broker 2, etc..).
  2. use the kafka rebalance partition tool (that gets the above json files as input) to do the actual moves.

The final goal should be to distribute some of the load (traffic etc..) from 3 brokers only to 5 (the newer ones are not doing much at the moment).

Related Objects
Search...

Event Timeline

elukey created this task.Aug 13 2021, 2:29 PM
elukey added a comment.Aug 13 2021, 2:33 PM

Ran the following command to day on kafka-main2004:

./topicmappr rebuild --out-path /home/elukey/T225005/json --force-rebuild --zk-addr conf2004.codfw.wmnet --zk-prefix kafka/main-codfw --brokers -2 --topics '^(eqiad|codfw)\.resource-purge$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.cirrusSearchElasticaWrite$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchElasticaWrite$|^(eqiad|codfw)\.change-prop\.transcludes\.resource-change$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchLinksUpdate$|^(eqiad|codfw)\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.resource_change|__consumer_offsets$|^(eqiad|codfw)\.mediawiki\.job\.RecordLintJob$|^(eqiad|codfw)\.mediawiki\.job\.wikibase-addUsagesForPage$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchCheckerJob$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchIncomingLinkCount$|^(eqiad|codfw)\.mediawiki\.job\.htmlCacheUpdate$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.htmlCacheUpdate$|^(eqiad|codfw)\.mediawiki\.recentchange$|^(eqiad|codfw)\.mediawiki\.job\.cdnPurge$|^(eqiad|codfw)\.mediawiki\.job\.recentChangesUpdate$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchLinksUpdatePrioritized$|^(eqiad|codfw)\.mediawiki\.revision-create$|^(eqiad|codfw)\.mediawiki\.page-links-change$|^(eqiad|codfw)\.mediawiki\.job\.ChangeDeletionNotification$|^(eqiad|codfw)\.mediawiki\.job\.categoryMembershipChange$|^(eqiad|codfw)\.mediawiki\.revision-score$|^(eqiad|codfw)\.mediawiki\.revision-tags-change$|^(eqiad|codfw)\.mediawiki\.job\.EchoNotificationDeleteJob$|^(eqiad|codfw)\.mediawiki\.job\.ORESFetchScoreJob$|^(eqiad|codfw)\.mediawiki\.job\.constraintsRunCheck$|^(eqiad|codfw)\.rdf-streaming-updater\.mutation$|^(eqiad|codfw)\.mediawiki\.page-properties-change$|^(eqiad|codfw)\.mediawiki\.job\.ThumbnailRender$|^(eqiad|codfw)\.mediawiki\.user-blocks-change$|^(eqiad|codfw)\.mediawiki\.job\.enotifNotify$|^(eqiad|codfw)\.mediawiki\.page-create$|^(eqiad|codfw)\.mediawiki\.job\.activityUpdateJob$|^(eqiad|codfw)\.mediawiki\.job\.ChangeNotification$|^(eqiad|codfw)\.mediawiki\.job\.CleanTermsIfUnused$|^(eqiad|codfw)\.mediawiki\.job\.wikibase-InjectRCRecords$|^(eqiad|codfw)\.mediawiki\.job\.globalUsageCachePurge$|^(eqiad|codfw)\.mediawiki\.job\.CognateCacheUpdateJob$|^(eqiad|codfw)\.change-prop\.backlinks\.resource-change$|^(eqiad|codfw)\.cpjobqueue\.retry\.mediawiki\.job\.categoryMembershipChange$|^(eqiad|codfw)\.mediawiki\.job\.watchlistExpiry$|^(eqiad|codfw)\.mediawiki\.job\.fetchGoogleCloudVisionAnnotations$|^(eqiad|codfw)\.mediawiki\.job\.flaggedrevs_CacheUpdate$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchOtherIndex$|^(eqiad|codfw)\.mediawiki\.job\.CognateLocalJobSubmitJob$|^(eqiad|codfw)\.mediawiki\.job\.webVideoTranscodePrioritized$|^(eqiad|codfw)\.eventgate-main\.test\.event$|^(eqiad|codfw)\.eventgate-main\.test\.event$|^(eqiad|codfw)\.change-prop\.wikidata\.resource-change$|^(eqiad|codfw)\.mediawiki\.job\.LoginNotifyChecks$|^(eqiad|codfw)\.mediawiki\.job\.CentralAuthCreateLocalAccountJob$|^(eqiad|codfw)\.mediawiki\.job\.refreshLinksPrioritized$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchDeletePages$|^(eqiad|codfw)\.mediawiki\.job\.revertedTagUpdate$|^(eqiad|codfw)\.mediawiki\.page-move$|^(eqiad|codfw)\.mediawiki\.job\.updateBetaFeaturesUserCounts$|^(eqiad|codfw)\.changeprop\.error$|^(eqiad|codfw)\.cpjobqueue\.retry\.mediawiki\.job\.ORESFetchScoreJob$|^(eqiad|codfw)\.mediawiki\.job\.newcomerTasksCacheRefreshJob$|^(eqiad|codfw)\.mediawiki\.job\.UpdateRepoOnMove$|^(eqiad|codfw)\.mediawiki\.revision-recommendation-create$|^(eqiad|codfw)\.mediawiki\.page-delete$|^(eqiad|codfw)\.mediawiki\.job\.TTMServerMessageUpdateJob$|^(eqiad|codfw)\.mediawiki\.job\.newUserMessageJob$|^(eqiad|codfw)\.mediawiki\.job\.TranslateRenderJob$|^(eqiad|codfw)\.mediawiki\.job\.AssembleUploadChunks$|^(eqiad|codfw)\.mediawiki\.job\.MessageGroupStatesUpdaterJob$|^(eqiad|codfw)\.cpjobqueue\.retry\.cpjobqueue\.partitioned\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.mediawiki\.job\.userOptionsUpdate$|^(eqiad|codfw)\.mediawiki\.job\.EchoPushNotificationRequest$|^(eqiad|codfw)\.mediawiki\.job\.UpdateRepoOnDelete$|^(eqiad|codfw)\.mediawiki\.revision-score$|^(eqiad|codfw)\.cpjobqueue\.error$|^(eqiad|codfw)\.changeprop\.retry\.change-prop\.backlinks\.resource-change$|^(eqiad|codfw)\.rdf-streaming-updater\.state-inconsistency$'

The list of topics comes from: https://thanos.wikimedia.org/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sort_desc(topk(1000%2C sum(irate(kafka_server_BrokerTopicMetrics_MessagesIn_total{kafka_cluster%3D"main-codfw"%2C topic%3D~".%2B"}[5m])) by (topic)) > 0)&g0.tab=1

The idea is to work on topics that are getting traffic (there are some basically with zero msg/s) including also their eqiad counterpart.

And the output was:

./topicmappr_cmd 

Topics:
  __consumer_offsets
  codfw.change-prop.backlinks.resource-change
  codfw.change-prop.transcludes.resource-change
  codfw.change-prop.wikidata.resource-change
  codfw.changeprop.error
  codfw.changeprop.retry.change-prop.backlinks.resource-change
  codfw.cpjobqueue.error
  codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob
  codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange
  codfw.eventgate-main.test.event
  codfw.mediawiki.job.AssembleUploadChunks
  codfw.mediawiki.job.CentralAuthCreateLocalAccountJob
  codfw.mediawiki.job.ChangeDeletionNotification
  codfw.mediawiki.job.ChangeNotification
  codfw.mediawiki.job.CleanTermsIfUnused
  codfw.mediawiki.job.CognateCacheUpdateJob
  codfw.mediawiki.job.CognateLocalJobSubmitJob
  codfw.mediawiki.job.EchoNotificationDeleteJob
  codfw.mediawiki.job.EchoPushNotificationRequest
  codfw.mediawiki.job.LoginNotifyChecks
  codfw.mediawiki.job.MessageGroupStatesUpdaterJob
  codfw.mediawiki.job.ORESFetchScoreJob
  codfw.mediawiki.job.RecordLintJob
  codfw.mediawiki.job.TTMServerMessageUpdateJob
  codfw.mediawiki.job.ThumbnailRender
  codfw.mediawiki.job.TranslateRenderJob
  codfw.mediawiki.job.UpdateRepoOnDelete
  codfw.mediawiki.job.UpdateRepoOnMove
  codfw.mediawiki.job.activityUpdateJob
  codfw.mediawiki.job.categoryMembershipChange
  codfw.mediawiki.job.cdnPurge
  codfw.mediawiki.job.cirrusSearchCheckerJob
  codfw.mediawiki.job.cirrusSearchDeletePages
  codfw.mediawiki.job.cirrusSearchElasticaWrite
  codfw.mediawiki.job.cirrusSearchIncomingLinkCount
  codfw.mediawiki.job.cirrusSearchLinksUpdate
  codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized
  codfw.mediawiki.job.cirrusSearchOtherIndex
  codfw.mediawiki.job.constraintsRunCheck
  codfw.mediawiki.job.enotifNotify
  codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations
  codfw.mediawiki.job.flaggedrevs_CacheUpdate
  codfw.mediawiki.job.globalUsageCachePurge
  codfw.mediawiki.job.htmlCacheUpdate
  codfw.mediawiki.job.newUserMessageJob
  codfw.mediawiki.job.newcomerTasksCacheRefreshJob
  codfw.mediawiki.job.recentChangesUpdate
  codfw.mediawiki.job.refreshLinks
  codfw.mediawiki.job.refreshLinksPrioritized
  codfw.mediawiki.job.revertedTagUpdate
  codfw.mediawiki.job.updateBetaFeaturesUserCounts
  codfw.mediawiki.job.userOptionsUpdate
  codfw.mediawiki.job.watchlistExpiry
  codfw.mediawiki.job.webVideoTranscodePrioritized
  codfw.mediawiki.job.wikibase-InjectRCRecords
  codfw.mediawiki.job.wikibase-addUsagesForPage
  codfw.mediawiki.page-create
  codfw.mediawiki.page-delete
  codfw.mediawiki.page-links-change
  codfw.mediawiki.page-move
  codfw.mediawiki.page-properties-change
  codfw.mediawiki.recentchange
  codfw.mediawiki.revision-create
  codfw.mediawiki.revision-recommendation-create
  codfw.mediawiki.revision-score
  codfw.mediawiki.revision-tags-change
  codfw.mediawiki.user-blocks-change
  codfw.rdf-streaming-updater.mutation
  codfw.rdf-streaming-updater.state-inconsistency
  codfw.resource-purge
  codfw.resource_change
  eqiad.change-prop.backlinks.resource-change
  eqiad.change-prop.transcludes.resource-change
  eqiad.change-prop.wikidata.resource-change
  eqiad.changeprop.error
  eqiad.changeprop.retry.change-prop.backlinks.resource-change
  eqiad.cpjobqueue.error
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob
  eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange
  eqiad.eventgate-main.test.event
  eqiad.mediawiki.job.AssembleUploadChunks
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob
  eqiad.mediawiki.job.ChangeDeletionNotification
  eqiad.mediawiki.job.ChangeNotification
  eqiad.mediawiki.job.CleanTermsIfUnused
  eqiad.mediawiki.job.CognateCacheUpdateJob
  eqiad.mediawiki.job.CognateLocalJobSubmitJob
  eqiad.mediawiki.job.EchoNotificationDeleteJob
  eqiad.mediawiki.job.EchoPushNotificationRequest
  eqiad.mediawiki.job.LoginNotifyChecks
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob
  eqiad.mediawiki.job.ORESFetchScoreJob
  eqiad.mediawiki.job.RecordLintJob
  eqiad.mediawiki.job.TTMServerMessageUpdateJob
  eqiad.mediawiki.job.ThumbnailRender
  eqiad.mediawiki.job.TranslateRenderJob
  eqiad.mediawiki.job.UpdateRepoOnDelete
  eqiad.mediawiki.job.UpdateRepoOnMove
  eqiad.mediawiki.job.activityUpdateJob
  eqiad.mediawiki.job.categoryMembershipChange
  eqiad.mediawiki.job.cdnPurge
  eqiad.mediawiki.job.cirrusSearchCheckerJob
  eqiad.mediawiki.job.cirrusSearchDeletePages
  eqiad.mediawiki.job.cirrusSearchElasticaWrite
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount
  eqiad.mediawiki.job.cirrusSearchLinksUpdate
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized
  eqiad.mediawiki.job.cirrusSearchOtherIndex
  eqiad.mediawiki.job.constraintsRunCheck
  eqiad.mediawiki.job.enotifNotify
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate
  eqiad.mediawiki.job.globalUsageCachePurge
  eqiad.mediawiki.job.htmlCacheUpdate
  eqiad.mediawiki.job.newUserMessageJob
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob
  eqiad.mediawiki.job.recentChangesUpdate
  eqiad.mediawiki.job.refreshLinks
  eqiad.mediawiki.job.refreshLinksPrioritized
  eqiad.mediawiki.job.revertedTagUpdate
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts
  eqiad.mediawiki.job.userOptionsUpdate
  eqiad.mediawiki.job.watchlistExpiry
  eqiad.mediawiki.job.webVideoTranscodePrioritized
  eqiad.mediawiki.job.wikibase-InjectRCRecords
  eqiad.mediawiki.job.wikibase-addUsagesForPage
  eqiad.mediawiki.page-create
  eqiad.mediawiki.page-delete
  eqiad.mediawiki.page-links-change
  eqiad.mediawiki.page-move
  eqiad.mediawiki.page-properties-change
  eqiad.mediawiki.recentchange
  eqiad.mediawiki.revision-create
  eqiad.mediawiki.revision-recommendation-create
  eqiad.mediawiki.revision-score
  eqiad.mediawiki.revision-tags-change
  eqiad.mediawiki.user-blocks-change
  eqiad.rdf-streaming-updater.mutation
  eqiad.rdf-streaming-updater.state-inconsistency
  eqiad.resource-purge
  eqiad.resource_change

Broker change summary:
  New broker 2005
  New broker 2004
  -
  Replacing 0, added 2, missing 0, total count changed by 2

Action:
  Expanding/rebalancing topic with 2 additional broker(s) (this is a no-op unless --force-rebuild is specified)
  Force rebuilding map

Partition map changes:
  __consumer_offsets p0: [2001 2002 2003] -> [2003 2002 2004] replaced broker
  __consumer_offsets p1: [2002 2003 2001] -> [2001 2005 2002] replaced broker
  __consumer_offsets p2: [2003 2001 2002] -> [2002 2004 2001] replaced broker
  __consumer_offsets p3: [2001 2003 2002] -> [2004 2003 2002] replaced broker
  __consumer_offsets p4: [2002 2001 2003] -> [2005 2001 2003] replaced broker
  __consumer_offsets p5: [2003 2002 2001] -> [2003 2002 2005] replaced broker
  __consumer_offsets p6: [2001 2002 2003] -> [2001 2004 2003] replaced broker
  __consumer_offsets p7: [2002 2003 2001] -> [2002 2005 2001] replaced broker
  __consumer_offsets p8: [2003 2001 2002] -> [2004 2003 2001] replaced broker
  __consumer_offsets p9: [2001 2003 2002] -> [2005 2001 2002] replaced broker
  __consumer_offsets p10: [2002 2001 2003] -> [2003 2005 2001] replaced broker
  __consumer_offsets p11: [2003 2002 2001] -> [2001 2002 2004] replaced broker
  __consumer_offsets p12: [2001 2002 2003] -> [2002 2004 2003] replaced broker
  __consumer_offsets p13: [2002 2003 2001] -> [2004 2001 2003] replaced broker
  __consumer_offsets p14: [2003 2001 2002] -> [2005 2002 2001] replaced broker
  __consumer_offsets p15: [2001 2003 2002] -> [2003 2004 2002] replaced broker
  __consumer_offsets p16: [2002 2001 2003] -> [2001 2003 2005] replaced broker
  __consumer_offsets p17: [2003 2002 2001] -> [2002 2005 2003] replaced broker
  __consumer_offsets p18: [2001 2002 2003] -> [2004 2001 2002] replaced broker
  __consumer_offsets p19: [2002 2003 2001] -> [2005 2003 2001] replaced broker
  __consumer_offsets p20: [2003 2001 2002] -> [2003 2002 2004] replaced broker
  __consumer_offsets p21: [2001 2003 2002] -> [2001 2005 2002] replaced broker
  __consumer_offsets p22: [2002 2001 2003] -> [2002 2004 2003] replaced broker
  __consumer_offsets p23: [2003 2002 2001] -> [2004 2001 2003] replaced broker
  __consumer_offsets p24: [2001 2002 2003] -> [2005 2002 2001] replaced broker
  __consumer_offsets p25: [2002 2003 2001] -> [2003 2005 2002] replaced broker
  __consumer_offsets p26: [2003 2001 2002] -> [2001 2004 2003] replaced broker
  __consumer_offsets p27: [2001 2003 2002] -> [2002 2003 2004] replaced broker
  __consumer_offsets p28: [2002 2001 2003] -> [2004 2001 2002] replaced broker
  __consumer_offsets p29: [2003 2002 2001] -> [2005 2003 2001] replaced broker
  __consumer_offsets p30: [2001 2002 2003] -> [2003 2002 2005] replaced broker
  __consumer_offsets p31: [2002 2003 2001] -> [2001 2004 2003] replaced broker
  __consumer_offsets p32: [2003 2001 2002] -> [2002 2005 2001] replaced broker
  __consumer_offsets p33: [2001 2003 2002] -> [2004 2001 2002] replaced broker
  __consumer_offsets p34: [2002 2001 2003] -> [2005 2003 2002] replaced broker
  __consumer_offsets p35: [2003 2002 2001] -> [2003 2005 2001] replaced broker
  __consumer_offsets p36: [2001 2002 2003] -> [2001 2002 2004] replaced broker
  __consumer_offsets p37: [2002 2003 2001] -> [2002 2004 2003] replaced broker
  __consumer_offsets p38: [2003 2001 2002] -> [2004 2002 2001] replaced broker
  __consumer_offsets p39: [2001 2003 2002] -> [2005 2001 2003] replaced broker
  __consumer_offsets p40: [2002 2001 2003] -> [2003 2004 2002] replaced broker
  __consumer_offsets p41: [2003 2002 2001] -> [2001 2005 2002] replaced broker
  __consumer_offsets p42: [2001 2002 2003] -> [2002 2003 2005] replaced broker
  __consumer_offsets p43: [2002 2003 2001] -> [2004 2002 2001] replaced broker
  __consumer_offsets p44: [2003 2001 2002] -> [2005 2003 2001] replaced broker
  __consumer_offsets p45: [2001 2003 2002] -> [2003 2001 2004] replaced broker
  __consumer_offsets p46: [2002 2001 2003] -> [2001 2005 2003] replaced broker
  __consumer_offsets p47: [2003 2002 2001] -> [2002 2004 2003] replaced broker
  __consumer_offsets p48: [2001 2002 2003] -> [2004 2003 2002] replaced broker
  __consumer_offsets p49: [2002 2003 2001] -> [2005 2002 2003] replaced broker
  codfw.change-prop.backlinks.resource-change p0: [2003 2001 2002] -> [2003 2001 2005] replaced broker
  codfw.change-prop.transcludes.resource-change p0: [2002 2003 2001] -> [2001 2004 2002] replaced broker
  codfw.change-prop.wikidata.resource-change p0: [2001 2002 2003] -> [2002 2005 2001] replaced broker
  codfw.changeprop.error p0: [2001 2003 2002] -> [2004 2002 2001] replaced broker
  codfw.changeprop.retry.change-prop.backlinks.resource-change p0: [2003 2001 2002] -> [2005 2003 2002] replaced broker
  codfw.cpjobqueue.error p0: [2002 2003 2001] -> [2003 2005 2002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p0: [2001 2002 2003] -> [2001 2004 2003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p1: [2002 2003 2001] -> [2002 2001 2005] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p2: [2003 2001 2002] -> [2004 2002 2003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p0: [2003 2001 2002] -> [2005 2003 2001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p1: [2001 2002 2003] -> [2003 2005 2001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p2: [2002 2003 2001] -> [2001 2004 2002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p3: [2003 2002 2001] -> [2002 2001 2004] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p4: [2001 2003 2002] -> [2004 2002 2003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p5: [2002 2001 2003] -> [2005 2003 2001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p6: [2003 2001 2002] -> [2003 2001 2005] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p7: [2001 2002 2003] -> [2001 2005 2003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [2001 2003 2002] -> [2002 2004 2003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p1: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p2: [2003 2002 2001] -> [2005 2003 2002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p3: [2001 2002 2003] -> [2003 2001 2004] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p4: [2002 2003 2001] -> [2001 2005 2002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p5: [2003 2001 2002] -> [2002 2004 2001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p6: [2001 2003 2002] -> [2004 2001 2002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p7: [2002 2001 2003] -> [2005 2003 2002] replaced broker
  codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [2001 2002 2003] -> [2003 2002 2005] replaced broker
  codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob p0: [2002 2003 2001] -> [2001 2004 2003] replaced broker
  codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange p0: [2001 2002 2003] -> [2002 2005 2003] replaced broker
  codfw.eventgate-main.test.event p0: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  codfw.mediawiki.job.AssembleUploadChunks p0: [2001 2003 2002] -> [2005 2003 2002] replaced broker
  codfw.mediawiki.job.CentralAuthCreateLocalAccountJob p0: [2003 2002 2001] -> [2003 2004 2001] replaced broker
  codfw.mediawiki.job.ChangeDeletionNotification p0: [2002 2003 2001] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.job.ChangeNotification p0: [2002 2003 2001] -> [2002 2001 2004] replaced broker
  codfw.mediawiki.job.CleanTermsIfUnused p0: [2001 2003 2002] -> [2004 2001 2003] replaced broker
  codfw.mediawiki.job.CognateCacheUpdateJob p0: [2002 2003 2001] -> [2005 2002 2001] replaced broker
  codfw.mediawiki.job.CognateLocalJobSubmitJob p0: [2001 2002 2003] -> [2003 2004 2002] replaced broker
  codfw.mediawiki.job.EchoNotificationDeleteJob p0: [2002 2003 2001] -> [2001 2003 2004] replaced broker
  codfw.mediawiki.job.EchoPushNotificationRequest p0: [2002 2003 2001] -> [2002 2005 2001] replaced broker
  codfw.mediawiki.job.LoginNotifyChecks p0: [2002 2001 2003] -> [2004 2001 2002] replaced broker
  codfw.mediawiki.job.MessageGroupStatesUpdaterJob p0: [2003 2001 2002] -> [2005 2003 2001] replaced broker
  codfw.mediawiki.job.ORESFetchScoreJob p0: [2003 2002 2001] -> [2003 2004 2002] replaced broker
  codfw.mediawiki.job.RecordLintJob p0: [2003 2002 2001] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.job.TTMServerMessageUpdateJob p0: [2003 2002 2001] -> [2002 2004 2003] replaced broker
  codfw.mediawiki.job.ThumbnailRender p0: [2003 2001 2002] -> [2004 2002 2003] replaced broker
  codfw.mediawiki.job.TranslateRenderJob p0: [2003 2001 2002] -> [2005 2002 2001] replaced broker
  codfw.mediawiki.job.UpdateRepoOnDelete p0: [2003 2002 2001] -> [2003 2005 2002] replaced broker
  codfw.mediawiki.job.UpdateRepoOnMove p0: [2002 2001 2003] -> [2001 2003 2005] replaced broker
  codfw.mediawiki.job.activityUpdateJob p0: [2003 2001 2002] -> [2002 2001 2004] replaced broker
  codfw.mediawiki.job.categoryMembershipChange p0: [2003 2002 2001] -> [2004 2001 2002] replaced broker
  codfw.mediawiki.job.cdnPurge p0: [2001 2002 2003] -> [2005 2003 2001] replaced broker
  codfw.mediawiki.job.cirrusSearchCheckerJob p0: [2001 2002 2003] -> [2003 2005 2002] replaced broker
  codfw.mediawiki.job.cirrusSearchDeletePages p0: [2001 2003 2002] -> [2001 2002 2005] replaced broker
  codfw.mediawiki.job.cirrusSearchElasticaWrite p0: [2003 2001 2002] -> [2002 2004 2003] replaced broker
  codfw.mediawiki.job.cirrusSearchElasticaWrite p1: [2001 2002 2003] -> [2004 2002 2003] replaced broker
  codfw.mediawiki.job.cirrusSearchElasticaWrite p2: [2002 2003 2001] -> [2005 2003 2001] replaced broker
  codfw.mediawiki.job.cirrusSearchIncomingLinkCount p0: [2003 2001 2002] -> [2003 2004 2001] replaced broker
  codfw.mediawiki.job.cirrusSearchLinksUpdate p0: [2001 2002 2003] -> [2001 2005 2002] replaced broker
  codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized p0: [2002 2003 2001] -> [2002 2001 2004] replaced broker
  codfw.mediawiki.job.cirrusSearchOtherIndex p0: [2001 2003 2002] -> [2004 2002 2003] replaced broker
  codfw.mediawiki.job.constraintsRunCheck p0: [2002 2003 2001] -> [2005 2001 2002] replaced broker
  codfw.mediawiki.job.enotifNotify p0: [2003 2001 2002] -> [2003 2004 2001] replaced broker
  codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations p0: [2003 2001 2002] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.job.flaggedrevs_CacheUpdate p0: [2001 2002 2003] -> [2002 2003 2005] replaced broker
  codfw.mediawiki.job.globalUsageCachePurge p0: [2002 2003 2001] -> [2004 2001 2003] replaced broker
  codfw.mediawiki.job.htmlCacheUpdate p0: [2003 2002 2001] -> [2005 2003 2001] replaced broker
  codfw.mediawiki.job.newUserMessageJob p0: [2001 2002 2003] -> [2003 2004 2002] replaced broker
  codfw.mediawiki.job.newcomerTasksCacheRefreshJob p0: [2002 2003 2001] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.job.recentChangesUpdate p0: [2003 2002 2001] -> [2002 2003 2005] replaced broker
  codfw.mediawiki.job.refreshLinks p0: [2003 2001 2002] -> [2004 2002 2001] replaced broker
  codfw.mediawiki.job.refreshLinksPrioritized p0: [2001 2003 2002] -> [2005 2002 2003] replaced broker
  codfw.mediawiki.job.revertedTagUpdate p0: [2003 2002 2001] -> [2003 2005 2002] replaced broker
  codfw.mediawiki.job.updateBetaFeaturesUserCounts p0: [2001 2003 2002] -> [2001 2004 2002] replaced broker
  codfw.mediawiki.job.userOptionsUpdate p0: [2002 2003 2001] -> [2002 2001 2004] replaced broker
  codfw.mediawiki.job.watchlistExpiry p0: [2001 2002 2003] -> [2004 2003 2001] replaced broker
  codfw.mediawiki.job.webVideoTranscodePrioritized p0: [2002 2003 2001] -> [2005 2001 2002] replaced broker
  codfw.mediawiki.job.wikibase-InjectRCRecords p0: [2001 2002 2003] -> [2003 2004 2001] replaced broker
  codfw.mediawiki.job.wikibase-addUsagesForPage p0: [2002 2001 2003] -> [2001 2002 2005] replaced broker
  codfw.mediawiki.page-create p0: [2002 2003 2001] -> [2002 2005 2003] replaced broker
  codfw.mediawiki.page-delete p0: [2003 2002 2001] -> [2004 2003 2001] replaced broker
  codfw.mediawiki.page-links-change p0: [2002 2003 2001] -> [2005 2002 2003] replaced broker
  codfw.mediawiki.page-move p0: [2001 2003 2002] -> [2003 2004 2002] replaced broker
  codfw.mediawiki.page-properties-change p0: [2002 2003 2001] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.recentchange p0: [2003 2002 2001] -> [2002 2001 2004] replaced broker
  codfw.mediawiki.revision-create p0: [2003 2002 2001] -> [2004 2003 2001] replaced broker
  codfw.mediawiki.revision-recommendation-create p0: [2003 2001 2002] -> [2005 2001 2002] replaced broker
  codfw.mediawiki.revision-score p0: [2002 2001 2003] -> [2003 2004 2001] replaced broker
  codfw.mediawiki.revision-tags-change p0: [2003 2001 2002] -> [2001 2005 2003] replaced broker
  codfw.mediawiki.user-blocks-change p0: [2002 2003 2001] -> [2002 2003 2005] replaced broker
  codfw.rdf-streaming-updater.mutation p0: [2002 2001 2003] -> [2004 2002 2003] replaced broker
  codfw.rdf-streaming-updater.state-inconsistency p0: [2001 2003 2002] -> [2005 2002 2001] replaced broker
  codfw.resource-purge p0: [2003 2001 2002] -> [2003 2005 2002] replaced broker
  codfw.resource-purge p1: [2001 2002 2003] -> [2001 2004 2002] replaced broker
  codfw.resource-purge p2: [2002 2003 2001] -> [2002 2001 2004] replaced broker
  codfw.resource_change p0: [2003 2002 2001] -> [2004 2001 2002] replaced broker
  codfw.resource_change p1: [2001 2002 2003] -> [2005 2002 2001] replaced broker
  codfw.resource_change p2: [2002 2003 2001] -> [2003 2005 2002] replaced broker
  eqiad.change-prop.backlinks.resource-change p0: [2001 2002 2003] -> [2001 2004 2003] replaced broker
  eqiad.change-prop.transcludes.resource-change p0: [2001 2002 2003] -> [2002 2003 2005] replaced broker
  eqiad.change-prop.wikidata.resource-change p0: [2003 2002 2001] -> [2004 2003 2001] replaced broker
  eqiad.changeprop.error p0: [2003 2002 2001] -> [2005 2001 2003] replaced broker
  eqiad.changeprop.retry.change-prop.backlinks.resource-change p0: [2002 2003 2001] -> [2003 2004 2002] replaced broker
  eqiad.cpjobqueue.error p0: [2002 2003 2001] -> [2001 2002 2004] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p0: [2002 2003 2001] -> [2002 2005 2001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p1: [2003 2001 2002] -> [2004 2003 2001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p2: [2001 2002 2003] -> [2005 2001 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p0: [2002 2003 2001] -> [2003 2004 2002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p1: [2003 2001 2002] -> [2001 2002 2004] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p2: [2001 2002 2003] -> [2002 2005 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p3: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p4: [2003 2002 2001] -> [2005 2001 2002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p5: [2001 2003 2002] -> [2003 2004 2002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p6: [2002 2003 2001] -> [2001 2005 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p7: [2003 2001 2002] -> [2002 2003 2005] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [2003 2001 2002] -> [2004 2001 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p1: [2001 2002 2003] -> [2005 2003 2001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p2: [2002 2003 2001] -> [2003 2005 2002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p3: [2003 2002 2001] -> [2001 2004 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p4: [2001 2003 2002] -> [2002 2001 2004] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p5: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p6: [2003 2001 2002] -> [2005 2002 2003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p7: [2001 2002 2003] -> [2003 2005 2002] replaced broker
  eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [2003 2001 2002] -> [2001 2003 2005] replaced broker
  eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob p0: [2002 2001 2003] -> [2002 2004 2001] replaced broker
  eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange p0: [2002 2001 2003] -> [2004 2003 2001] replaced broker
  eqiad.eventgate-main.test.event p0: [2001 2003 2002] -> [2005 2001 2003] replaced broker
  eqiad.mediawiki.job.AssembleUploadChunks p0: [2001 2002 2003] -> [2003 2004 2002] replaced broker
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob p0: [2001 2003 2002] -> [2001 2002 2004] replaced broker
  eqiad.mediawiki.job.ChangeDeletionNotification p0: [2002 2001 2003] -> [2002 2005 2003] replaced broker
  eqiad.mediawiki.job.ChangeNotification p0: [2003 2001 2002] -> [2004 2003 2002] replaced broker
  eqiad.mediawiki.job.CleanTermsIfUnused p0: [2001 2003 2002] -> [2005 2001 2003] replaced broker
  eqiad.mediawiki.job.CognateCacheUpdateJob p0: [2003 2001 2002] -> [2003 2002 2005] replaced broker
  eqiad.mediawiki.job.CognateLocalJobSubmitJob p0: [2002 2003 2001] -> [2001 2005 2002] replaced broker
  eqiad.mediawiki.job.EchoNotificationDeleteJob p0: [2001 2002 2003] -> [2002 2004 2001] replaced broker
  eqiad.mediawiki.job.EchoPushNotificationRequest p0: [2003 2001 2002] -> [2004 2003 2001] replaced broker
  eqiad.mediawiki.job.LoginNotifyChecks p0: [2001 2003 2002] -> [2005 2002 2003] replaced broker
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob p0: [2001 2002 2003] -> [2003 2001 2004] replaced broker
  eqiad.mediawiki.job.ORESFetchScoreJob p0: [2003 2001 2002] -> [2001 2005 2002] replaced broker
  eqiad.mediawiki.job.RecordLintJob p0: [2003 2001 2002] -> [2002 2004 2001] replaced broker
  eqiad.mediawiki.job.TTMServerMessageUpdateJob p0: [2003 2001 2002] -> [2004 2002 2003] replaced broker
  eqiad.mediawiki.job.ThumbnailRender p0: [2001 2002 2003] -> [2005 2001 2002] replaced broker
  eqiad.mediawiki.job.TranslateRenderJob p0: [2003 2001 2002] -> [2003 2005 2001] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnDelete p0: [2003 2001 2002] -> [2001 2004 2003] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnMove p0: [2001 2002 2003] -> [2002 2003 2005] replaced broker
  eqiad.mediawiki.job.activityUpdateJob p0: [2003 2001 2002] -> [2004 2003 2002] replaced broker
  eqiad.mediawiki.job.categoryMembershipChange p0: [2001 2002 2003] -> [2005 2001 2003] replaced broker
  eqiad.mediawiki.job.cdnPurge p0: [2002 2001 2003] -> [2003 2005 2001] replaced broker
  eqiad.mediawiki.job.cirrusSearchCheckerJob p0: [2002 2003 2001] -> [2001 2002 2005] replaced broker
  eqiad.mediawiki.job.cirrusSearchDeletePages p0: [2002 2003 2001] -> [2002 2004 2001] replaced broker
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p0: [2001 2002 2003] -> [2004 2003 2002] replaced broker
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p1: [2002 2003 2001] -> [2005 2001 2003] replaced broker
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p2: [2003 2001 2002] -> [2003 2005 2001] replaced broker
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount p0: [2001 2002 2003] -> [2001 2002 2004] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdate p0: [2001 2002 2003] -> [2002 2004 2003] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized p0: [2002 2003 2001] -> [2004 2002 2001] replaced broker
  eqiad.mediawiki.job.cirrusSearchOtherIndex p0: [2003 2002 2001] -> [2005 2001 2002] replaced broker
  eqiad.mediawiki.job.constraintsRunCheck p0: [2003 2001 2002] -> [2003 2004 2002] replaced broker
  eqiad.mediawiki.job.enotifNotify p0: [2003 2001 2002] -> [2001 2005 2003] replaced broker
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations p0: [2002 2003 2001] -> [2002 2003 2005] replaced broker
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate p0: [2002 2003 2001] -> [2004 2001 2002] replaced broker
  eqiad.mediawiki.job.globalUsageCachePurge p0: [2003 2001 2002] -> [2005 2003 2001] replaced broker
  eqiad.mediawiki.job.htmlCacheUpdate p0: [2003 2002 2001] -> [2003 2002 2004] replaced broker
  eqiad.mediawiki.job.newUserMessageJob p0: [2001 2003 2002] -> [2001 2005 2003] replaced broker
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob p0: [2002 2003 2001] -> [2002 2004 2001] replaced broker
  eqiad.mediawiki.job.recentChangesUpdate p0: [2003 2001 2002] -> [2004 2001 2002] replaced broker
  eqiad.mediawiki.job.refreshLinks p0: [2003 2001 2002] -> [2005 2002 2001] replaced broker
  eqiad.mediawiki.job.refreshLinksPrioritized p0: [2002 2003 2001] -> [2003 2005 2002] replaced broker
  eqiad.mediawiki.job.revertedTagUpdate p0: [2003 2001 2002] -> [2001 2004 2003] replaced broker
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts p0: [2002 2001 2003] -> [2002 2003 2004] replaced broker
  eqiad.mediawiki.job.userOptionsUpdate p0: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  eqiad.mediawiki.job.watchlistExpiry p0: [2003 2001 2002] -> [2005 2001 2002] replaced broker
  eqiad.mediawiki.job.webVideoTranscodePrioritized p0: [2002 2003 2001] -> [2003 2005 2001] replaced broker
  eqiad.mediawiki.job.wikibase-InjectRCRecords p0: [2002 2003 2001] -> [2001 2003 2005] replaced broker
  eqiad.mediawiki.job.wikibase-addUsagesForPage p0: [2001 2002 2003] -> [2002 2004 2003] replaced broker
  eqiad.mediawiki.page-create p0: [2001 2002 2003] -> [2004 2002 2003] replaced broker
  eqiad.mediawiki.page-delete p0: [2001 2002 2003] -> [2005 2001 2002] replaced broker
  eqiad.mediawiki.page-links-change p0: [2001 2002 2003] -> [2003 2004 2001] replaced broker
  eqiad.mediawiki.page-move p0: [2003 2001 2002] -> [2001 2003 2004] replaced broker
  eqiad.mediawiki.page-properties-change p0: [2003 2001 2002] -> [2002 2005 2003] replaced broker
  eqiad.mediawiki.recentchange p0: [2001 2003 2002] -> [2004 2002 2001] replaced broker
  eqiad.mediawiki.revision-create p0: [2002 2003 2001] -> [2005 2001 2002] replaced broker
  eqiad.mediawiki.revision-recommendation-create p0: [2001 2003 2002] -> [2003 2004 2002] replaced broker
  eqiad.mediawiki.revision-score p0: [2003 2001 2002] -> [2001 2005 2003] replaced broker
  eqiad.mediawiki.revision-tags-change p0: [2002 2003 2001] -> [2002 2003 2005] replaced broker
  eqiad.mediawiki.user-blocks-change p0: [2002 2001 2003] -> [2004 2002 2001] replaced broker
  eqiad.rdf-streaming-updater.mutation p0: [2002 2003 2001] -> [2005 2003 2002] replaced broker
  eqiad.rdf-streaming-updater.state-inconsistency p0: [2003 2001 2002] -> [2003 2001 2004] replaced broker
  eqiad.resource-purge p0: [2001 2002 2003] -> [2001 2005 2003] replaced broker
  eqiad.resource-purge p1: [2002 2003 2001] -> [2002 2004 2001] replaced broker
  eqiad.resource-purge p2: [2003 2001 2002] -> [2004 2003 2002] replaced broker
  eqiad.resource_change p0: [2003 2002 2001] -> [2005 2002 2003] replaced broker
  eqiad.resource_change p1: [2001 2002 2003] -> [2003 2001 2005] replaced broker
  eqiad.resource_change p2: [2002 2003 2001] -> [2001 2005 2002] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/2/2.00 -> 3/4/3.60
  -
  Broker 2001 - leader: 49, follower: 112, total: 161
  Broker 2002 - leader: 48, follower: 114, total: 162
  Broker 2003 - leader: 49, follower: 112, total: 161
  Broker 2004 - leader: 48, follower: 73, total: 121
  Broker 2005 - leader: 48, follower: 73, total: 121

WARN:
  [none]

New partition maps:
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchCheckerJob.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-tags-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.EchoPushNotificationRequest.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.constraintsRunCheck.json
  /home/elukey/T225005/json/codfw.mediawiki.job.MessageGroupStatesUpdaterJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.enotifNotify.json
  /home/elukey/T225005/json/codfw.mediawiki.page-delete.json
  /home/elukey/T225005/json/codfw.resource_change.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.changeprop.retry.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.EchoNotificationDeleteJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.TranslateRenderJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.UpdateRepoOnDelete.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-delete.json
  /home/elukey/T225005/json/codfw.eventgate-main.test.event.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-score.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.recentChangesUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.webVideoTranscodePrioritized.json
  /home/elukey/T225005/json/codfw.mediawiki.job.AssembleUploadChunks.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ChangeNotification.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchElasticaWrite.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchOtherIndex.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.watchlistExpiry.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.wikibase-addUsagesForPage.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-links-change.json
  /home/elukey/T225005/json/codfw.mediawiki.recentchange.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CognateLocalJobSubmitJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/eqiad.mediawiki.recentchange.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CleanTermsIfUnused.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ThumbnailRender.json
  /home/elukey/T225005/json/eqiad.resource_change.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.wikibase-InjectRCRecords.json
  /home/elukey/T225005/json/codfw.mediawiki.user-blocks-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.activityUpdateJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-move.json
  /home/elukey/T225005/json/codfw.mediawiki.job.EchoNotificationDeleteJob.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.updateBetaFeaturesUserCounts.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cdnPurge.json
  /home/elukey/T225005/json/codfw.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.refreshLinksPrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchDeletePages.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchLinksUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cdnPurge.json
  /home/elukey/T225005/json/codfw.mediawiki.job.wikibase-addUsagesForPage.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-properties-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchIncomingLinkCount.json
  /home/elukey/T225005/json/codfw.mediawiki.job.activityUpdateJob.json
  /home/elukey/T225005/json/eqiad.changeprop.retry.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.UpdateRepoOnMove.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.RecordLintJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.enotifNotify.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-create.json
  /home/elukey/T225005/json/codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchElasticaWrite.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchLinksUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.refreshLinksPrioritized.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchDeletePages.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.newUserMessageJob.json
  /home/elukey/T225005/json/eqiad.rdf-streaming-updater.state-inconsistency.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.globalUsageCachePurge.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CognateCacheUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.userOptionsUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.page-move.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.error.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.UpdateRepoOnMove.json
  /home/elukey/T225005/json/__consumer_offsets.json
  /home/elukey/T225005/json/codfw.mediawiki.job.webVideoTranscodePrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchCheckerJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchIncomingLinkCount.json
  /home/elukey/T225005/json/eqiad.resource-purge.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ChangeDeletionNotification.json
  /home/elukey/T225005/json/codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
  /home/elukey/T225005/json/codfw.mediawiki.job.TTMServerMessageUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ThumbnailRender.json
  /home/elukey/T225005/json/codfw.mediawiki.job.newUserMessageJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-recommendation-create.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-tags-change.json
  /home/elukey/T225005/json/codfw.cpjobqueue.error.json
  /home/elukey/T225005/json/codfw.mediawiki.job.RecordLintJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.AssembleUploadChunks.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.userOptionsUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.EchoPushNotificationRequest.json
  /home/elukey/T225005/json/codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
  /home/elukey/T225005/json/codfw.mediawiki.page-properties-change.json
  /home/elukey/T225005/json/eqiad.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CognateCacheUpdateJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.wikibase-InjectRCRecords.json
  /home/elukey/T225005/json/codfw.change-prop.wikidata.resource-change.json
  /home/elukey/T225005/json/codfw.resource-purge.json
  /home/elukey/T225005/json/eqiad.change-prop.wikidata.resource-change.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchOtherIndex.json
  /home/elukey/T225005/json/eqiad.mediawiki.user-blocks-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.globalUsageCachePurge.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.LoginNotifyChecks.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-create.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-score.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.UpdateRepoOnDelete.json
  /home/elukey/T225005/json/codfw.mediawiki.job.newcomerTasksCacheRefreshJob.json
  /home/elukey/T225005/json/codfw.mediawiki.page-create.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-recommendation-create.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.revertedTagUpdate.json
  /home/elukey/T225005/json/eqiad.rdf-streaming-updater.mutation.json
  /home/elukey/T225005/json/codfw.mediawiki.job.TranslateRenderJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.watchlistExpiry.json
  /home/elukey/T225005/json/codfw.mediawiki.job.flaggedrevs_CacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.recentChangesUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-create.json
  /home/elukey/T225005/json/codfw.rdf-streaming-updater.mutation.json
  /home/elukey/T225005/json/eqiad.changeprop.error.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
  /home/elukey/T225005/json/codfw.changeprop.error.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/codfw.mediawiki.job.LoginNotifyChecks.json
  /home/elukey/T225005/json/eqiad.change-prop.transcludes.resource-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
  /home/elukey/T225005/json/codfw.change-prop.transcludes.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.page-links-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CleanTermsIfUnused.json
  /home/elukey/T225005/json/codfw.mediawiki.job.revertedTagUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ChangeDeletionNotification.json
  /home/elukey/T225005/json/codfw.rdf-streaming-updater.state-inconsistency.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/eqiad.eventgate-main.test.event.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CentralAuthCreateLocalAccountJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.constraintsRunCheck.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ChangeNotification.json
elukey added a comment.Aug 13 2021, 3:30 PM

All the files generated are like:

elukey@kafka-main2001:~$ cat T225005/json/eqiad.mediawiki.page-move.json 
{"version":1,"partitions":[{"topic":"eqiad.mediawiki.page-move","partition":0,"replicas":[2001,2003,2004]}]}

That is fine for topics with only one partition, but when we have more everything is in one file:

elukey@kafka-main2001:~/T225005/json$ cat codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json 
{"version":1,"partitions":[{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":0,"replicas":[2002,2004,2003]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":1,"replicas":[2004,2002,2001]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":2,"replicas":[2005,2003,2002]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":3,"replicas":[2003,2001,2004]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":4,"replicas":[2001,2005,2002]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":5,"replicas":[2002,2004,2001]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":6,"replicas":[2004,2001,2002]},{"topic":"codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks","partition":7,"replicas":[2005,2003,2002]}]}

The above is not great since it means kicking off a move for all the partitions of the topics, meanwhile we'd likely want to go one at the time. I had a chat with Razzi and we had the same issue on Jumbo, the solution was to split manually these files in multiple ones. We could ask to upstream if they are willing to add a feature to support our use case, but the number of topics with more than one partition is not that much for main so a manual split is probably doable/fine.

We'd probably want to version the files produced somewhere (maybe testing gitlab?), so that once we have the plan set in stone we'll have a way of recovering it in case of need.

herron added a comment.Aug 13 2021, 6:21 PM

Nice! Regarding upstream improvements, on a related note there will hopefully in the future be better control over partition movement within Kafka itself with https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Internal+Partition+Reassignment+Batching and similar work (although afaict is currently stalled). But splitting them out manually seems fine for now.

Sure, let's try creating a project in gitlab for this, although I don't think I have permission yet to create a non-personal project (e.g. not gitlab.wm.o/Herron/kafka_rebalance) do you? Fully agree about versioning and staging the files/plan that way.

fgiunchedi triaged this task as Medium priority.Aug 30 2021, 8:07 AM
elukey added a comment.EditedSep 21 2021, 1:04 PM

I was able to add the list of json commands to https://gitlab.wikimedia.org/Elukey/kafka_main_rebalance/-/tree/main/main-codfw/topicmappr_json

A couple of notes:

  • We have switched back to eqiad, and now the top trending topics by requests are eqiad. ones of course.
  • The regex used to get the list of topics to operate contained eqiad|codfw, so we should be good. It seems that the original list that we created before the switchover still holds (namely the same topics are the targets).
  • Most of the topics have one partition, meanwhile a few of them have more than one. We may want to split the command to move the multi-partitions topic in order to be less aggressive with Kafka.
elukey added a comment.Sep 21 2021, 1:09 PM

The final goal is to make https://grafana.wikimedia.org/d/000000027/kafka?viewPanel=48&orgId=1&var-datasource=codfw%20prometheus%2Fops&var-kafka_cluster=main-codfw&var-cluster=kafka_main&var-kafka_broker=All&var-disk_device=All better (the new brokers have very little partitions compared to the other ones).

Stashbot added a comment.Oct 11 2021, 7:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-11T07:57:56Z] <elukey> start kafka topics rebalancing for main-codfw (long running maintenance) - T288825

Stashbot added a comment.Oct 11 2021, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-11T09:09:48Z] <elukey> force kafka preferred-replica-election on kafka-main2001 after the first 50 topic partitions moves - T288825

Stashbot added a comment.Oct 11 2021, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-11T09:37:25Z] <elukey> force kafka preferred-replica-election on kafka-main2001 after another batch of topic partitions moves - T288825

elukey added a comment.EditedOct 11 2021, 9:39 AM

Topics move done so far and their start timings (they coincide with the file creation on kafka-main2001):

Oct 12 07:38 codfw.resource-purge.json
Oct 12 07:13 eqiad.resource-purge.json
Oct 12 07:12 codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
Oct 12 06:33 eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
Oct 12 06:31 codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
Oct 12 06:23 eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
Oct 12 06:18 codfw.change-prop.transcludes.resource-change.json
Oct 12 06:03 eqiad.change-prop.transcludes.resource-change.json
Oct 12 06:02 codfw.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 11 17:07 codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 11 17:00 eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 11 16:58 codfw.resource_change.json
Oct 11 16:40 eqiad.resource_change.json
Oct 11 16:39 codfw.mediawiki.revision-tags-change.json
Oct 11 16:37 eqiad.mediawiki.revision-tags-change.json
Oct 11 16:34 codfw.mediawiki.revision-score.json
Oct 11 16:17 eqiad.mediawiki.revision-score.json
Oct 11 16:16 codfw.mediawiki.job.refreshLinks.json
Oct 11 16:06 eqiad.mediawiki.job.refreshLinks.json
Oct 11 16:05 codfw.mediawiki.job.recentChangesUpdate.json
Oct 11 16:03 eqiad.mediawiki.job.recentChangesUpdate.json
Oct 11 16:02 codfw.mediawiki.job.htmlCacheUpdate.json
Oct 11 15:45 eqiad.mediawiki.job.htmlCacheUpdate.json
Oct 11 15:28 codfw.mediawiki.recentchange.json
Oct 11 15:15 eqiad.mediawiki.recentchange.json
Oct 11 15:11 codfw.mediawiki.revision-create.json
Oct 11 15:03 eqiad.mediawiki.revision-create.json
Oct 11 15:02 codfw.mediawiki.job.RecordLintJob.json
Oct 11 14:55 eqiad.mediawiki.job.RecordLintJob.json
Oct 11 14:51 codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
Oct 11 14:50 eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
Oct 11 14:49 codfw.mediawiki.job.cirrusSearchLinksUpdate.json
Oct 11 14:39 eqiad.mediawiki.job.cirrusSearchLinksUpdate.json
Oct 11 14:38 codfw.mediawiki.job.cirrusSearchIncomingLinkCount.json
Oct 11 14:34 eqiad.mediawiki.job.cirrusSearchIncomingLinkCount.json
Oct 11 14:34 codfw.mediawiki.job.cirrusSearchElasticaWrite.json
Oct 11 13:49 eqiad.mediawiki.job.cirrusSearchElasticaWrite.json
Oct 11 13:47 codfw.rdf-streaming-updater.state-inconsistency.json
Oct 11 13:46 eqiad.rdf-streaming-updater.state-inconsistency.json
Oct 11 13:45 codfw.mediawiki.revision-recommendation-create.json
Oct 11 13:45 eqiad.mediawiki.revision-recommendation-create.json
Oct 11 13:41 codfw.mediawiki.job.cdnPurge.json
Oct 11 13:37 eqiad.mediawiki.job.cdnPurge.json
Oct 11 13:37 codfw.mediawiki.job.wikibase-addUsagesForPage.json
Oct 11 13:28 eqiad.mediawiki.job.wikibase-addUsagesForPage.json
Oct 11 13:27 codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 11 13:25 eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 11 13:25 codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 11 13:24 eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 11 13:24 codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 11 13:23 eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 11 13:22 codfw.change-prop.wikidata.resource-change.json
Oct 11 13:21 eqiad.change-prop.wikidata.resource-change.json
Oct 11 13:19 codfw.mediawiki.job.refreshLinksPrioritized.json
Oct 11 13:19 eqiad.mediawiki.job.refreshLinksPrioritized.json
Oct 11 13:16 codfw.mediawiki.page-properties-change.json
Oct 11 13:14 eqiad.mediawiki.page-properties-change.json
Oct 11 13:13 codfw.mediawiki.page-move.json
Oct 11 13:13 eqiad.mediawiki.page-move.json
Oct 11 13:10 codfw.mediawiki.page-links-change.json
Oct 11 13:05 eqiad.mediawiki.page-links-change.json
Oct 11 13:03 codfw.mediawiki.page-delete.json
Oct 11 13:03 eqiad.mediawiki.page-delete.json
Oct 11 13:02 codfw.eventgate-main.test.event.json
Oct 11 13:01 eqiad.eventgate-main.test.event.json
Oct 11 09:44 codfw.rdf-streaming-updater.mutation.json
Oct 11 09:41 eqiad.rdf-streaming-updater.mutation.json
Oct 11 09:41 codfw.mediawiki.user-blocks-change.json
Oct 11 09:40 eqiad.mediawiki.user-blocks-change.json
Oct 11 09:35 codfw.mediawiki.page-create.json
Oct 11 09:35 eqiad.mediawiki.page-create.json
Oct 11 09:34 codfw.mediawiki.job.wikibase-InjectRCRecords.json
Oct 11 09:33 eqiad.mediawiki.job.wikibase-InjectRCRecords.json
Oct 11 09:33 codfw.mediawiki.job.webVideoTranscodePrioritized.json
Oct 11 09:32 eqiad.mediawiki.job.webVideoTranscodePrioritized.json
Oct 11 09:32 codfw.mediawiki.job.watchlistExpiry.json
Oct 11 09:31 eqiad.mediawiki.job.watchlistExpiry.json
Oct 11 09:31 codfw.mediawiki.job.userOptionsUpdate.json
Oct 11 09:30 eqiad.mediawiki.job.userOptionsUpdate.json
Oct 11 09:29 codfw.mediawiki.job.UpdateRepoOnMove.json
Oct 11 09:28 eqiad.mediawiki.job.UpdateRepoOnMove.json
Oct 11 09:28 codfw.mediawiki.job.UpdateRepoOnDelete.json
Oct 11 09:27 eqiad.mediawiki.job.UpdateRepoOnDelete.json
Oct 11 09:26 codfw.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 11 09:26 eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 11 09:25 codfw.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 11 09:25 eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 11 09:24 codfw.mediawiki.job.TranslateRenderJob.json
Oct 11 09:24 eqiad.mediawiki.job.TranslateRenderJob.json
Oct 11 09:23 codfw.mediawiki.job.ThumbnailRender.json
Oct 11 09:23 eqiad.mediawiki.job.ThumbnailRender.json
Oct 11 09:22 codfw.mediawiki.job.revertedTagUpdate.json
Oct 11 09:22 eqiad.mediawiki.job.revertedTagUpdate.json
Oct 11 09:20 codfw.mediawiki.job.ORESFetchScoreJob.json
Oct 11 09:19 eqiad.mediawiki.job.ORESFetchScoreJob.json
Oct 11 09:19 codfw.mediawiki.job.newUserMessageJob.json
Oct 11 09:19 eqiad.mediawiki.job.newUserMessageJob.json
Oct 11 09:17 codfw.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 11 09:17 eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 11 09:16 codfw.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 11 09:16 eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 11 09:14 codfw.mediawiki.job.LoginNotifyChecks.json
Oct 11 09:14 eqiad.mediawiki.job.LoginNotifyChecks.json
Oct 11 09:13 codfw.mediawiki.job.globalUsageCachePurge.json
Oct 11 09:13 eqiad.mediawiki.job.globalUsageCachePurge.json
Oct 11 09:12 codfw.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 11 09:12 eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 11 09:04 codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 11 09:04 eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 11 09:03 codfw.mediawiki.job.enotifNotify.json
Oct 11 09:02 eqiad.mediawiki.job.enotifNotify.json
Oct 11 09:01 codfw.mediawiki.job.EchoPushNotificationRequest.json
Oct 11 09:01 eqiad.mediawiki.job.EchoPushNotificationRequest.json
Oct 11 08:59 codfw.mediawiki.job.EchoNotificationDeleteJob.json
Oct 11 08:58 eqiad.mediawiki.job.EchoNotificationDeleteJob.json
Oct 11 08:56 codfw.mediawiki.job.constraintsRunCheck.json
Oct 11 08:56 eqiad.mediawiki.job.constraintsRunCheck.json
Oct 11 08:53 eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 11 08:52 codfw.mediawiki.job.CognateCacheUpdateJob.json
Oct 11 08:52 eqiad.mediawiki.job.CognateCacheUpdateJob.json
Oct 11 08:51 codfw.mediawiki.job.CleanTermsIfUnused.json
Oct 11 08:51 eqiad.mediawiki.job.CleanTermsIfUnused.json
Oct 11 08:50 codfw.mediawiki.job.cirrusSearchOtherIndex.json
Oct 11 08:50 eqiad.mediawiki.job.cirrusSearchOtherIndex.json
Oct 11 08:41 codfw.mediawiki.job.cirrusSearchDeletePages.json
Oct 11 08:41 eqiad.mediawiki.job.cirrusSearchDeletePages.json
Oct 11 08:37 codfw.mediawiki.job.cirrusSearchCheckerJob.json
Oct 11 08:37 eqiad.mediawiki.job.cirrusSearchCheckerJob.json
Oct 11 08:36 codfw.mediawiki.job.ChangeNotification.json
Oct 11 08:35 eqiad.mediawiki.job.ChangeNotification.json
Oct 11 08:31 codfw.mediawiki.job.ChangeDeletionNotification.json
Oct 11 08:30 eqiad.mediawiki.job.ChangeDeletionNotification.json
Oct 11 08:25 codfw.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 11 08:25 eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 11 08:22 codfw.mediawiki.job.categoryMembershipChange.json
Oct 11 08:21 eqiad.mediawiki.job.categoryMembershipChange.json
Oct 11 08:16 codfw.mediawiki.job.AssembleUploadChunks.json
Oct 11 08:16 eqiad.mediawiki.job.AssembleUploadChunks.json
Oct 11 08:12 codfw.mediawiki.job.activityUpdateJob.json
Oct 11 08:12 eqiad.mediawiki.job.activityUpdateJob.json
Oct 11 08:05 eqiad.cpjobqueue.error.json
Oct 11 08:05 codfw.cpjobqueue.error.json
Oct 11 08:00 eqiad.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 11 08:00 codfw.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 11 07:53 eqiad.changeprop.error.json
Oct 11 07:52 codfw.changeprop.error.json
Oct 11 07:46 eqiad.change-prop.backlinks.resource-change.json
Oct 11 07:43 codfw.change-prop.backlinks.resource-change.json
elukey added a comment.Oct 11 2021, 9:55 AM

Note - I added two directories to https://gitlab.wikimedia.org/Elukey/kafka_main_rebalance/-/tree/main/main-codfw/topicmappr_json:

  1. rollback - containing the output of kafka-reassign-partitions (basically the previous state if one wants to restore it)
  2. completed - containing the topics already moved

What is left under topic_mappr_json are the topics left to move.

Stashbot added a comment.Oct 11 2021, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-11T13:42:12Z] <elukey> force kafka preferred-replica-election on kafka-main2001 after another batch of topic partitions moves - T288825

Stashbot added a comment.Oct 11 2021, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-11T17:08:37Z] <elukey> force kafka preferred-replica-election on kafka-main2001 after another batch of topic partitions moves - T288825

elukey added a comment.Oct 11 2021, 5:13 PM

Topics left to move for main-codfw:

codfw.change-prop.transcludes.resource-change.json
codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
codfw.mediawiki.job.CognateLocalJobSubmitJob.json
codfw.resource-purge.json
__consumer_offsets.json
eqiad.change-prop.transcludes.resource-change.json
eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
eqiad.resource-purge.json
Stashbot added a comment.Oct 12 2021, 7:40 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T07:40:23Z] <elukey> run kafka preferred-replica-election on kafka-main2001 to rebalance partition leaders after the last topic moves - T288825

elukey added a comment.Oct 12 2021, 10:30 AM

All topics done except __consumer_offsets, seemed a little delicate to move and not super traffic-intensive.

Proposal: I'd bump the partitions of the topmost traffic intensive topics from 3 to 5 to spread the traffic out more.

kafka topics --alter --topic eqiad.resource-purge --partitions 5
kafka topics --alter --topic codfw.resource-purge --partitions 5

kafka topics --alter --topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

kafka topics --alter --topic eqiad.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

The above assuming that the producers do not use any custom partitioning-by-key scheme (but the the default round-robin).

Joe added a subscriber: Joe.Oct 12 2021, 10:35 AM
In T288825#7419227, @elukey wrote:

All topics done except __consumer_offsets, seemed a little delicate to move and not super traffic-intensive.

Proposal: I'd bump the partitions of the topmost traffic intensive topics from 3 to 5 to spread the traffic out more.

kafka topics --alter --topic eqiad.resource-purge --partitions 5
kafka topics --alter --topic codfw.resource-purge --partitions 5

kafka topics --alter --topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

kafka topics --alter --topic eqiad.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

Seems sensible given the vast majority of traffic is for those topics

The above assuming that the producers do not use any custom partitioning-by-key scheme (but the the default round-robin).

The producer is eventgate, which AFAIR does round-robin indeed.

Stashbot added a comment.Oct 12 2021, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T11:49:41Z] <elukey> kafka topics --alter --topic eqiad.resource-purge --partitions 5 on kafka-main2001 - T288825

Stashbot added a comment.Oct 12 2021, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T11:58:24Z] <elukey> kafka topics --alter --topic codfw.resource-purge --partitions 5 on kafka-main2001 - T288825

Stashbot added a comment.Oct 12 2021, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T12:09:58Z] <elukey> kafka topics --alter --topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5 - T288825

Stashbot added a comment.Oct 12 2021, 12:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T12:10:18Z] <elukey> kafka topics --alter --topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5 - T288825

Stashbot added a comment.Oct 12 2021, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T12:15:22Z] <elukey> kafka topics --alter --topic eqiad.mediawiki.job.cirrusSearchElasticaWrite --partitions 5 - T288825

Stashbot added a comment.Oct 12 2021, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-12T12:15:31Z] <elukey> kafka topics --alter --topic codfw.mediawiki.job.cirrusSearchElasticaWrite --partitions 5 - T288825

elukey added a comment.Oct 12 2021, 1:00 PM

The main-codfw cluster looks way better now, even if in some places is still unbalanced, like the kafka bytes out graph. One of the reason is that the topic partition leaders for the highest trafficated topics is not spread among all broekers (even if the partition count is 5), but only on 4 of them. Will try to see if anything can be done.

elukey added a comment.Oct 12 2021, 1:04 PM

Example for the highest traffic volume:

Topic:eqiad.resource-purge      PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.resource-purge     Partition: 0    Leader: 2001    Replicas: 2001,2005,2003        Isr: 2001,2003,2005
        Topic: eqiad.resource-purge     Partition: 1    Leader: 2002    Replicas: 2002,2004,2001        Isr: 2001,2002,2004
        Topic: eqiad.resource-purge     Partition: 2    Leader: 2004    Replicas: 2004,2003,2002        Isr: 2003,2002,2004
        Topic: eqiad.resource-purge     Partition: 3    Leader: 2004    Replicas: 2004,2001,2002        Isr: 2004,2001,2002
        Topic: eqiad.resource-purge     Partition: 4    Leader: 2005    Replicas: 2005,2001,2002        Isr: 2005,2001,2002

The Leaders are 4/5 of the brokers, it seems that 2003 is excluded, but in the ISR set 2003 is mentioned in two cases. It seems that 2004 takes the load of two partitions, but I am not sure why.

elukey added a comment.Oct 12 2021, 1:15 PM

In theory we can use the following json file to hint Kafka:

{"partitions": [
    {"topic": "eqiad.resource-purge", "partition": 0, "replicas": [2001,2005,2003]},
    {"topic": "eqiad.resource-purge", "partition": 1, "replicas": [2002,2004,2001]},
    {"topic": "eqiad.resource-purge", "partition": 2, "replicas": [2003,2004,2002]},
    {"topic": "eqiad.resource-purge", "partition": 3, "replicas": [2004,2001,2002]},
    {"topic": "eqiad.resource-purge", "partition": 4, "replicas": [2005,2001,2002]},
  ],
  "version":1
}

The part that changes from the current setup is:

{"topic": "eqiad.resource-purge", "partition": 2, "replicas": [2003,2004,2002]},

The first replica, in theory, should be the one that gets (more likely) a leadership position. With another round of kafka reassign-partitions we may get 5/5 brokers used as partition leaders for the topic.

elukey added a comment.Oct 12 2021, 1:27 PM

It worked! Going to do the same for other topics:

Topic:codfw.resource-purge      PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.resource-purge     Partition: 0    Leader: 2003    Replicas: 2003,2005,2002        Isr: 2003,2002,2005
        Topic: codfw.resource-purge     Partition: 1    Leader: 2001    Replicas: 2001,2004,2002        Isr: 2001,2002,2004
        Topic: codfw.resource-purge     Partition: 2    Leader: 2002    Replicas: 2002,2001,2004        Isr: 2001,2002,2004
        Topic: codfw.resource-purge     Partition: 3    Leader: 2001    Replicas: 2001,2002,2003        Isr: 2001,2002,2003
        Topic: codfw.resource-purge     Partition: 4    Leader: 2002    Replicas: 2002,2003,2004        Isr: 2002,2003,2004


{"partitions": [
    {"topic": "codfw.resource-purge", "partition": 0, "replicas": [2005,2003,2002]},
    {"topic": "codfw.resource-purge", "partition": 1, "replicas": [2001,2004,2002]},
    {"topic": "codfw.resource-purge", "partition": 2, "replicas": [2002,2001,2004]},
    {"topic": "codfw.resource-purge", "partition": 3, "replicas": [2003,2002,2001]},
    {"topic": "codfw.resource-purge", "partition": 4, "replicas": [2004,2003,2002]},
  ],
  "version":1
}
elukey added a comment.Oct 12 2021, 1:38 PM

@Joe Let me know if the current status of main-codfw is good in your opinion, it is not perfectly balanced in the incoming traffic but I think the procedure should be good to be applied in eqiad. If so I'll organize the work for the next days (or likely next week).

elukey added a comment.EditedOct 13 2021, 6:17 AM

Checked a little bit more the top 10 high traffic topics, and eqiad.change-prop.transcludes.resource-change (4th) still runs with a single partition with 500 msg/s. I would increase the partition size to at least 3 for this topic (and its codfw variant).

elukey added a comment.Oct 13 2021, 6:24 AM
kafka topics --alter --topic eqiad.change-prop.transcludes.resource-change --partitions 3
kafka topics --alter --topic codfw.change-prop.transcludes.resource-change --partitions 3
Stashbot added a comment.Oct 13 2021, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-13T06:26:30Z] <elukey> kafka topics --alter --topic {eqiad,codfw}.change-prop.transcludes.resource-change --partitions 3 on kafka-main2001 - T288825

elukey added a comment.Oct 13 2021, 7:25 AM

In main-eqiad the starting point can be the high traffic topics:

kafka topics --alter --topic eqiad.resource-purge --partitions 5
kafka topics --alter --topic codfw.resource-purge --partitions 5

kafka topics --alter --topic eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

kafka topics --alter --topic eqiad.mediawiki.job.cirrusSearchElasticaWrite --partitions 5
kafka topics --alter --topic codfw.mediawiki.job.cirrusSearchElasticaWrite --partitions 5

kafka topics --alter --topic eqiad.change-prop.transcludes.resource-change --partitions 3
kafka topics --alter --topic codfw.change-prop.transcludes.resource-change --partitions 3
Stashbot added a comment.Oct 13 2021, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-13T07:33:41Z] <elukey> increase kafka topic partition size of the top 4 high traffic topics of main-eqiad as described in https://phabricator.wikimedia.org/T288825#7422726

elukey added a comment.Oct 13 2021, 7:47 AM

The codfw.resource-purge topic got expanded to:

Topic:codfw.resource-purge      PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.resource-purge     Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1003,1002
        Topic: codfw.resource-purge     Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1003,1002
        Topic: codfw.resource-purge     Partition: 2    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: codfw.resource-purge     Partition: 3    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: codfw.resource-purge     Partition: 4    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004

In order to balance it properly, I am going to apply:

{"partitions": [
    {"topic": "codfw.resource-purge", "partition": 0, "replicas": [1005,1001,1002]},
    {"topic": "codfw.resource-purge", "partition": 1, "replicas": [1003,1002,1001]},
    {"topic": "codfw.resource-purge", "partition": 2, "replicas": [1002,1003,1001]},
    {"topic": "codfw.resource-purge", "partition": 3, "replicas": [1001,1002,1003]},
    {"topic": "codfw.resource-purge", "partition": 4, "replicas": [1004,1003,1002]},
  ],
  "version":1
}
elukey added a comment.EditedOct 13 2021, 8:10 AM

Adjustments for main-eqiad:

Topic:eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite      PartitionCount:5        ReplicationFactor:3     
Configs:
        Topic: eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1003,1002
        Topic: eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1003,1002
        Topic: eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 2    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 3    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 4    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004
{"partitions": [
    {"topic": "eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [1003,1001,1002]},
    {"topic": "eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [1001,1002,1003]},
    {"topic": "eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [1002,1003,1001]},
    {"topic": "eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [1005,1002,1003]},
    {"topic": "eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [1004,1003,1002]}
  ],
  "version":1
}
Topic:codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite      PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 0    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1001,1003,1002
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1003,1002
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 3    Leader: 1005    Replicas: 1005,1001,1002        Isr: 1005,1001,1002
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 4    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
{"partitions": [
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [1002,1003,1001]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [1003,1001,1002]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [1001,1002,1003]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [1005,1001,1002]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [1004,1002,1003]}
  ],
  "version":1
}
elukey added a comment.Oct 13 2021, 8:17 AM
Topic:eqiad.mediawiki.job.cirrusSearchElasticaWrite     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.mediawiki.job.cirrusSearchElasticaWrite    Partition: 0    Leader: 1002    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: eqiad.mediawiki.job.cirrusSearchElasticaWrite    Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1001,1003,1002
        Topic: eqiad.mediawiki.job.cirrusSearchElasticaWrite    Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1001,1003,1002
        Topic: eqiad.mediawiki.job.cirrusSearchElasticaWrite    Partition: 3    Leader: 1005    Replicas: 1005,1001,1002        Isr: 1005,1001,1002
        Topic: eqiad.mediawiki.job.cirrusSearchElasticaWrite    Partition: 4    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
{"partitions": [
    {"topic": "eqiad.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [1002,1001,1003]},
    {"topic": "eqiad.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [1003,1002,1001]},
    {"topic": "eqiad.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [1001,1003,1002]},
    {"topic": "eqiad.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [1005,1001,1002]},
    {"topic": "eqiad.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [1004,1002,1003]}
  ],
  "version":1
}
Topic:codfw.mediawiki.job.cirrusSearchElasticaWrite     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 0    Leader: 1002    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1001,1003,1002
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1001,1003,1002
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 3    Leader: 1005    Replicas: 1005,1001,1002        Isr: 1005,1001,1002
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 4    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
{"partitions": [
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [1002,1003,1001},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [1003,1002,1001]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [1001,1003,1002]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [1005,1001,1002]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [1004,1002,1003]}
  ],
  "version":1
}
Stashbot added a comment.Oct 13 2021, 8:21 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-13T08:21:38Z] <elukey> run kafka preferred-replica-election on kafka-main1001 to rebalance partition leaders - T288825

elukey added a comment.Oct 13 2021, 8:27 AM

Two little fine-tune changes for main-codfw topics (inspired from the work done on main-eqiad):

Topic:codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite      PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 0    Leader: 2001    Replicas: 2001,2004,2003        Isr: 2001,2003,2004
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 1    Leader: 2002    Replicas: 2002,2001,2005        Isr: 2001,2002,2005
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 2    Leader: 2004    Replicas: 2004,2002,2003        Isr: 2003,2002,2004
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 3    Leader: 2004    Replicas: 2004,2001,2002        Isr: 2004,2001,2002
        Topic: codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite     Partition: 4    Leader: 2005    Replicas: 2005,2001,2002        Isr: 2005,2001,2002
{"partitions": [
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [2001,2004,2003]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [2002,2001,2005]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [2003,2002,2004]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [2004,2001,2002]},
    {"topic": "codfw.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [2005,2001,2002]}
  ],
  "version":1
}
Topic:codfw.mediawiki.job.cirrusSearchElasticaWrite     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 0    Leader: 2002    Replicas: 2002,2004,2003        Isr: 2003,2002,2004
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 1    Leader: 2004    Replicas: 2004,2002,2003        Isr: 2003,2002,2004
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 2    Leader: 2005    Replicas: 2005,2003,2001        Isr: 2001,2003,2005
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 3    Leader: 2005    Replicas: 2005,2001,2002        Isr: 2005,2001,2002
        Topic: codfw.mediawiki.job.cirrusSearchElasticaWrite    Partition: 4    Leader: 2001    Replicas: 2001,2002,2003        Isr: 2001,2002,2003
{"partitions": [
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 0, "replicas": [2002,2004,2003},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 1, "replicas": [2004,2002,2003]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 2, "replicas": [2003,2005,2001]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 3, "replicas": [2005,2001,2002]},
    {"topic": "codfw.mediawiki.job.cirrusSearchElasticaWrite", "partition": 4, "replicas": [2001,2002,2003]}
  ],
  "version":1
}

These topics are not in use but in case of a switchover it is nice to have the top-most topics as balaanced as possible.

elukey added a comment.Oct 13 2021, 8:33 AM

Next steps:

  • come up with a topic-mappr plan for main-eqiad as done in T288825#7281471, excluding the aforementioned top 3 topics already running with 5 partitions.
elukey added a comment.EditedOct 15 2021, 8:51 AM

Topic mappr for eqiad:

./topicmappr rebuild --out-path /home/elukey/T225005/json --force-rebuild --zk-addr conf1004.eqiad.wmnet --zk-prefix kafka/main-eqiad --brokers -2 --topics '^(eqiad|codfw)\.change-prop\.transcludes\.resource-change$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchLinksUpdate$|^(eqiad|codfw)\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.resource_change|__consumer_offsets$|^(eqiad|codfw)\.mediawiki\.job\.RecordLintJob$|^(eqiad|codfw)\.mediawiki\.job\.wikibase-addUsagesForPage$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchCheckerJob$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchIncomingLinkCount$|^(eqiad|codfw)\.mediawiki\.job\.htmlCacheUpdate$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.htmlCacheUpdate$|^(eqiad|codfw)\.mediawiki\.recentchange$|^(eqiad|codfw)\.mediawiki\.job\.cdnPurge$|^(eqiad|codfw)\.mediawiki\.job\.recentChangesUpdate$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchLinksUpdatePrioritized$|^(eqiad|codfw)\.mediawiki\.revision-create$|^(eqiad|codfw)\.mediawiki\.page-links-change$|^(eqiad|codfw)\.mediawiki\.job\.ChangeDeletionNotification$|^(eqiad|codfw)\.mediawiki\.job\.categoryMembershipChange$|^(eqiad|codfw)\.mediawiki\.revision-score$|^(eqiad|codfw)\.mediawiki\.revision-tags-change$|^(eqiad|codfw)\.mediawiki\.job\.EchoNotificationDeleteJob$|^(eqiad|codfw)\.mediawiki\.job\.ORESFetchScoreJob$|^(eqiad|codfw)\.mediawiki\.job\.constraintsRunCheck$|^(eqiad|codfw)\.rdf-streaming-updater\.mutation$|^(eqiad|codfw)\.mediawiki\.page-properties-change$|^(eqiad|codfw)\.mediawiki\.job\.ThumbnailRender$|^(eqiad|codfw)\.mediawiki\.user-blocks-change$|^(eqiad|codfw)\.mediawiki\.job\.enotifNotify$|^(eqiad|codfw)\.mediawiki\.page-create$|^(eqiad|codfw)\.mediawiki\.job\.activityUpdateJob$|^(eqiad|codfw)\.mediawiki\.job\.ChangeNotification$|^(eqiad|codfw)\.mediawiki\.job\.CleanTermsIfUnused$|^(eqiad|codfw)\.mediawiki\.job\.wikibase-InjectRCRecords$|^(eqiad|codfw)\.mediawiki\.job\.globalUsageCachePurge$|^(eqiad|codfw)\.mediawiki\.job\.CognateCacheUpdateJob$|^(eqiad|codfw)\.change-prop\.backlinks\.resource-change$|^(eqiad|codfw)\.cpjobqueue\.retry\.mediawiki\.job\.categoryMembershipChange$|^(eqiad|codfw)\.mediawiki\.job\.watchlistExpiry$|^(eqiad|codfw)\.mediawiki\.job\.fetchGoogleCloudVisionAnnotations$|^(eqiad|codfw)\.mediawiki\.job\.flaggedrevs_CacheUpdate$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchOtherIndex$|^(eqiad|codfw)\.mediawiki\.job\.CognateLocalJobSubmitJob$|^(eqiad|codfw)\.mediawiki\.job\.webVideoTranscodePrioritized$|^(eqiad|codfw)\.eventgate-main\.test\.event$|^(eqiad|codfw)\.eventgate-main\.test\.event$|^(eqiad|codfw)\.change-prop\.wikidata\.resource-change$|^(eqiad|codfw)\.mediawiki\.job\.LoginNotifyChecks$|^(eqiad|codfw)\.mediawiki\.job\.CentralAuthCreateLocalAccountJob$|^(eqiad|codfw)\.mediawiki\.job\.refreshLinksPrioritized$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchDeletePages$|^(eqiad|codfw)\.mediawiki\.job\.revertedTagUpdate$|^(eqiad|codfw)\.mediawiki\.page-move$|^(eqiad|codfw)\.mediawiki\.job\.updateBetaFeaturesUserCounts$|^(eqiad|codfw)\.changeprop\.error$|^(eqiad|codfw)\.cpjobqueue\.retry\.mediawiki\.job\.ORESFetchScoreJob$|^(eqiad|codfw)\.mediawiki\.job\.newcomerTasksCacheRefreshJob$|^(eqiad|codfw)\.mediawiki\.job\.UpdateRepoOnMove$|^(eqiad|codfw)\.mediawiki\.revision-recommendation-create$|^(eqiad|codfw)\.mediawiki\.page-delete$|^(eqiad|codfw)\.mediawiki\.job\.TTMServerMessageUpdateJob$|^(eqiad|codfw)\.mediawiki\.job\.newUserMessageJob$|^(eqiad|codfw)\.mediawiki\.job\.TranslateRenderJob$|^(eqiad|codfw)\.mediawiki\.job\.AssembleUploadChunks$|^(eqiad|codfw)\.mediawiki\.job\.MessageGroupStatesUpdaterJob$|^(eqiad|codfw)\.cpjobqueue\.retry\.cpjobqueue\.partitioned\.mediawiki\.job\.refreshLinks$|^(eqiad|codfw)\.mediawiki\.job\.userOptionsUpdate$|^(eqiad|codfw)\.mediawiki\.job\.EchoPushNotificationRequest$|^(eqiad|codfw)\.mediawiki\.job\.UpdateRepoOnDelete$|^(eqiad|codfw)\.mediawiki\.revision-score$|^(eqiad|codfw)\.cpjobqueue\.error$|^(eqiad|codfw)\.changeprop\.retry\.change-prop\.backlinks\.resource-change$|^(eqiad|codfw)\.rdf-streaming-updater\.state-inconsistency$'

It is the same list as for main-codfw but without ^(eqiad|codfw)\.resource-purge$|^(eqiad|codfw)\.cpjobqueue\.partitioned\.mediawiki\.job\.cirrusSearchElasticaWrite$|^(eqiad|codfw)\.mediawiki\.job\.cirrusSearchElasticaWrite$ (topics already expanded to 5 partitions)

Output:

Topics:
  __consumer_offsets
  codfw.change-prop.backlinks.resource-change
  codfw.change-prop.transcludes.resource-change
  codfw.change-prop.wikidata.resource-change
  codfw.changeprop.error
  codfw.changeprop.retry.change-prop.backlinks.resource-change
  codfw.cpjobqueue.error
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob
  codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange
  codfw.eventgate-main.test.event
  codfw.mediawiki.job.AssembleUploadChunks
  codfw.mediawiki.job.CentralAuthCreateLocalAccountJob
  codfw.mediawiki.job.ChangeDeletionNotification
  codfw.mediawiki.job.ChangeNotification
  codfw.mediawiki.job.CleanTermsIfUnused
  codfw.mediawiki.job.CognateCacheUpdateJob
  codfw.mediawiki.job.CognateLocalJobSubmitJob
  codfw.mediawiki.job.EchoNotificationDeleteJob
  codfw.mediawiki.job.EchoPushNotificationRequest
  codfw.mediawiki.job.LoginNotifyChecks
  codfw.mediawiki.job.MessageGroupStatesUpdaterJob
  codfw.mediawiki.job.ORESFetchScoreJob
  codfw.mediawiki.job.RecordLintJob
  codfw.mediawiki.job.TTMServerMessageUpdateJob
  codfw.mediawiki.job.ThumbnailRender
  codfw.mediawiki.job.TranslateRenderJob
  codfw.mediawiki.job.UpdateRepoOnDelete
  codfw.mediawiki.job.UpdateRepoOnMove
  codfw.mediawiki.job.activityUpdateJob
  codfw.mediawiki.job.categoryMembershipChange
  codfw.mediawiki.job.cdnPurge
  codfw.mediawiki.job.cirrusSearchCheckerJob
  codfw.mediawiki.job.cirrusSearchDeletePages
  codfw.mediawiki.job.cirrusSearchIncomingLinkCount
  codfw.mediawiki.job.cirrusSearchLinksUpdate
  codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized
  codfw.mediawiki.job.cirrusSearchOtherIndex
  codfw.mediawiki.job.constraintsRunCheck
  codfw.mediawiki.job.enotifNotify
  codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations
  codfw.mediawiki.job.flaggedrevs_CacheUpdate
  codfw.mediawiki.job.globalUsageCachePurge
  codfw.mediawiki.job.htmlCacheUpdate
  codfw.mediawiki.job.newUserMessageJob
  codfw.mediawiki.job.newcomerTasksCacheRefreshJob
  codfw.mediawiki.job.recentChangesUpdate
  codfw.mediawiki.job.refreshLinks
  codfw.mediawiki.job.refreshLinksPrioritized
  codfw.mediawiki.job.revertedTagUpdate
  codfw.mediawiki.job.updateBetaFeaturesUserCounts
  codfw.mediawiki.job.userOptionsUpdate
  codfw.mediawiki.job.watchlistExpiry
  codfw.mediawiki.job.webVideoTranscodePrioritized
  codfw.mediawiki.job.wikibase-InjectRCRecords
  codfw.mediawiki.job.wikibase-addUsagesForPage
  codfw.mediawiki.page-create
  codfw.mediawiki.page-delete
  codfw.mediawiki.page-links-change
  codfw.mediawiki.page-move
  codfw.mediawiki.page-properties-change
  codfw.mediawiki.recentchange
  codfw.mediawiki.revision-create
  codfw.mediawiki.revision-recommendation-create
  codfw.mediawiki.revision-score
  codfw.mediawiki.revision-tags-change
  codfw.mediawiki.user-blocks-change
  codfw.rdf-streaming-updater.mutation
  codfw.rdf-streaming-updater.state-inconsistency
  codfw.resource_change
  eqiad.change-prop.backlinks.resource-change
  eqiad.change-prop.transcludes.resource-change
  eqiad.change-prop.wikidata.resource-change
  eqiad.changeprop.error
  eqiad.changeprop.retry.change-prop.backlinks.resource-change
  eqiad.cpjobqueue.error
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks
  eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob
  eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange
  eqiad.eventgate-main.test.event
  eqiad.mediawiki.job.AssembleUploadChunks
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob
  eqiad.mediawiki.job.ChangeDeletionNotification
  eqiad.mediawiki.job.ChangeNotification
  eqiad.mediawiki.job.CleanTermsIfUnused
  eqiad.mediawiki.job.CognateCacheUpdateJob
  eqiad.mediawiki.job.CognateLocalJobSubmitJob
  eqiad.mediawiki.job.EchoNotificationDeleteJob
  eqiad.mediawiki.job.EchoPushNotificationRequest
  eqiad.mediawiki.job.LoginNotifyChecks
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob
  eqiad.mediawiki.job.ORESFetchScoreJob
  eqiad.mediawiki.job.RecordLintJob
  eqiad.mediawiki.job.TTMServerMessageUpdateJob
  eqiad.mediawiki.job.ThumbnailRender
  eqiad.mediawiki.job.TranslateRenderJob
  eqiad.mediawiki.job.UpdateRepoOnDelete
  eqiad.mediawiki.job.UpdateRepoOnMove
  eqiad.mediawiki.job.activityUpdateJob
  eqiad.mediawiki.job.categoryMembershipChange
  eqiad.mediawiki.job.cdnPurge
  eqiad.mediawiki.job.cirrusSearchCheckerJob
  eqiad.mediawiki.job.cirrusSearchDeletePages
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount
  eqiad.mediawiki.job.cirrusSearchLinksUpdate
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized
  eqiad.mediawiki.job.cirrusSearchOtherIndex
  eqiad.mediawiki.job.constraintsRunCheck
  eqiad.mediawiki.job.enotifNotify
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate
  eqiad.mediawiki.job.globalUsageCachePurge
  eqiad.mediawiki.job.htmlCacheUpdate
  eqiad.mediawiki.job.newUserMessageJob
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob
  eqiad.mediawiki.job.recentChangesUpdate
  eqiad.mediawiki.job.refreshLinks
  eqiad.mediawiki.job.refreshLinksPrioritized
  eqiad.mediawiki.job.revertedTagUpdate
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts
  eqiad.mediawiki.job.userOptionsUpdate
  eqiad.mediawiki.job.watchlistExpiry
  eqiad.mediawiki.job.webVideoTranscodePrioritized
  eqiad.mediawiki.job.wikibase-InjectRCRecords
  eqiad.mediawiki.job.wikibase-addUsagesForPage
  eqiad.mediawiki.page-create
  eqiad.mediawiki.page-delete
  eqiad.mediawiki.page-links-change
  eqiad.mediawiki.page-move
  eqiad.mediawiki.page-properties-change
  eqiad.mediawiki.recentchange
  eqiad.mediawiki.revision-create
  eqiad.mediawiki.revision-recommendation-create
  eqiad.mediawiki.revision-score
  eqiad.mediawiki.revision-tags-change
  eqiad.mediawiki.user-blocks-change
  eqiad.rdf-streaming-updater.mutation
  eqiad.rdf-streaming-updater.state-inconsistency
  eqiad.resource_change

Broker change summary:
  Replacing 0, added 0, missing 0, total count changed by 0

Action:
  Force rebuilding map

Partition map changes:
  __consumer_offsets p0: [1002 1003 1001] -> [1003 1004 1001] replaced broker
  __consumer_offsets p1: [1003 1001 1002] -> [1001 1005 1002] replaced broker
  __consumer_offsets p2: [1001 1002 1003] -> [1002 1001 1003] preferred leader
  __consumer_offsets p3: [1002 1001 1003] -> [1004 1003 1002] replaced broker
  __consumer_offsets p4: [1003 1002 1001] -> [1005 1002 1003] replaced broker
  __consumer_offsets p5: [1001 1003 1002] -> [1003 1005 1001] replaced broker
  __consumer_offsets p6: [1002 1003 1001] -> [1001 1004 1003] replaced broker
  __consumer_offsets p7: [1003 1001 1002] -> [1002 1001 1004] replaced broker
  __consumer_offsets p8: [1001 1002 1003] -> [1004 1003 1002] replaced broker
  __consumer_offsets p9: [1002 1001 1003] -> [1005 1002 1001] replaced broker
  __consumer_offsets p10: [1003 1002 1001] -> [1003 1005 1001] replaced broker
  __consumer_offsets p11: [1001 1003 1002] -> [1001 1004 1002] replaced broker
  __consumer_offsets p12: [1002 1003 1001] -> [1002 1004 1003] replaced broker
  __consumer_offsets p13: [1003 1001 1002] -> [1004 1002 1003] replaced broker
  __consumer_offsets p14: [1001 1002 1003] -> [1005 1001 1002] replaced broker
  __consumer_offsets p15: [1002 1001 1003] -> [1003 1005 1001] replaced broker
  __consumer_offsets p16: [1003 1002 1001] -> [1001 1003 1005] replaced broker
  __consumer_offsets p17: [1001 1003 1002] -> [1002 1004 1003] replaced broker
  __consumer_offsets p18: [1002 1003 1001] -> [1004 1001 1002] replaced broker
  __consumer_offsets p19: [1003 1001 1002] -> [1005 1003 1001] replaced broker
  __consumer_offsets p20: [1001 1002 1003] -> [1003 1002 1004] replaced broker
  __consumer_offsets p21: [1002 1001 1003] -> [1001 1005 1003] replaced broker
  __consumer_offsets p22: [1003 1002 1001] -> [1002 1004 1001] replaced broker
  __consumer_offsets p23: [1001 1003 1002] -> [1004 1001 1002] replaced broker
  __consumer_offsets p24: [1002 1003 1001] -> [1005 1003 1002] replaced broker
  __consumer_offsets p25: [1003 1001 1002] -> [1003 1005 1001] replaced broker
  __consumer_offsets p26: [1001 1002 1003] -> [1001 1002 1004] replaced broker
  __consumer_offsets p27: [1002 1001 1003] -> [1002 1004 1003] replaced broker
  __consumer_offsets p28: [1003 1002 1001] -> [1004 1002 1001] replaced broker
  __consumer_offsets p29: [1001 1003 1002] -> [1005 1003 1002] replaced broker
  __consumer_offsets p30: [1002 1003 1001] -> [1003 1005 1002] replaced broker
  __consumer_offsets p31: [1003 1001 1002] -> [1001 1003 1005] replaced broker
  __consumer_offsets p32: [1001 1002 1003] -> [1002 1001 1005] replaced broker
  __consumer_offsets p33: [1002 1001 1003] -> [1004 1001 1003] replaced broker
  __consumer_offsets p34: [1003 1002 1001] -> [1005 1002 1003] replaced broker
  __consumer_offsets p35: [1001 1003 1002] -> [1003 1005 1001] replaced broker
  __consumer_offsets p36: [1002 1003 1001] -> [1001 1004 1002] replaced broker
  __consumer_offsets p37: [1003 1001 1002] -> [1002 1004 1001] replaced broker
  __consumer_offsets p38: [1001 1002 1003] -> [1004 1001 1003] replaced broker
  __consumer_offsets p39: [1002 1001 1003] -> [1005 1003 1002] replaced broker
  __consumer_offsets p40: [1003 1002 1001] -> [1003 1002 1004] replaced broker
  __consumer_offsets p41: [1001 1003 1002] -> [1001 1005 1003] replaced broker
  __consumer_offsets p42: [1002 1003 1001] -> [1002 1004 1001] replaced broker
  __consumer_offsets p43: [1003 1001 1002] -> [1004 1001 1002] replaced broker
  __consumer_offsets p44: [1001 1002 1003] -> [1005 1003 1001] replaced broker
  __consumer_offsets p45: [1002 1001 1003] -> [1003 1002 1004] replaced broker
  __consumer_offsets p46: [1003 1002 1001] -> [1001 1005 1003] replaced broker
  __consumer_offsets p47: [1001 1003 1002] -> [1002 1005 1001] replaced broker
  __consumer_offsets p48: [1002 1003 1001] -> [1004 1001 1003] replaced broker
  __consumer_offsets p49: [1003 1001 1002] -> [1005 1003 1002] replaced broker
  codfw.change-prop.backlinks.resource-change p0: [1003 1001 1002] -> [1003 1004 1001] replaced broker
  codfw.change-prop.transcludes.resource-change p0: [1003 1002 1001] -> [1001 1002 1005] replaced broker
  codfw.change-prop.transcludes.resource-change p1: [1004 1001 1002] -> [1002 1001 1004] preferred leader
  codfw.change-prop.transcludes.resource-change p2: [1005 1001 1002] -> [1004 1003 1002] replaced broker
  codfw.change-prop.wikidata.resource-change p0: [1003 1002 1001] -> [1005 1002 1003] replaced broker
  codfw.changeprop.error p0: [1003 1002 1001] -> [1003 1005 1002] replaced broker
  codfw.changeprop.retry.change-prop.backlinks.resource-change p0: [1001 1002 1003] -> [1001 1004 1003] replaced broker
  codfw.cpjobqueue.error p0: [1002 1003 1001] -> [1002 1004 1001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p0: [1002 1001 1003] -> [1004 1002 1003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p1: [1003 1002 1001] -> [1005 1001 1002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p2: [1001 1003 1002] -> [1003 1005 1001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p3: [1002 1003 1001] -> [1001 1003 1005] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p4: [1003 1001 1002] -> [1002 1003 1004] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p5: [1001 1002 1003] -> [1004 1002 1001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p6: [1002 1001 1003] -> [1005 1001 1002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p7: [1003 1002 1001] -> [1003 1004 1002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [1003 1001 1002] -> [1001 1005 1003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p1: [1001 1002 1003] -> [1002 1001 1005] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p2: [1002 1003 1001] -> [1004 1002 1001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p3: [1003 1002 1001] -> [1005 1003 1002] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p4: [1001 1003 1002] -> [1003 1004 1001] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p5: [1002 1001 1003] -> [1001 1005 1003] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p6: [1003 1001 1002] -> [1002 1003 1004] replaced broker
  codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks p7: [1001 1002 1003] -> [1004 1001 1003] replaced broker
  codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [1001 1002 1003] -> [1005 1002 1003] replaced broker
  codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob p0: [1001 1002 1003] -> [1003 1005 1002] replaced broker
  codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange p0: [1002 1003 1001] -> [1001 1004 1003] replaced broker
  codfw.eventgate-main.test.event p0: [1003 1001 1002] -> [1002 1001 1005] replaced broker
  codfw.mediawiki.job.AssembleUploadChunks p0: [1003 1001 1002] -> [1004 1003 1001] replaced broker
  codfw.mediawiki.job.CentralAuthCreateLocalAccountJob p0: [1002 1003 1001] -> [1005 1002 1001] replaced broker
  codfw.mediawiki.job.ChangeDeletionNotification p0: [1003 1002 1001] -> [1003 1005 1002] replaced broker
  codfw.mediawiki.job.ChangeNotification p0: [1002 1003 1001] -> [1001 1004 1002] replaced broker
  codfw.mediawiki.job.CleanTermsIfUnused p0: [1001 1002 1003] -> [1002 1001 1005] replaced broker
  codfw.mediawiki.job.CognateCacheUpdateJob p0: [1002 1001 1003] -> [1004 1002 1003] replaced broker
  codfw.mediawiki.job.CognateLocalJobSubmitJob p0: [1001 1003 1002] -> [1005 1003 1001] replaced broker
  codfw.mediawiki.job.EchoNotificationDeleteJob p0: [1003 1001 1002] -> [1003 1005 1001] replaced broker
  codfw.mediawiki.job.EchoPushNotificationRequest p0: [1002 1001 1003] -> [1001 1004 1002] replaced broker
  codfw.mediawiki.job.LoginNotifyChecks p0: [1002 1001 1003] -> [1002 1003 1004] replaced broker
  codfw.mediawiki.job.MessageGroupStatesUpdaterJob p0: [1003 1001 1002] -> [1004 1001 1003] replaced broker
  codfw.mediawiki.job.ORESFetchScoreJob p0: [1002 1003 1001] -> [1005 1002 1003] replaced broker
  codfw.mediawiki.job.RecordLintJob p0: [1002 1003 1001] -> [1003 1004 1001] replaced broker
  codfw.mediawiki.job.TTMServerMessageUpdateJob p0: [1003 1001 1002] -> [1001 1005 1002] replaced broker
  codfw.mediawiki.job.ThumbnailRender p0: [1001 1002 1003] -> [1002 1004 1003] replaced broker
  codfw.mediawiki.job.TranslateRenderJob p0: [1003 1002 1001] -> [1004 1003 1002] replaced broker
  codfw.mediawiki.job.UpdateRepoOnDelete p0: [1002 1003 1001] -> [1005 1002 1001] replaced broker
  codfw.mediawiki.job.UpdateRepoOnMove p0: [1002 1003 1001] -> [1003 1005 1001] replaced broker
  codfw.mediawiki.job.activityUpdateJob p0: [1001 1002 1003] -> [1001 1003 1004] replaced broker
  codfw.mediawiki.job.categoryMembershipChange p0: [1001 1003 1002] -> [1002 1001 1005] replaced broker
  codfw.mediawiki.job.cdnPurge p0: [1002 1001 1003] -> [1004 1001 1003] replaced broker
  codfw.mediawiki.job.cirrusSearchCheckerJob p0: [1002 1003 1001] -> [1005 1002 1003] replaced broker
  codfw.mediawiki.job.cirrusSearchDeletePages p0: [1001 1003 1002] -> [1003 1005 1002] replaced broker
  codfw.mediawiki.job.cirrusSearchIncomingLinkCount p0: [1001 1002 1003] -> [1001 1004 1002] replaced broker
  codfw.mediawiki.job.cirrusSearchLinksUpdate p0: [1003 1001 1002] -> [1002 1004 1001] replaced broker
  codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized p0: [1002 1001 1003] -> [1004 1003 1002] replaced broker
  codfw.mediawiki.job.cirrusSearchOtherIndex p0: [1003 1001 1002] -> [1005 1001 1003] replaced broker
  codfw.mediawiki.job.constraintsRunCheck p0: [1001 1002 1003] -> [1003 1005 1001] replaced broker
  codfw.mediawiki.job.enotifNotify p0: [1003 1002 1001] -> [1001 1002 1004] replaced broker
  codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations p0: [1001 1002 1003] -> [1002 1004 1001] replaced broker
  codfw.mediawiki.job.flaggedrevs_CacheUpdate p0: [1003 1001 1002] -> [1004 1001 1003] replaced broker
  codfw.mediawiki.job.globalUsageCachePurge p0: [1003 1001 1002] -> [1005 1003 1002] replaced broker
  codfw.mediawiki.job.htmlCacheUpdate p0: [1002 1003 1001] -> [1003 1002 1005] replaced broker
  codfw.mediawiki.job.newUserMessageJob p0: [1002 1003 1001] -> [1001 1005 1003] replaced broker
  codfw.mediawiki.job.newcomerTasksCacheRefreshJob p0: [1002 1003 1001] -> [1002 1003 1005] replaced broker
  codfw.mediawiki.job.recentChangesUpdate p0: [1001 1002 1003] -> [1004 1001 1002] replaced broker
  codfw.mediawiki.job.refreshLinks p0: [1001 1002 1003] -> [1005 1002 1001] replaced broker
  codfw.mediawiki.job.refreshLinksPrioritized p0: [1003 1001 1002] -> [1003 1004 1001] replaced broker
  codfw.mediawiki.job.revertedTagUpdate p0: [1001 1002 1003] -> [1001 1005 1002] replaced broker
  codfw.mediawiki.job.updateBetaFeaturesUserCounts p0: [1002 1001 1003] -> [1002 1003 1004] replaced broker
  codfw.mediawiki.job.userOptionsUpdate p0: [1002 1003 1001] -> [1004 1001 1003] replaced broker
  codfw.mediawiki.job.watchlistExpiry p0: [1002 1001 1003] -> [1005 1002 1001] replaced broker
  codfw.mediawiki.job.webVideoTranscodePrioritized p0: [1001 1002 1003] -> [1003 1005 1002] replaced broker
  codfw.mediawiki.job.wikibase-InjectRCRecords p0: [1001 1002 1003] -> [1001 1004 1003] replaced broker
  codfw.mediawiki.job.wikibase-addUsagesForPage p0: [1001 1002 1003] -> [1002 1005 1003] replaced broker
  codfw.mediawiki.page-create p0: [1001 1003 1002] -> [1004 1003 1002] replaced broker
  codfw.mediawiki.page-delete p0: [1001 1003 1002] -> [1005 1001 1002] replaced broker
  codfw.mediawiki.page-links-change p0: [1002 1001 1003] -> [1003 1004 1001] replaced broker
  codfw.mediawiki.page-move p0: [1003 1001 1002] -> [1001 1002 1005] replaced broker
  codfw.mediawiki.page-properties-change p0: [1002 1001 1003] -> [1002 1005 1003] replaced broker
  codfw.mediawiki.recentchange p0: [1002 1001 1003] -> [1004 1003 1001] replaced broker
  codfw.mediawiki.revision-create p0: [1003 1002 1001] -> [1005 1001 1003] replaced broker
  codfw.mediawiki.revision-recommendation-create p0: [1002 1001 1003] -> [1003 1004 1001] replaced broker
  codfw.mediawiki.revision-score p0: [1003 1002 1001] -> [1001 1002 1004] replaced broker
  codfw.mediawiki.revision-tags-change p0: [1001 1002 1003] -> [1002 1004 1003] replaced broker
  codfw.mediawiki.user-blocks-change p0: [1003 1002 1001] -> [1004 1001 1002] replaced broker
  codfw.rdf-streaming-updater.mutation p0: [1002 1001 1003] -> [1005 1003 1001] replaced broker
  codfw.rdf-streaming-updater.state-inconsistency p0: [1001 1002 1003] -> [1003 1002 1005] replaced broker
  codfw.resource_change p0: [1001 1003 1002] -> [1001 1005 1002] replaced broker
  codfw.resource_change p1: [1002 1003 1001] -> [1002 1003 1004] replaced broker
  codfw.resource_change p2: [1003 1001 1002] -> [1004 1001 1002] replaced broker
  eqiad.change-prop.backlinks.resource-change p0: [1001 1003 1002] -> [1005 1002 1001] replaced broker
  eqiad.change-prop.transcludes.resource-change p0: [1001 1003 1002] -> [1003 1004 1002] replaced broker
  eqiad.change-prop.transcludes.resource-change p1: [1002 1003 1004] -> [1001 1005 1003] replaced broker
  eqiad.change-prop.transcludes.resource-change p2: [1003 1004 1001] -> [1002 1003 1004] replaced broker
  eqiad.change-prop.wikidata.resource-change p0: [1001 1003 1002] -> [1004 1001 1003] replaced broker
  eqiad.changeprop.error p0: [1002 1003 1001] -> [1005 1002 1001] replaced broker
  eqiad.changeprop.retry.change-prop.backlinks.resource-change p0: [1002 1003 1001] -> [1003 1005 1002] replaced broker
  eqiad.cpjobqueue.error p0: [1001 1003 1002] -> [1001 1004 1003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p0: [1003 1001 1002] -> [1002 1001 1005] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p1: [1001 1002 1003] -> [1004 1003 1001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p2: [1002 1003 1001] -> [1005 1002 1001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p3: [1003 1002 1001] -> [1003 1005 1002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p4: [1001 1003 1002] -> [1001 1004 1003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p5: [1002 1001 1003] -> [1002 1003 1005] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p6: [1003 1001 1002] -> [1004 1002 1003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate p7: [1001 1002 1003] -> [1005 1001 1002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [1003 1001 1002] -> [1003 1005 1001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p1: [1001 1002 1003] -> [1001 1004 1003] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p2: [1002 1003 1001] -> [1002 1005 1001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p3: [1003 1002 1001] -> [1004 1002 1001] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p4: [1001 1003 1002] -> [1005 1003 1002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p5: [1002 1001 1003] -> [1003 1001 1004] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p6: [1003 1001 1002] -> [1001 1004 1002] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks p7: [1001 1002 1003] -> [1002 1003 1005] replaced broker
  eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks p0: [1003 1001 1002] -> [1004 1001 1003] replaced broker
  eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob p0: [1002 1001 1003] -> [1005 1002 1003] replaced broker
  eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange p0: [1002 1001 1003] -> [1003 1004 1001] replaced broker
  eqiad.eventgate-main.test.event p0: [1002 1001 1003] -> [1001 1005 1002] replaced broker
  eqiad.mediawiki.job.AssembleUploadChunks p0: [1003 1001 1002] -> [1002 1001 1004] replaced broker
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob p0: [1003 1001 1002] -> [1004 1003 1001] replaced broker
  eqiad.mediawiki.job.ChangeDeletionNotification p0: [1003 1002 1001] -> [1005 1002 1003] replaced broker
  eqiad.mediawiki.job.ChangeNotification p0: [1003 1002 1001] -> [1003 1005 1002] replaced broker
  eqiad.mediawiki.job.CleanTermsIfUnused p0: [1002 1003 1001] -> [1001 1004 1002] replaced broker
  eqiad.mediawiki.job.CognateCacheUpdateJob p0: [1003 1001 1002] -> [1002 1001 1005] replaced broker
  eqiad.mediawiki.job.CognateLocalJobSubmitJob p0: [1002 1003 1001] -> [1004 1003 1001] replaced broker
  eqiad.mediawiki.job.EchoNotificationDeleteJob p0: [1002 1003 1001] -> [1005 1002 1003] replaced broker
  eqiad.mediawiki.job.EchoPushNotificationRequest p0: [1002 1003 1001] -> [1003 1005 1002] replaced broker
  eqiad.mediawiki.job.LoginNotifyChecks p0: [1001 1003 1002] -> [1001 1004 1003] replaced broker
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob p0: [1002 1003 1001] -> [1002 1005 1001] replaced broker
  eqiad.mediawiki.job.ORESFetchScoreJob p0: [1003 1001 1002] -> [1004 1002 1003] replaced broker
  eqiad.mediawiki.job.RecordLintJob p0: [1002 1003 1001] -> [1005 1001 1002] replaced broker
  eqiad.mediawiki.job.TTMServerMessageUpdateJob p0: [1001 1002 1003] -> [1003 1004 1001] replaced broker
  eqiad.mediawiki.job.ThumbnailRender p0: [1003 1001 1002] -> [1001 1003 1004] replaced broker
  eqiad.mediawiki.job.TranslateRenderJob p0: [1002 1001 1003] -> [1002 1005 1003] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnDelete p0: [1002 1003 1001] -> [1004 1002 1001] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnMove p0: [1003 1001 1002] -> [1005 1003 1002] replaced broker
  eqiad.mediawiki.job.activityUpdateJob p0: [1001 1002 1003] -> [1003 1001 1004] replaced broker
  eqiad.mediawiki.job.categoryMembershipChange p0: [1001 1002 1003] -> [1001 1004 1003] replaced broker
  eqiad.mediawiki.job.cdnPurge p0: [1002 1003 1001] -> [1002 1003 1005] replaced broker
  eqiad.mediawiki.job.cirrusSearchCheckerJob p0: [1002 1003 1001] -> [1004 1001 1002] replaced broker
  eqiad.mediawiki.job.cirrusSearchDeletePages p0: [1001 1002 1003] -> [1005 1002 1001] replaced broker
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount p0: [1003 1001 1002] -> [1003 1004 1002] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdate p0: [1003 1001 1002] -> [1001 1005 1003] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized p0: [1003 1001 1002] -> [1002 1005 1001] replaced broker
  eqiad.mediawiki.job.cirrusSearchOtherIndex p0: [1003 1001 1002] -> [1004 1003 1001] replaced broker
  eqiad.mediawiki.job.constraintsRunCheck p0: [1002 1003 1001] -> [1005 1001 1003] replaced broker
  eqiad.mediawiki.job.enotifNotify p0: [1001 1002 1003] -> [1003 1004 1002] replaced broker
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations p0: [1002 1003 1001] -> [1001 1002 1005] replaced broker
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate p0: [1002 1003 1001] -> [1002 1004 1003] replaced broker
  eqiad.mediawiki.job.globalUsageCachePurge p0: [1002 1003 1001] -> [1004 1003 1002] replaced broker
  eqiad.mediawiki.job.htmlCacheUpdate p0: [1001 1003 1002] -> [1005 1001 1002] replaced broker
  eqiad.mediawiki.job.newUserMessageJob p0: [1001 1002 1003] -> [1003 1005 1001] replaced broker
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob p0: [1002 1001 1003] -> [1001 1002 1004] replaced broker
  eqiad.mediawiki.job.recentChangesUpdate p0: [1003 1001 1002] -> [1002 1005 1003] replaced broker
  eqiad.mediawiki.job.refreshLinks p0: [1002 1003 1001] -> [1004 1003 1001] replaced broker
  eqiad.mediawiki.job.refreshLinksPrioritized p0: [1001 1003 1002] -> [1005 1001 1002] replaced broker
  eqiad.mediawiki.job.revertedTagUpdate p0: [1002 1003 1001] -> [1003 1002 1005] replaced broker
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts p0: [1001 1002 1003] -> [1001 1004 1003] replaced broker
  eqiad.mediawiki.job.userOptionsUpdate p0: [1001 1002 1003] -> [1002 1004 1001] replaced broker
  eqiad.mediawiki.job.watchlistExpiry p0: [1002 1003 1001] -> [1004 1001 1003] replaced broker
  eqiad.mediawiki.job.webVideoTranscodePrioritized p0: [1003 1001 1002] -> [1005 1003 1001] replaced broker
  eqiad.mediawiki.job.wikibase-InjectRCRecords p0: [1003 1001 1002] -> [1003 1002 1004] replaced broker
  eqiad.mediawiki.job.wikibase-addUsagesForPage p0: [1002 1003 1001] -> [1001 1005 1002] replaced broker
  eqiad.mediawiki.page-create p0: [1003 1001 1002] -> [1002 1004 1001] replaced broker
  eqiad.mediawiki.page-delete p0: [1001 1002 1003] -> [1004 1001 1003] replaced broker
  eqiad.mediawiki.page-links-change p0: [1003 1001 1002] -> [1005 1003 1002] replaced broker
  eqiad.mediawiki.page-move p0: [1001 1003 1002] -> [1003 1005 1001] replaced broker
  eqiad.mediawiki.page-properties-change p0: [1003 1002 1001] -> [1001 1002 1005] replaced broker
  eqiad.mediawiki.recentchange p0: [1003 1001 1002] -> [1002 1004 1003] replaced broker
  eqiad.mediawiki.revision-create p0: [1003 1002 1001] -> [1004 1003 1002] replaced broker
  eqiad.mediawiki.revision-recommendation-create p0: [1001 1002 1003] -> [1005 1002 1001] replaced broker
  eqiad.mediawiki.revision-score p0: [1003 1001 1002] -> [1003 1005 1002] replaced broker
  eqiad.mediawiki.revision-tags-change p0: [1003 1001 1002] -> [1001 1004 1003] replaced broker
  eqiad.mediawiki.user-blocks-change p0: [1001 1003 1002] -> [1002 1001 1004] replaced broker
  eqiad.rdf-streaming-updater.mutation p0: [1002 1003 1001] -> [1004 1002 1001] replaced broker
  eqiad.rdf-streaming-updater.state-inconsistency p0: [1001 1002 1003] -> [1005 1001 1002] replaced broker
  eqiad.resource_change p0: [1002 1001 1003] -> [1003 1005 1001] replaced broker
  eqiad.resource_change p1: [1003 1002 1001] -> [1001 1003 1004] replaced broker
  eqiad.resource_change p2: [1001 1003 1002] -> [1002 1005 1003] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/4/3.20 -> 3/4/3.60
  -
  Broker 1001 - leader: 46, follower: 107, total: 153
  Broker 1002 - leader: 46, follower: 106, total: 152
  Broker 1003 - leader: 46, follower: 106, total: 152
  Broker 1004 - leader: 45, follower: 69, total: 114
  Broker 1005 - leader: 45, follower: 68, total: 113

WARN:
  [none]

New partition maps:
  /home/elukey/T225005/json/codfw.change-prop.wikidata.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.watchlistExpiry.json
  /home/elukey/T225005/json/eqiad.change-prop.wikidata.resource-change.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CleanTermsIfUnused.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.AssembleUploadChunks.json
  /home/elukey/T225005/json/codfw.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.globalUsageCachePurge.json
  /home/elukey/T225005/json/codfw.changeprop.error.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CognateLocalJobSubmitJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchDeletePages.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.wikibase-InjectRCRecords.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.newUserMessageJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.AssembleUploadChunks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CentralAuthCreateLocalAccountJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.TranslateRenderJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.updateBetaFeaturesUserCounts.json
  /home/elukey/T225005/json/eqiad.eventgate-main.test.event.json
  /home/elukey/T225005/json/codfw.cpjobqueue.error.json
  /home/elukey/T225005/json/codfw.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.wikibase-addUsagesForPage.json
  /home/elukey/T225005/json/codfw.mediawiki.page-delete.json
  /home/elukey/T225005/json/codfw.changeprop.retry.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchDeletePages.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.UpdateRepoOnMove.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-recommendation-create.json
  /home/elukey/T225005/json/codfw.mediawiki.job.constraintsRunCheck.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-score.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-tags-change.json
  /home/elukey/T225005/json/codfw.resource_change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
  /home/elukey/T225005/json/eqiad.resource_change.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-recommendation-create.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.activityUpdateJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.constraintsRunCheck.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-tags-change.json
  /home/elukey/T225005/json/eqiad.rdf-streaming-updater.state-inconsistency.json
  /home/elukey/T225005/json/codfw.mediawiki.job.UpdateRepoOnMove.json
  /home/elukey/T225005/json/codfw.mediawiki.job.newcomerTasksCacheRefreshJob.json
  /home/elukey/T225005/json/codfw.mediawiki.page-create.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ThumbnailRender.json
  /home/elukey/T225005/json/eqiad.mediawiki.recentchange.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.EchoPushNotificationRequest.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.TranslateRenderJob.json
  /home/elukey/T225005/json/codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.TTMServerMessageUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchIncomingLinkCount.json
  /home/elukey/T225005/json/codfw.mediawiki.job.newUserMessageJob.json
  /home/elukey/T225005/json/codfw.rdf-streaming-updater.mutation.json
  /home/elukey/T225005/json/codfw.mediawiki.job.EchoNotificationDeleteJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.EchoPushNotificationRequest.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchCheckerJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CleanTermsIfUnused.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchOtherIndex.json
  /home/elukey/T225005/json/codfw.mediawiki.user-blocks-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cdnPurge.json
  /home/elukey/T225005/json/codfw.mediawiki.job.UpdateRepoOnDelete.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-properties-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.user-blocks-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.refreshLinksPrioritized.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.RecordLintJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.userOptionsUpdate.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.error.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchLinksUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-links-change.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/codfw.eventgate-main.test.event.json
  /home/elukey/T225005/json/codfw.mediawiki.job.LoginNotifyChecks.json
  /home/elukey/T225005/json/eqiad.changeprop.retry.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/codfw.rdf-streaming-updater.state-inconsistency.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ChangeNotification.json
  /home/elukey/T225005/json/codfw.mediawiki.job.flaggedrevs_CacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ThumbnailRender.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.revertedTagUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-move.json
  /home/elukey/T225005/json/codfw.mediawiki.job.webVideoTranscodePrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-score.json
  /home/elukey/T225005/json/codfw.change-prop.transcludes.resource-change.json
  /home/elukey/T225005/json/codfw.mediawiki.job.CognateCacheUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.userOptionsUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ORESFetchScoreJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.watchlistExpiry.json
  /home/elukey/T225005/json/codfw.mediawiki.job.ChangeDeletionNotification.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchOtherIndex.json
  /home/elukey/T225005/json/codfw.mediawiki.page-move.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.LoginNotifyChecks.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchCheckerJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.UpdateRepoOnDelete.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.MessageGroupStatesUpdaterJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.wikibase-InjectRCRecords.json
  /home/elukey/T225005/json/codfw.mediawiki.page-properties-change.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
  /home/elukey/T225005/json/eqiad.rdf-streaming-updater.mutation.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ChangeDeletionNotification.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.EchoNotificationDeleteJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.webVideoTranscodePrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-create.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchLinksUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.job.revertedTagUpdate.json
  /home/elukey/T225005/json/codfw.mediawiki.revision-create.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.enotifNotify.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.globalUsageCachePurge.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.wikibase-addUsagesForPage.json
  /home/elukey/T225005/json/__consumer_offsets.json
  /home/elukey/T225005/json/codfw.mediawiki.job.categoryMembershipChange.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cdnPurge.json
  /home/elukey/T225005/json/codfw.mediawiki.page-links-change.json
  /home/elukey/T225005/json/eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.refreshLinksPrioritized.json
  /home/elukey/T225005/json/codfw.mediawiki.recentchange.json
  /home/elukey/T225005/json/eqiad.change-prop.backlinks.resource-change.json
  /home/elukey/T225005/json/eqiad.change-prop.transcludes.resource-change.json
  /home/elukey/T225005/json/eqiad.changeprop.error.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.recentChangesUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.revision-create.json
  /home/elukey/T225005/json/codfw.mediawiki.job.enotifNotify.json
  /home/elukey/T225005/json/codfw.mediawiki.job.recentChangesUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.ChangeNotification.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
  /home/elukey/T225005/json/eqiad.mediawiki.page-delete.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.CognateCacheUpdateJob.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.cirrusSearchIncomingLinkCount.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.htmlCacheUpdate.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.RecordLintJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.activityUpdateJob.json
  /home/elukey/T225005/json/codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
  /home/elukey/T225005/json/codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
  /home/elukey/T225005/json/eqiad.mediawiki.job.categoryMembershipChange.json
elukey added a comment.Oct 15 2021, 9:01 AM

In this case it is interesting to notice the difference between main-codfw and eqiad:

Broker change summary:
  New broker 2005
  New broker 2004
  -
  Replacing 0, added 2, missing 0, total count changed by 2

Broker change summary:
  Replacing 0, added 0, missing 0, total count changed by 0

But there is a clear move of partitions to kafka-main1004 and kafka-main1005, that is exactly what we needed (to balance out the number of partitions across brokers)

elukey added a comment.Oct 15 2021, 9:04 AM

Everything committed to https://gitlab.wikimedia.org/elukey/kafka_main_rebalance/-/tree/main/main-eqiad, I'll probably execute the plan on Monday.

elukey added a comment.Oct 15 2021, 4:29 PM

Topics moved today and their timings (main-eqiad):

Oct 15 16:25 codfw.mediawiki.job.cdnPurge.json
Oct 15 16:21 eqiad.mediawiki.job.cdnPurge.json
Oct 15 16:15 codfw.mediawiki.job.wikibase-addUsagesForPage.json
Oct 15 16:06 eqiad.mediawiki.job.wikibase-addUsagesForPage.json
Oct 15 16:06 codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 15 16:06 eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 15 16:05 codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 15 16:05 eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 15 16:05 codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 15 16:04 eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 15 16:04 codfw.change-prop.wikidata.resource-change.json
Oct 15 16:03 eqiad.change-prop.wikidata.resource-change.json
Oct 15 16:03 codfw.mediawiki.job.refreshLinksPrioritized.json
Oct 15 16:02 eqiad.mediawiki.job.refreshLinksPrioritized.json
Oct 15 16:00 codfw.mediawiki.page-properties-change.json
Oct 15 15:57 eqiad.mediawiki.page-properties-change.json
Oct 15 15:48 codfw.mediawiki.page-move.json
Oct 15 15:48 eqiad.mediawiki.page-move.json
Oct 15 15:47 codfw.mediawiki.page-links-change.json
Oct 15 15:45 eqiad.mediawiki.page-links-change.json
Oct 15 15:44 codfw.mediawiki.page-delete.json
Oct 15 15:43 eqiad.mediawiki.page-delete.json
Oct 15 15:42 codfw.eventgate-main.test.event.json
Oct 15 15:41 eqiad.eventgate-main.test.event.json
Oct 15 15:35 codfw.rdf-streaming-updater.mutation.json
Oct 15 15:31 eqiad.rdf-streaming-updater.mutation.json
Oct 15 15:31 codfw.mediawiki.user-blocks-change.json
Oct 15 15:30 eqiad.mediawiki.user-blocks-change.json
Oct 15 15:30 codfw.mediawiki.page-create.json
Oct 15 15:26 eqiad.mediawiki.page-create.json
Oct 15 15:23 codfw.mediawiki.job.wikibase-InjectRCRecords.json
Oct 15 15:20 eqiad.mediawiki.job.wikibase-InjectRCRecords.json
Oct 15 14:46 codfw.mediawiki.job.webVideoTranscodePrioritized.json
Oct 15 14:46 eqiad.mediawiki.job.webVideoTranscodePrioritized.json
Oct 15 14:45 codfw.mediawiki.job.watchlistExpiry.json
Oct 15 14:45 eqiad.mediawiki.job.watchlistExpiry.json
Oct 15 14:44 codfw.mediawiki.job.userOptionsUpdate.json
Oct 15 14:44 eqiad.mediawiki.job.userOptionsUpdate.json
Oct 15 14:44 codfw.mediawiki.job.UpdateRepoOnMove.json
Oct 15 14:43 eqiad.mediawiki.job.UpdateRepoOnMove.json
Oct 15 14:43 codfw.mediawiki.job.UpdateRepoOnDelete.json
Oct 15 14:42 eqiad.mediawiki.job.UpdateRepoOnDelete.json
Oct 15 14:40 codfw.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 15 14:40 eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 15 14:37 codfw.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 15 14:36 eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 15 14:35 codfw.mediawiki.job.TranslateRenderJob.json
Oct 15 14:35 eqiad.mediawiki.job.TranslateRenderJob.json
Oct 15 14:34 codfw.mediawiki.job.ThumbnailRender.json
Oct 15 14:34 eqiad.mediawiki.job.ThumbnailRender.json
Oct 15 14:34 codfw.mediawiki.job.revertedTagUpdate.json
Oct 15 14:33 eqiad.mediawiki.job.revertedTagUpdate.json
Oct 15 14:29 codfw.mediawiki.job.ORESFetchScoreJob.json
Oct 15 14:27 eqiad.mediawiki.job.ORESFetchScoreJob.json
Oct 15 14:27 codfw.mediawiki.job.newUserMessageJob.json
Oct 15 14:27 eqiad.mediawiki.job.newUserMessageJob.json
Oct 15 14:26 codfw.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 15 14:25 eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 15 14:25 codfw.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 15 14:24 eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 15 14:24 codfw.mediawiki.job.LoginNotifyChecks.json
Oct 15 14:23 eqiad.mediawiki.job.LoginNotifyChecks.json
Oct 15 14:23 codfw.mediawiki.job.globalUsageCachePurge.json
Oct 15 14:23 eqiad.mediawiki.job.globalUsageCachePurge.json
Oct 15 14:22 codfw.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 15 14:21 eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 15 14:21 codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 15 14:21 eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 15 14:19 codfw.mediawiki.job.enotifNotify.json
Oct 15 14:18 eqiad.mediawiki.job.enotifNotify.json
Oct 15 14:18 codfw.mediawiki.job.EchoPushNotificationRequest.json
Oct 15 14:17 eqiad.mediawiki.job.EchoPushNotificationRequest.json
Oct 15 14:17 codfw.mediawiki.job.EchoNotificationDeleteJob.json
Oct 15 14:15 eqiad.mediawiki.job.EchoNotificationDeleteJob.json
Oct 15 14:14 codfw.mediawiki.job.constraintsRunCheck.json
Oct 15 14:14 eqiad.mediawiki.job.constraintsRunCheck.json
Oct 15 14:11 codfw.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 15 14:10 eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 15 14:09 codfw.mediawiki.job.CognateCacheUpdateJob.json
Oct 15 14:09 eqiad.mediawiki.job.CognateCacheUpdateJob.json
Oct 15 14:08 codfw.mediawiki.job.CleanTermsIfUnused.json
Oct 15 14:08 eqiad.mediawiki.job.CleanTermsIfUnused.json
Oct 15 14:07 codfw.mediawiki.job.cirrusSearchOtherIndex.json
Oct 15 14:07 eqiad.mediawiki.job.cirrusSearchOtherIndex.json
Oct 15 14:06 codfw.mediawiki.job.cirrusSearchDeletePages.json
Oct 15 14:06 eqiad.mediawiki.job.cirrusSearchDeletePages.json
Oct 15 14:05 codfw.mediawiki.job.cirrusSearchCheckerJob.json
Oct 15 14:03 eqiad.mediawiki.job.cirrusSearchCheckerJob.json
Oct 15 14:02 codfw.mediawiki.job.ChangeNotification.json
Oct 15 14:01 eqiad.mediawiki.job.ChangeNotification.json
Oct 15 14:01 codfw.mediawiki.job.ChangeDeletionNotification.json
Oct 15 14:00 eqiad.mediawiki.job.ChangeDeletionNotification.json
Oct 15 13:56 codfw.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 15 13:56 eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 15 13:55 codfw.mediawiki.job.categoryMembershipChange.json
Oct 15 13:49 eqiad.mediawiki.job.categoryMembershipChange.json
Oct 15 13:48 codfw.mediawiki.job.AssembleUploadChunks.json
Oct 15 13:48 eqiad.mediawiki.job.AssembleUploadChunks.json
Oct 15 13:46 codfw.mediawiki.job.activityUpdateJob.json
Oct 15 13:45 eqiad.mediawiki.job.activityUpdateJob.json
Oct 15 13:44 eqiad.cpjobqueue.error.json
Oct 15 13:44 codfw.cpjobqueue.error.json
Oct 15 13:43 eqiad.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 15 13:43 codfw.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 15 13:42 eqiad.changeprop.error.json
Oct 15 13:42 codfw.changeprop.error.json
Oct 15 13:41 eqiad.change-prop.backlinks.resource-change.json
Oct 15 13:40 codfw.change-prop.backlinks.resource-change.json

Will proceed with the rest of the topics on Monday!

elukey added a comment.Oct 18 2021, 1:47 PM

main-eqiad is done, I've done this extra moves for the statsv topics (in both clusters):

{
  "partitions": [
    {"topic": "statsv", "partition": 0, "replicas": [2005,2002,2003]}
  ],
  "version":1
}

{
  "partitions": [
    {"topic": "statsv", "partition": 0, "replicas": [1005,1002,1001]}
  ],
  "version":1
}
elukey added a comment.Oct 18 2021, 2:05 PM

The topic moves have been completed for kafka main-eqiad, here's a list of timings of when the rebalance kicked off for each topic:

Oct 18 13:30 statsv.json
Oct 18 13:08 codfw.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
Oct 18 12:59 eqiad.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate.json
Oct 18 10:36 codfw.change-prop.transcludes.resource-change.json
Oct 18 10:04 eqiad.change-prop.transcludes.resource-change.json
Oct 18 10:04 codfw.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 18 09:56 eqiad.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 18 09:55 codfw.resource_change.json
Oct 18 09:31 eqiad.resource_change.json
Oct 18 09:30 codfw.mediawiki.revision-tags-change.json
Oct 18 09:26 eqiad.mediawiki.revision-tags-change.json
Oct 18 09:26 codfw.mediawiki.revision-score.json
Oct 18 09:02 eqiad.mediawiki.revision-score.json
Oct 18 09:02 codfw.mediawiki.job.refreshLinks.json
Oct 18 08:41 eqiad.mediawiki.job.refreshLinks.json
Oct 18 08:40 codfw.mediawiki.job.recentChangesUpdate.json
Oct 18 08:35 eqiad.mediawiki.job.recentChangesUpdate.json
Oct 18 08:34 codfw.mediawiki.job.htmlCacheUpdate.json
Oct 18 08:18 eqiad.mediawiki.job.htmlCacheUpdate.json
Oct 18 08:15 codfw.mediawiki.recentchange.json
Oct 18 07:56 eqiad.mediawiki.recentchange.json
Oct 18 07:55 codfw.mediawiki.revision-create.json
Oct 18 07:41 eqiad.mediawiki.revision-create.json
Oct 18 07:39 codfw.mediawiki.job.RecordLintJob.json
Oct 18 07:05 eqiad.mediawiki.job.RecordLintJob.json
Oct 18 07:04 codfw.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
Oct 18 07:01 eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
Oct 18 07:00 codfw.mediawiki.job.cirrusSearchLinksUpdate.json
Oct 18 06:34 eqiad.mediawiki.job.cirrusSearchLinksUpdate.json
Oct 18 06:31 codfw.mediawiki.job.cirrusSearchIncomingLinkCount.json
Oct 18 06:19 eqiad.mediawiki.job.cirrusSearchIncomingLinkCount.json
Oct 18 06:19 codfw.rdf-streaming-updater.state-inconsistency.json
Oct 18 06:18 eqiad.rdf-streaming-updater.state-inconsistency.json
Oct 18 06:18 codfw.mediawiki.revision-recommendation-create.json
Oct 18 06:18 eqiad.mediawiki.revision-recommendation-create.json
Oct 15 16:25 codfw.mediawiki.job.cdnPurge.json
Oct 15 16:21 eqiad.mediawiki.job.cdnPurge.json
Oct 15 16:15 codfw.mediawiki.job.wikibase-addUsagesForPage.json
Oct 15 16:06 eqiad.mediawiki.job.wikibase-addUsagesForPage.json
Oct 15 16:06 codfw.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 15 16:06 eqiad.cpjobqueue.retry.mediawiki.job.ORESFetchScoreJob.json
Oct 15 16:05 codfw.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 15 16:05 eqiad.cpjobqueue.retry.mediawiki.job.categoryMembershipChange.json
Oct 15 16:05 codfw.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 15 16:04 eqiad.cpjobqueue.retry.cpjobqueue.partitioned.mediawiki.job.refreshLinks.json
Oct 15 16:04 codfw.change-prop.wikidata.resource-change.json
Oct 15 16:03 eqiad.change-prop.wikidata.resource-change.json
Oct 15 16:03 codfw.mediawiki.job.refreshLinksPrioritized.json
Oct 15 16:02 eqiad.mediawiki.job.refreshLinksPrioritized.json
Oct 15 16:00 codfw.mediawiki.page-properties-change.json
Oct 15 15:57 eqiad.mediawiki.page-properties-change.json
Oct 15 15:48 codfw.mediawiki.page-move.json
Oct 15 15:48 eqiad.mediawiki.page-move.json
Oct 15 15:47 codfw.mediawiki.page-links-change.json
Oct 15 15:45 eqiad.mediawiki.page-links-change.json
Oct 15 15:44 codfw.mediawiki.page-delete.json
Oct 15 15:43 eqiad.mediawiki.page-delete.json
Oct 15 15:42 codfw.eventgate-main.test.event.json
Oct 15 15:41 eqiad.eventgate-main.test.event.json
Oct 15 15:35 codfw.rdf-streaming-updater.mutation.json
Oct 15 15:31 eqiad.rdf-streaming-updater.mutation.json
Oct 15 15:31 codfw.mediawiki.user-blocks-change.json
Oct 15 15:30 eqiad.mediawiki.user-blocks-change.json
Oct 15 15:30 codfw.mediawiki.page-create.json
Oct 15 15:26 eqiad.mediawiki.page-create.json
Oct 15 15:23 codfw.mediawiki.job.wikibase-InjectRCRecords.json
Oct 15 15:20 eqiad.mediawiki.job.wikibase-InjectRCRecords.json
Oct 15 14:46 codfw.mediawiki.job.webVideoTranscodePrioritized.json
Oct 15 14:46 eqiad.mediawiki.job.webVideoTranscodePrioritized.json
Oct 15 14:45 codfw.mediawiki.job.watchlistExpiry.json
Oct 15 14:45 eqiad.mediawiki.job.watchlistExpiry.json
Oct 15 14:44 codfw.mediawiki.job.userOptionsUpdate.json
Oct 15 14:44 eqiad.mediawiki.job.userOptionsUpdate.json
Oct 15 14:44 codfw.mediawiki.job.UpdateRepoOnMove.json
Oct 15 14:43 eqiad.mediawiki.job.UpdateRepoOnMove.json
Oct 15 14:43 codfw.mediawiki.job.UpdateRepoOnDelete.json
Oct 15 14:42 eqiad.mediawiki.job.UpdateRepoOnDelete.json
Oct 15 14:40 codfw.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 15 14:40 eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
Oct 15 14:37 codfw.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 15 14:36 eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
Oct 15 14:35 codfw.mediawiki.job.TranslateRenderJob.json
Oct 15 14:35 eqiad.mediawiki.job.TranslateRenderJob.json
Oct 15 14:34 codfw.mediawiki.job.ThumbnailRender.json
Oct 15 14:34 eqiad.mediawiki.job.ThumbnailRender.json
Oct 15 14:34 codfw.mediawiki.job.revertedTagUpdate.json
Oct 15 14:33 eqiad.mediawiki.job.revertedTagUpdate.json
Oct 15 14:29 codfw.mediawiki.job.ORESFetchScoreJob.json
Oct 15 14:27 eqiad.mediawiki.job.ORESFetchScoreJob.json
Oct 15 14:27 codfw.mediawiki.job.newUserMessageJob.json
Oct 15 14:27 eqiad.mediawiki.job.newUserMessageJob.json
Oct 15 14:26 codfw.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 15 14:25 eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
Oct 15 14:25 codfw.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 15 14:24 eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
Oct 15 14:24 codfw.mediawiki.job.LoginNotifyChecks.json
Oct 15 14:23 eqiad.mediawiki.job.LoginNotifyChecks.json
Oct 15 14:23 codfw.mediawiki.job.globalUsageCachePurge.json
Oct 15 14:23 eqiad.mediawiki.job.globalUsageCachePurge.json
Oct 15 14:22 codfw.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 15 14:21 eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
Oct 15 14:21 codfw.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 15 14:21 eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
Oct 15 14:19 codfw.mediawiki.job.enotifNotify.json
Oct 15 14:18 eqiad.mediawiki.job.enotifNotify.json
Oct 15 14:18 codfw.mediawiki.job.EchoPushNotificationRequest.json
Oct 15 14:17 eqiad.mediawiki.job.EchoPushNotificationRequest.json
Oct 15 14:17 codfw.mediawiki.job.EchoNotificationDeleteJob.json
Oct 15 14:15 eqiad.mediawiki.job.EchoNotificationDeleteJob.json
Oct 15 14:14 codfw.mediawiki.job.constraintsRunCheck.json
Oct 15 14:14 eqiad.mediawiki.job.constraintsRunCheck.json
Oct 15 14:11 codfw.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 15 14:10 eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
Oct 15 14:09 codfw.mediawiki.job.CognateCacheUpdateJob.json
Oct 15 14:09 eqiad.mediawiki.job.CognateCacheUpdateJob.json
Oct 15 14:08 codfw.mediawiki.job.CleanTermsIfUnused.json
Oct 15 14:08 eqiad.mediawiki.job.CleanTermsIfUnused.json
Oct 15 14:07 codfw.mediawiki.job.cirrusSearchOtherIndex.json
Oct 15 14:07 eqiad.mediawiki.job.cirrusSearchOtherIndex.json
Oct 15 14:06 codfw.mediawiki.job.cirrusSearchDeletePages.json
Oct 15 14:06 eqiad.mediawiki.job.cirrusSearchDeletePages.json
Oct 15 14:05 codfw.mediawiki.job.cirrusSearchCheckerJob.json
Oct 15 14:03 eqiad.mediawiki.job.cirrusSearchCheckerJob.json
Oct 15 14:02 codfw.mediawiki.job.ChangeNotification.json
Oct 15 14:01 eqiad.mediawiki.job.ChangeNotification.json
Oct 15 14:01 codfw.mediawiki.job.ChangeDeletionNotification.json
Oct 15 14:00 eqiad.mediawiki.job.ChangeDeletionNotification.json
Oct 15 13:56 codfw.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 15 13:56 eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
Oct 15 13:55 codfw.mediawiki.job.categoryMembershipChange.json
Oct 15 13:49 eqiad.mediawiki.job.categoryMembershipChange.json
Oct 15 13:48 codfw.mediawiki.job.AssembleUploadChunks.json
Oct 15 13:48 eqiad.mediawiki.job.AssembleUploadChunks.json
Oct 15 13:46 codfw.mediawiki.job.activityUpdateJob.json
Oct 15 13:45 eqiad.mediawiki.job.activityUpdateJob.json
Oct 15 13:44 eqiad.cpjobqueue.error.json
Oct 15 13:44 codfw.cpjobqueue.error.json
Oct 15 13:43 eqiad.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 15 13:43 codfw.changeprop.retry.change-prop.backlinks.resource-change.json
Oct 15 13:42 eqiad.changeprop.error.json
Oct 15 13:42 codfw.changeprop.error.json
Oct 15 13:41 eqiad.change-prop.backlinks.resource-change.json
Oct 15 13:40 codfw.change-prop.backlinks.resource-change.json
elukey closed this task as Resolved.Oct 18 2021, 2:11 PM
elukey claimed this task.

Both clusters done, there is still some unbalance between the workers but for now it seems good enough for our use cases. It is a good trade-off between fine tuning and overall scalability, but we may want to invest some time in automation tools able to automatically suggest topic/partition moves when needed. We could explore the possibility of using topicmappr with prometheus metrics for example.

Please re-open if anything is missing :)

herron awarded a token.Oct 18 2021, 2:16 PM
Ottomata mentioned this in T292048: Queuing jobs is extremely slow.Nov 12 2021, 2:19 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL