Decommission kafka-jumbo100[1-6]
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	BTullis
	May 5 2023, 10:05 AM

Description

The kafka-jumbo100[1-6] servers are due to be decommissioned.

However, this can only be carried out when T336041: Bring kafka-jumbo10[09-15] into service has been accomplished.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Drop kafka-jumbo100[1-6].eqiad.wmnet from the puppet site	operations/puppet	production	+1 -1
Reflect the fact that kafka-jumbo1001.eqiad.wmnet has been decommissioned	wikidata/query/rdf	master	+1 -1
Replace the IPs of kafka-jumbo100[1-6] with kafka-jumbo101[0-5]	operations/deployment-charts	master	+12 -12
Remove kafka-jumbo100[1-6] brokers from the inventory	operations/puppet	production	+0 -30
Remove kafka-jumbo100[1-6] from puppet	operations/puppet	production	+1 -39
Remove kafka-jumbo100[1-6] brokers from bootstrap hosts	operations/deployment-charts	master	+8 -8
Remove kafka-jumbo100[1-6] brokers from bootstrap hosts	analytics/refinery	master	+4 -4
Change the underlying host for the kafka-jumbo-canary cumin alias	operations/puppet	production	+1 -1
Change the kafka-jumbo bootstrap host for the analytics cluster hosts	operations/puppet	production	+1 -1
Change the kafka-jumbo bootstrap host for karapace	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T336041 Bring kafka-jumbo10[09-15] into service
Resolved	brouberol	T346425 Reassign partitions away from kafka-jumbo100[1-6] to kafka-jumbo10[07-15] brokers
Resolved	brouberol	T336044 Decommission kafka-jumbo100[1-6]
Resolved	brouberol	T348495 Upgrade kafka-jumbo100[7-9] to Debian Bullseye

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Next batch

rouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.android.image_recommendation_event|eqiad.android.image_recommendation_interaction|eqiad.android.install_referrer_event|eqiad.android.metrics_platform.article_link_preview_interaction|eqiad.android.metrics_platform.article_toc_interaction|eqiad.android.metrics_platform.article_toolbar_interaction|eqiad.android.metrics_platform.find_in_page_interaction|eqiad.android.notification_interaction|eqiad.android.reading_list_interaction|eqiad.android.user_contribution_screen|eqiad.api-gateway.request|eqiad.app_session|eqiad.change-prop.backlinks.resource-change|eqiad.change-prop.error|eqiad.change-prop.retry.change-prop.backlinks.resource-change|eqiad.change-prop.retry.change-prop.transcludes.resource-change|eqiad.change-prop.retry.change-prop.wikidata.resource-change|eqiad.change-prop.retry.mediawiki.job.MessageIndexRebuildJob|eqiad.change-prop.retry.mediawiki.job.RecordLintJob|eqiad.change-prop.retry.mediawiki.job.cdnPurge|eqiad.change-prop.retry.mediawiki.job.deleteLinks)$'   --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch2

Topics:
  eqiad.android.image_recommendation_event
  eqiad.android.image_recommendation_interaction
  ...
  eqiad.change-prop.retry.mediawiki.job.cdnPurge
  eqiad.change-prop.retry.mediawiki.job.deleteLinks

Broker change summary:
  Broker 1002 marked for removal
  Broker 1006 marked for removal
  Broker 1001 marked for removal
  Broker 1005 marked for removal
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  New broker 1014
  New broker 1015
  New broker 1010
  New broker 1012
  New broker 1013
  New broker 1011
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.android.image_recommendation_event p0: [1007 1009 1001] -> [1011 1007 1009] replaced broker
  eqiad.android.image_recommendation_interaction p0: [1002 1005 1008] -> [1012 1015 1008] replaced broker
  eqiad.android.install_referrer_event p0: [1008 1007 1001] -> [1007 1015 1008] replaced broker
  eqiad.android.metrics_platform.article_link_preview_interaction p0: [1009 1001 1003] -> [1011 1013 1009] replaced broker
  eqiad.android.metrics_platform.article_toc_interaction p0: [1005 1003 1006] -> [1013 1008 1012] replaced broker
  eqiad.android.metrics_platform.article_toolbar_interaction p0: [1009 1001 1003] -> [1014 1011 1009] replaced broker
  eqiad.android.metrics_platform.find_in_page_interaction p0: [1008 1007 1001] -> [1008 1007 1014] replaced broker
  eqiad.android.notification_interaction p0: [1007 1006 1002] -> [1007 1010 1015] replaced broker
  eqiad.android.reading_list_interaction p0: [1007 1009 1001] -> [1009 1007 1012] replaced broker
  eqiad.android.user_contribution_screen p0: [1001 1003 1004] -> [1008 1010 1007] replaced broker
  eqiad.api-gateway.request p0: [1006 1002 1005] -> [1015 1009 1011] replaced broker
  eqiad.app_session p0: [1009 1001 1003] -> [1010 1009 1014] replaced broker
  eqiad.change-prop.backlinks.resource-change p0: [1005 1001 1003] -> [1013 1009 1011] replaced broker
  eqiad.change-prop.error p0: [1003 1004 1006] -> [1010 1008 1014] replaced broker
  eqiad.change-prop.retry.change-prop.backlinks.resource-change p0: [1004 1002 1003] -> [1007 1012 1015] replaced broker
  eqiad.change-prop.retry.change-prop.transcludes.resource-change p0: [1002 1003 1004] -> [1012 1013 1008] replaced broker
  eqiad.change-prop.retry.change-prop.wikidata.resource-change p0: [1005 1003 1006] -> [1009 1013 1010] replaced broker
  eqiad.change-prop.retry.mediawiki.job.MessageIndexRebuildJob p0: [1002 1006 1005] -> [1011 1014 1007] replaced broker
  eqiad.change-prop.retry.mediawiki.job.RecordLintJob p0: [1001 1005 1003] -> [1013 1008 1011] replaced broker
  eqiad.change-prop.retry.mediawiki.job.cdnPurge p0: [1002 1003 1004] -> [1009 1014 1012] replaced broker
  eqiad.change-prop.retry.mediawiki.job.deleteLinks p0: [1006 1003 1004] -> [1015 1010 1007] replaced broker

Broker distribution:
  degree [min/max/avg]: 3/6/4.89 -> 5/7/6.22
  -
  Broker 1007 - leader: 3, follower: 6, total: 9
  Broker 1008 - leader: 2, follower: 6, total: 8
  Broker 1009 - leader: 3, follower: 6, total: 9
  Broker 1010 - leader: 2, follower: 4, total: 6
  Broker 1011 - leader: 3, follower: 4, total: 7
  Broker 1012 - leader: 2, follower: 4, total: 6
  Broker 1013 - leader: 3, follower: 3, total: 6
  Broker 1014 - leader: 1, follower: 5, total: 6
  Broker 1015 - leader: 2, follower: 4, total: 6

WARN:
  [none]

New partition maps:
  e-batch2.json [combined map]
...
  eqiad.android.user_contribution_screen.json

Next batch

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate|eqiad.change-prop.retry.mediawiki.job.htmlCacheUpdate|eqiad.change-prop.retry.mediawiki.job.refreshLinks|eqiad.change-prop.retry.mediawiki.job.refreshLinksPrioritized|eqiad.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts|eqiad.change-prop.retry.mediawiki.job.wikibase-addUsagesForPage|eqiad.change-prop.retry.mediawiki.page-create|eqiad.change-prop.retry.mediawiki.page-delete|eqiad.change-prop.retry.mediawiki.page-move|eqiad.change-prop.retry.mediawiki.page-properties-change|eqiad.change-prop.retry.mediawiki.page-undelete|eqiad.change-prop.retry.mediawiki.revision-create|eqiad.change-prop.retry.mediawiki.revision-visibility-change|eqiad.change-prop.retry.resource_change|eqiad.change-prop.transcludes.resource-change|eqiad.change-prop.wikidata.resource-change|eqiad.changeprop.error|eqiad.changeprop.retry.mediawiki.page-create|eqiad.changeprop.retry.mediawiki.page-delete|eqiad.changeprop.retry.mediawiki.page-move|eqiad.changeprop.retry.mediawiki.page-properties-change)$'   --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch3

Topics:
  eqiad.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate
  eqiad.change-prop.retry.mediawiki.job.htmlCacheUpdate
  eqiad.change-prop.retry.mediawiki.job.refreshLinks
  eqiad.change-prop.retry.mediawiki.job.refreshLinksPrioritized
  eqiad.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts
  eqiad.change-prop.retry.mediawiki.job.wikibase-addUsagesForPage
  eqiad.change-prop.retry.mediawiki.page-create
  eqiad.change-prop.retry.mediawiki.page-delete
  eqiad.change-prop.retry.mediawiki.page-move
  eqiad.change-prop.retry.mediawiki.page-properties-change
  eqiad.change-prop.retry.mediawiki.page-undelete
  eqiad.change-prop.retry.mediawiki.revision-create
  eqiad.change-prop.retry.mediawiki.revision-visibility-change
  eqiad.change-prop.retry.resource_change
  eqiad.change-prop.transcludes.resource-change
  eqiad.change-prop.wikidata.resource-change
  eqiad.changeprop.error
  eqiad.changeprop.retry.mediawiki.page-create
  eqiad.changeprop.retry.mediawiki.page-delete
  eqiad.changeprop.retry.mediawiki.page-move
  eqiad.changeprop.retry.mediawiki.page-properties-change

Broker change summary:
  Broker 1001 marked for removal
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  Broker 1006 marked for removal
  New broker 1007
  New broker 1009
  New broker 1008
  New broker 1010
  New broker 1014
  New broker 1012
  New broker 1013
  New broker 1011
  New broker 1015
  -
  Replacing 6, added 9, missing 0, total count changed by 3

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate p0: [1002 1005 1003] -> [1008 1012 1015] replaced broker
  eqiad.change-prop.retry.mediawiki.job.htmlCacheUpdate p0: [1004 1002 1003] -> [1012 1009 1013] replaced broker
  eqiad.change-prop.retry.mediawiki.job.refreshLinks p0: [1006 1002 1005] -> [1009 1014 1007] replaced broker
  eqiad.change-prop.retry.mediawiki.job.refreshLinksPrioritized p0: [1004 1003 1006] -> [1011 1015 1009] replaced broker
  eqiad.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts p0: [1006 1001 1003] -> [1014 1011 1008] replaced broker
  eqiad.change-prop.retry.mediawiki.job.wikibase-addUsagesForPage p0: [1006 1002 1005] -> [1015 1009 1010] replaced broker
  eqiad.change-prop.retry.mediawiki.page-create p0: [1002 1004 1006] -> [1009 1013 1010] replaced broker
  eqiad.change-prop.retry.mediawiki.page-delete p0: [1002 1005 1003] -> [1007 1014 1012] replaced broker
  eqiad.change-prop.retry.mediawiki.page-move p0: [1001 1004 1006] -> [1014 1009 1011] replaced broker
  eqiad.change-prop.retry.mediawiki.page-properties-change p0: [1005 1003 1006] -> [1013 1010 1008] replaced broker
  eqiad.change-prop.retry.mediawiki.page-undelete p0: [1004 1003 1006] -> [1008 1011 1013] replaced broker
  eqiad.change-prop.retry.mediawiki.revision-create p0: [1004 1002 1003] -> [1010 1014 1007] replaced broker
  eqiad.change-prop.retry.mediawiki.revision-visibility-change p0: [1005 1003 1006] -> [1007 1012 1015] replaced broker
  eqiad.change-prop.retry.resource_change p0: [1001 1003 1004] -> [1011 1007 1013] replaced broker
  eqiad.change-prop.transcludes.resource-change p0: [1006 1002 1005] -> [1012 1009 1015] replaced broker
  eqiad.change-prop.wikidata.resource-change p0: [1005 1001 1003] -> [1014 1007 1010] replaced broker
  eqiad.changeprop.error p0: [1004 1006 1002] -> [1013 1012 1008] replaced broker
  eqiad.changeprop.retry.mediawiki.page-create p0: [1004 1006 1002] -> [1007 1010 1014] replaced broker
  eqiad.changeprop.retry.mediawiki.page-delete p0: [1006 1004 1002] -> [1015 1008 1011] replaced broker
  eqiad.changeprop.retry.mediawiki.page-move p0: [1002 1004 1006] -> [1009 1011 1013] replaced broker
  eqiad.changeprop.retry.mediawiki.page-properties-change p0: [1004 1003 1006] -> [1010 1008 1007] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/5/4.33 -> 6/8/6.44
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 2, follower: 5, total: 7
  Broker 1009 - leader: 3, follower: 5, total: 8
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 2, follower: 5, total: 7
  Broker 1012 - leader: 2, follower: 4, total: 6
  Broker 1013 - leader: 2, follower: 5, total: 7
  Broker 1014 - leader: 3, follower: 4, total: 7
  Broker 1015 - leader: 2, follower: 4, total: 6

WARN:
  [none]

New partition maps:
  e-batch3.json [combined map]
  eqiad.change-prop.retry.mediawiki.page-undelete.json
  eqiad.change-prop.retry.mediawiki.revision-create.json
  eqiad.changeprop.error.json
  eqiad.changeprop.retry.mediawiki.page-properties-change.json
  eqiad.change-prop.retry.mediawiki.job.refreshLinks.json
  eqiad.change-prop.retry.mediawiki.job.updateBetaFeaturesUserCounts.json
  eqiad.change-prop.retry.mediawiki.job.wikibase-addUsagesForPage.json
  eqiad.change-prop.retry.mediawiki.page-create.json
  eqiad.change-prop.retry.mediawiki.page-delete.json
  eqiad.change-prop.wikidata.resource-change.json
  eqiad.change-prop.retry.resource_change.json
  eqiad.change-prop.transcludes.resource-change.json
  eqiad.changeprop.retry.mediawiki.page-create.json
  eqiad.changeprop.retry.mediawiki.page-delete.json
  eqiad.change-prop.retry.mediawiki.job.htmlCacheUpdate.json
  eqiad.change-prop.retry.mediawiki.job.refreshLinksPrioritized.json
  eqiad.change-prop.retry.mediawiki.page-properties-change.json
  eqiad.changeprop.retry.mediawiki.page-move.json
  eqiad.change-prop.retry.mediawiki.job.flaggedrevs_CacheUpdate.json
  eqiad.change-prop.retry.mediawiki.page-move.json
  eqiad.change-prop.retry.mediawiki.revision-visibility-change.json

Next batch

topicmappr rebuild  --topics '^(eqiad.changeprop.retry.mediawiki.revision-create|eqiad.changeprop.retry.mediawiki.revision-visibility-change|eqiad.changeprop.retry.resource_change|eqiad.cirrussearch.page-index-update|eqiad.cirrussearch.update_pipeline.fetch_error.rc0|eqiad.cirrussearch.update_pipeline.fetch_failure|eqiad.cirrussearch.update_pipeline.update.rc0|eqiad.cpjobqueue.error|eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite|eqiad.cqs-external.sparql-query|eqiad.development.network.probe|eqiad.eqiad|eqiad.error|eqiad.eventgate-analytics-external.error.validation|eqiad.eventgate-analytics-external.test.event|eqiad.eventgate-analytics-external.validation.error|eqiad.eventgate-analytics.error.validation|eqiad.eventgate-analytics.test.event|eqiad.eventgate-logging-external.test.event|eqiad.eventgate-main.error.validation|eqiad.eventgate-main.test.even)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch4

Topics:
  eqiad.changeprop.retry.mediawiki.revision-create
  eqiad.changeprop.retry.mediawiki.revision-visibility-change
  eqiad.changeprop.retry.resource_change
  eqiad.cirrussearch.page-index-update
  eqiad.cirrussearch.update_pipeline.fetch_error.rc0
  eqiad.cirrussearch.update_pipeline.fetch_failure
  eqiad.cirrussearch.update_pipeline.update.rc0
  eqiad.cpjobqueue.error
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite
  eqiad.cqs-external.sparql-query
  eqiad.development.network.probe
  eqiad.eqiad
  eqiad.error
  eqiad.eventgate-analytics-external.error.validation
  eqiad.eventgate-analytics-external.test.event
  eqiad.eventgate-analytics-external.validation.error
  eqiad.eventgate-analytics.error.validation
  eqiad.eventgate-analytics.test.event
  eqiad.eventgate-logging-external.test.event
  eqiad.eventgate-main.error.validation
  eqiad.eventgate-main.test.even

Broker change summary:
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  Broker 1006 marked for removal
  Broker 1005 marked for removal
  Broker 1004 marked for removal
  Broker 1002 marked for removal
  New broker 1015
  New broker 1010
  New broker 1011
  New broker 1013
  New broker 1012
  -
  Replacing 6, added 5, missing 0, total count changed by -1

Action:
  Shrinking topic by 1 broker(s)
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.changeprop.retry.mediawiki.revision-create p0: [1001 1003 1004] -> [1014 1007 1012] replaced broker
  eqiad.changeprop.retry.mediawiki.revision-visibility-change p0: [1006 1001 1003] -> [1014 1010 1009] replaced broker
  eqiad.changeprop.retry.resource_change p0: [1005 1001 1003] -> [1011 1015 1007] replaced broker
  eqiad.cirrussearch.page-index-update p0: [1004 1001 1003] -> [1013 1008 1012] replaced broker
  eqiad.cirrussearch.update_pipeline.fetch_error.rc0 p0: [1002 1007 1009] -> [1007 1015 1009] replaced broker
  eqiad.cirrussearch.update_pipeline.fetch_failure p0: [1003 1004 1006] -> [1008 1015 1012] replaced broker
  eqiad.cirrussearch.update_pipeline.update.rc0 p0: [1014 1001 1003] -> [1011 1014 1007] replaced broker
  eqiad.cpjobqueue.error p0: [1002 1005 1003] -> [1012 1014 1008] replaced broker
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite p0: [1004 1009 1001] -> [1009 1013 1010] replaced broker
  eqiad.cqs-external.sparql-query p0: [1006 1003 1004] -> [1009 1015 1011] replaced broker
  eqiad.development.network.probe p0: [1008 1004 1002] -> [1008 1013 1010] replaced broker
  eqiad.eqiad p0: [1005 1008 1001] -> [1013 1008 1011] replaced broker
  eqiad.error p0: [1001 1005 1003] -> [1010 1014 1008] replaced broker
  eqiad.eventgate-analytics-external.error.validation p0: [1004 1003 1006] -> [1013 1008 1011] replaced broker
  eqiad.eventgate-analytics-external.test.event p0: [1003 1004 1006] -> [1007 1012 1013] replaced broker
  eqiad.eventgate-analytics-external.validation.error p0: [1004 1009 1001] -> [1015 1012 1009] replaced broker
  eqiad.eventgate-analytics.error.validation p0: [1003 1002 1005] -> [1015 1011 1009] replaced broker
  eqiad.eventgate-analytics.test.event p0: [1007 1009 1001] -> [1010 1007 1009] replaced broker
  eqiad.eventgate-logging-external.test.event p0: [1009 1001 1003] -> [1012 1007 1009] replaced broker
  eqiad.eventgate-main.error.validation p0: [1005 1002 1003] -> [1008 1013 1010] replaced broker
  eqiad.eventgate-main.test.even p0: [1006 1002 1005] -> [1007 1014 1010] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/8/5.20 -> 5/7/6.00
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 3, follower: 5, total: 8
  Broker 1009 - leader: 2, follower: 6, total: 8
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 2, follower: 4, total: 6
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 3, follower: 4, total: 7
  Broker 1014 - leader: 2, follower: 4, total: 6
  Broker 1015 - leader: 2, follower: 4, total: 6

WARN:
  [none]

New partition maps:
  e-batch4.json [combined map]
  eqiad.cirrussearch.update_pipeline.update.rc0.json
  eqiad.cpjobqueue.error.json
  eqiad.development.network.probe.json
  eqiad.error.json
  eqiad.eventgate-analytics-external.validation.error.json
  eqiad.eventgate-main.test.even.json
  eqiad.changeprop.retry.mediawiki.revision-create.json
  eqiad.changeprop.retry.mediawiki.revision-visibility-change.json
  eqiad.eventgate-analytics.error.validation.json
  eqiad.eventgate-analytics.test.event.json
  eqiad.changeprop.retry.resource_change.json
  eqiad.cqs-external.sparql-query.json
  eqiad.eventgate-analytics-external.error.validation.json
  eqiad.cirrussearch.page-index-update.json
  eqiad.eqiad.json
  eqiad.cpjobqueue.partitioned.mediawiki.job.cirrusSearchElasticaWrite.json
  eqiad.eventgate-analytics-external.test.event.json
  eqiad.eventgate-logging-external.test.event.json
  eqiad.eventgate-main.error.validation.json
  eqiad.cirrussearch.update_pipeline.fetch_error.rc0.json
  eqiad.cirrussearch.update_pipeline.fetch_failure.json

Next batch

topicmappr rebuild  --topics '^(eqiad.eventgate-main.test.event|eqiad.eventlogging_ContentTranslationAbuseFilter|eqiad.eventlogging_EditAttemptStep|eqiad.ios.edit_history_compare|eqiad.ios.login_action|eqiad.ios.navigation_events|eqiad.ios.notification_interaction|eqiad.ios.reading_lists|eqiad.ios.search|eqiad.ios.sessions|eqiad.ios.setting_action|eqiad.ios.talk_page_interaction|eqiad.ios.user_history|eqiad.ios.watchlists|eqiad.maps.tile_change|eqiad.maps.tiles_change|eqiad.mediainfo-streaming-updater.mutation|eqiad.mediainfo-streaming-updater.mutation-staging|eqiad.mediainfo-streaming-updater.reconcile|eqiad.mediawiki-page-move|eqiad.mediawiki-revision-create|eqiad.mediawiki-revision-delete)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch5

Topics:
  eqiad.eventgate-main.test.event
  eqiad.eventlogging_ContentTranslationAbuseFilter
  eqiad.eventlogging_EditAttemptStep
  eqiad.ios.edit_history_compare
  eqiad.ios.login_action
  eqiad.ios.navigation_events
  eqiad.ios.notification_interaction
  eqiad.ios.reading_lists
  eqiad.ios.search
  eqiad.ios.sessions
  eqiad.ios.setting_action
  eqiad.ios.talk_page_interaction
  eqiad.ios.user_history
  eqiad.ios.watchlists
  eqiad.maps.tile_change
  eqiad.maps.tiles_change
  eqiad.mediainfo-streaming-updater.mutation
  eqiad.mediainfo-streaming-updater.mutation-staging
  eqiad.mediainfo-streaming-updater.reconcile
  eqiad.mediawiki-page-move
  eqiad.mediawiki-revision-create
  eqiad.mediawiki-revision-delete

Broker change summary:
  Broker 1006 marked for removal
  Broker 1002 marked for removal
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  Broker 1005 marked for removal
  Broker 1004 marked for removal
  New broker 1013
  New broker 1010
  New broker 1011
  New broker 1012
  New broker 1015
  New broker 1014
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.eventgate-main.test.event p0: [1009 1001 1003] -> [1007 1015 1009] replaced broker
  eqiad.eventlogging_ContentTranslationAbuseFilter p0: [1005 1003 1006] -> [1012 1008 1015] replaced broker
  eqiad.eventlogging_EditAttemptStep p0: [1006 1001 1003] -> [1012 1014 1007] replaced broker
  eqiad.ios.edit_history_compare p0: [1008 1007 1001] -> [1011 1007 1008] replaced broker
  eqiad.ios.login_action p0: [1009 1001 1003] -> [1013 1010 1009] replaced broker
  eqiad.ios.navigation_events p0: [1003 1004 1006] -> [1010 1013 1008] replaced broker
  eqiad.ios.notification_interaction p0: [1009 1002 1005] -> [1015 1009 1011] replaced broker
  eqiad.ios.reading_lists p0: [1005 1003 1006] -> [1009 1015 1011] replaced broker
  eqiad.ios.search p0: [1009 1001 1003] -> [1013 1010 1009] replaced broker
  eqiad.ios.sessions p0: [1001 1003 1004] -> [1010 1007 1013] replaced broker
  eqiad.ios.setting_action p0: [1004 1009 1001] -> [1009 1014 1012] replaced broker
  eqiad.ios.talk_page_interaction p0: [1003 1004 1006] -> [1015 1008 1011] replaced broker
  eqiad.ios.user_history p0: [1006 1002 1005] -> [1008 1014 1012] replaced broker
  eqiad.ios.watchlists p0: [1003 1004 1006] -> [1013 1007 1010] replaced broker
  eqiad.maps.tile_change p0: [1001 1003 1004] -> [1009 1011 1015] replaced broker
  eqiad.maps.tiles_change p0: [1008 1007 1001] -> [1007 1008 1015] replaced broker
  eqiad.maps.tiles_change p1: [1009 1002 1005] -> [1014 1012 1009] replaced broker
  eqiad.maps.tiles_change p2: [1001 1005 1008] -> [1011 1008 1013] replaced broker
  eqiad.maps.tiles_change p3: [1003 1008 1007] -> [1008 1007 1014] replaced broker
  eqiad.maps.tiles_change p4: [1004 1009 1001] -> [1015 1010 1009] replaced broker
  eqiad.maps.tiles_change p5: [1006 1001 1003] -> [1007 1014 1012] replaced broker
  eqiad.mediainfo-streaming-updater.mutation p0: [1003 1004 1006] -> [1010 1007 1013] replaced broker
  eqiad.mediainfo-streaming-updater.mutation-staging p0: [1003 1008 1007] -> [1008 1007 1011] replaced broker
  eqiad.mediainfo-streaming-updater.reconcile p0: [1001 1005 1008] -> [1012 1008 1013] replaced broker
  eqiad.mediawiki-page-move p0: [1006 1002 1005] -> [1008 1014 1010] replaced broker
  eqiad.mediawiki-revision-create p0: [1007 1006 1002] -> [1014 1007 1012] replaced broker
  eqiad.mediawiki-revision-delete p0: [1009 1002 1005] -> [1014 1011 1009] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/7/5.33 -> 6/8/6.44
  -
  Broker 1007 - leader: 3, follower: 8, total: 11
  Broker 1008 - leader: 4, follower: 7, total: 11
  Broker 1009 - leader: 3, follower: 7, total: 10
  Broker 1010 - leader: 3, follower: 5, total: 8
  Broker 1011 - leader: 2, follower: 6, total: 8
  Broker 1012 - leader: 3, follower: 5, total: 8
  Broker 1013 - leader: 3, follower: 5, total: 8
  Broker 1014 - leader: 3, follower: 6, total: 9
  Broker 1015 - leader: 3, follower: 5, total: 8

WARN:
  [none]

New partition maps:
  e-batch5.json [combined map]
  eqiad.ios.reading_lists.json
  eqiad.ios.search.json
  eqiad.ios.talk_page_interaction.json
  eqiad.ios.user_history.json
  eqiad.mediainfo-streaming-updater.mutation-staging.json
  eqiad.mediawiki-revision-create.json
  eqiad.mediawiki-revision-delete.json
  eqiad.ios.login_action.json
  eqiad.ios.edit_history_compare.json
  eqiad.ios.navigation_events.json
  eqiad.ios.setting_action.json
  eqiad.ios.watchlists.json
  eqiad.mediainfo-streaming-updater.reconcile.json
  eqiad.eventlogging_EditAttemptStep.json
  eqiad.mediawiki-page-move.json
  eqiad.maps.tile_change.json
  eqiad.eventlogging_ContentTranslationAbuseFilter.json
  eqiad.ios.notification_interaction.json
  eqiad.ios.sessions.json
  eqiad.maps.tiles_change.json
  eqiad.mediainfo-streaming-updater.mutation.json
  eqiad.eventgate-main.test.event.json

Next reassignment will only span eqiad.mediawiki.api-request as it's a large topic.

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics 'eqiad.mediawiki.api-request' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership

Topics:
  eqiad.mediawiki.api-request

Broker change summary:
  Broker 1005 marked for removal
  Broker 1006 marked for removal
  Broker 1001 marked for removal
  Broker 1004 marked for removal
  Broker 1002 marked for removal
  Broker 1003 marked for removal
  New broker 1014
  New broker 1015
  New broker 1013
  New broker 1010
  New broker 1012
  New broker 1011
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.api-request p0: [1001 1008 1007] -> [1007 1008 1015] replaced broker
  eqiad.mediawiki.api-request p1: [1009 1001 1004] -> [1009 1011 1015] replaced broker
  eqiad.mediawiki.api-request p2: [1005 1002 1008] -> [1008 1013 1012] replaced broker
  eqiad.mediawiki.api-request p3: [1003 1007 1009] -> [1010 1007 1009] replaced broker
  eqiad.mediawiki.api-request p4: [1004 1009 1001] -> [1011 1013 1009] replaced broker
  eqiad.mediawiki.api-request p5: [1006 1002 1005] -> [1013 1008 1010] replaced broker
  eqiad.mediawiki.api-request p6: [1008 1003 1004] -> [1008 1014 1012] replaced broker
  eqiad.mediawiki.api-request p7: [1002 1004 1008] -> [1014 1011 1008] replaced broker
  eqiad.mediawiki.api-request p8: [1007 1008 1002] -> [1014 1008 1007] replaced broker
  eqiad.mediawiki.api-request p9: [1001 1006 1007] -> [1007 1015 1010] replaced broker
  eqiad.mediawiki.api-request p10: [1009 1007 1003] -> [1012 1009 1007] replaced broker
  eqiad.mediawiki.api-request p11: [1005 1009 1001] -> [1009 1012 1014] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/6/5.11 -> 5/7/5.56
  -
  Broker 1007 - leader: 2, follower: 3, total: 5
  Broker 1008 - leader: 2, follower: 4, total: 6
  Broker 1009 - leader: 2, follower: 3, total: 5
  Broker 1010 - leader: 1, follower: 2, total: 3
  Broker 1011 - leader: 1, follower: 2, total: 3
  Broker 1012 - leader: 1, follower: 3, total: 4
  Broker 1013 - leader: 1, follower: 2, total: 3
  Broker 1014 - leader: 2, follower: 2, total: 4
  Broker 1015 - leader: 0, follower: 3, total: 3

WARN:
  [none]

New partition maps:
  eqiad.mediawiki.api-request.json

Next batch

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.mediawiki.accountcreation_block|eqiad.mediawiki.api-action|eqiad.mediawiki.centralnotice.campaign-change|eqiad.mediawiki.centralnotice.campaign-create|eqiad.mediawiki.cirrussearch_request|eqiad.mediawiki.content_translation_event|eqiad.mediawiki.delete|eqiad.mediawiki.editattempt_block|eqiad.mediawiki.editgrowthconfig|eqiad.mediawiki.image_suggestions_feedback|eqiad.mediawiki.ipinfo_interaction|eqiad.mediawiki.job.ArticleChangedJob|eqiad.mediawiki.job.AssembleUploadChunks|eqiad.mediawiki.job.BounceHandlerJob|eqiad.mediawiki.job.BounceHandlerNotificationJob|eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob|eqiad.mediawiki.job.CentralAuthUnattachUserJob)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch6

Topics:
  eqiad.mediawiki.accountcreation_block
  eqiad.mediawiki.api-action
  eqiad.mediawiki.centralnotice.campaign-change
  eqiad.mediawiki.centralnotice.campaign-create
  eqiad.mediawiki.cirrussearch_request
  eqiad.mediawiki.content_translation_event
  eqiad.mediawiki.delete
  eqiad.mediawiki.editattempt_block
  eqiad.mediawiki.editgrowthconfig
  eqiad.mediawiki.image_suggestions_feedback
  eqiad.mediawiki.ipinfo_interaction
  eqiad.mediawiki.job.ArticleChangedJob
  eqiad.mediawiki.job.AssembleUploadChunks
  eqiad.mediawiki.job.BounceHandlerJob
  eqiad.mediawiki.job.BounceHandlerNotificationJob
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob
  eqiad.mediawiki.job.CentralAuthUnattachUserJob

Broker change summary:
  Broker 1003 marked for removal
  Broker 1006 marked for removal
  Broker 1004 marked for removal
  Broker 1001 marked for removal
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  New broker 1015
  New broker 1010
  New broker 1014
  New broker 1013
  New broker 1011
  New broker 1012
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.accountcreation_block p0: [1004 1009 1001] -> [1009 1015 1010] replaced broker
  eqiad.mediawiki.api-action p0: [1002 1003 1004] -> [1012 1014 1008] replaced broker
  eqiad.mediawiki.centralnotice.campaign-change p0: [1003 1004 1006] -> [1015 1010 1007] replaced broker
  eqiad.mediawiki.centralnotice.campaign-create p0: [1001 1005 1003] -> [1011 1013 1009] replaced broker
  eqiad.mediawiki.cirrussearch_request p0: [1005 1001 1003] -> [1010 1008 1013] replaced broker
  eqiad.mediawiki.content_translation_event p0: [1009 1001 1003] -> [1009 1015 1012] replaced broker
  eqiad.mediawiki.delete p0: [1003 1006 1002] -> [1014 1011 1007] replaced broker
  eqiad.mediawiki.editattempt_block p0: [1002 1005 1008] -> [1015 1007 1008] replaced broker
  eqiad.mediawiki.editgrowthconfig p0: [1008 1007 1001] -> [1007 1008 1012] replaced broker
  eqiad.mediawiki.image_suggestions_feedback p0: [1009 1003 1004] -> [1014 1010 1009] replaced broker
  eqiad.mediawiki.ipinfo_interaction p0: [1004 1006 1002] -> [1008 1011 1015] replaced broker
  eqiad.mediawiki.job.ArticleChangedJob p0: [1001 1005 1008] -> [1008 1013 1012] replaced broker
  eqiad.mediawiki.job.AssembleUploadChunks p0: [1002 1004 1006] -> [1010 1009 1014] replaced broker
  eqiad.mediawiki.job.BounceHandlerJob p0: [1004 1006 1002] -> [1013 1011 1009] replaced broker
  eqiad.mediawiki.job.BounceHandlerNotificationJob p0: [1006 1005 1001] -> [1012 1013 1007] replaced broker
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob p0: [1006 1004 1002] -> [1007 1011 1014] replaced broker
  eqiad.mediawiki.job.CentralAuthUnattachUserJob p0: [1003 1006 1002] -> [1011 1007 1014] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/7/4.67 -> 6/7/6.22
  -
  Broker 1007 - leader: 2, follower: 5, total: 7
  Broker 1008 - leader: 2, follower: 4, total: 6
  Broker 1009 - leader: 2, follower: 4, total: 6
  Broker 1010 - leader: 2, follower: 3, total: 5
  Broker 1011 - leader: 2, follower: 4, total: 6
  Broker 1012 - leader: 2, follower: 3, total: 5
  Broker 1013 - leader: 1, follower: 4, total: 5
  Broker 1014 - leader: 2, follower: 4, total: 6
  Broker 1015 - leader: 2, follower: 3, total: 5

WARN:
  [none]

New partition maps:
  e-batch6.json [combined map]
  eqiad.mediawiki.api-action.json
  eqiad.mediawiki.cirrussearch_request.json
  eqiad.mediawiki.delete.json
  eqiad.mediawiki.job.ArticleChangedJob.json
  eqiad.mediawiki.accountcreation_block.json
  eqiad.mediawiki.content_translation_event.json
  eqiad.mediawiki.editgrowthconfig.json
  eqiad.mediawiki.ipinfo_interaction.json
  eqiad.mediawiki.job.AssembleUploadChunks.json
  eqiad.mediawiki.job.BounceHandlerNotificationJob.json
  eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob.json
  eqiad.mediawiki.job.CentralAuthUnattachUserJob.json
  eqiad.mediawiki.centralnotice.campaign-change.json
  eqiad.mediawiki.centralnotice.campaign-create.json
  eqiad.mediawiki.editattempt_block.json
  eqiad.mediawiki.image_suggestions_feedback.json
  eqiad.mediawiki.job.BounceHandlerJob.json

Next batch. All topics are ~empty:

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.mediawiki.job.ChangeDeletionNotification|eqiad.mediawiki.job.ChangeNotification|eqiad.mediawiki.job.ChangeVisibilityNotification|eqiad.mediawiki.job.CleanTermsIfUnused|eqiad.mediawiki.job.CognateCacheUpdateJob|eqiad.mediawiki.job.CognateLocalJobSubmitJob|eqiad.mediawiki.job.DeleteTranslatableBundleJob|eqiad.mediawiki.job.DispatchChangeDeletionNotification|eqiad.mediawiki.job.DispatchChangeVisibilityNotification|eqiad.mediawiki.job.DispatchChanges|eqiad.mediawiki.job.EchoNotificationDeleteJob|eqiad.mediawiki.job.EchoNotificationJob|eqiad.mediawiki.job.EchoPushNotificationRequest|eqiad.mediawiki.job.EntityChangeNotification|eqiad.mediawiki.job.GlobalUserPageLocalJobSubmitJob|eqiad.mediawiki.job.ImageSuggestionsNotifications|eqiad.mediawiki.job.LocalGlobalUserPageCacheUpdateJob|eqiad.mediawiki.job.LocalPageMoveJob|eqiad.mediawiki.job.LocalRenameUserJob|eqiad.mediawiki.job.LoginNotifyChecks|eqiad.mediawiki.job.MassMessageJob)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch9

Topics:
  eqiad.mediawiki.job.ChangeDeletionNotification
  eqiad.mediawiki.job.ChangeNotification
  eqiad.mediawiki.job.ChangeVisibilityNotification
  eqiad.mediawiki.job.CleanTermsIfUnused
  eqiad.mediawiki.job.CognateCacheUpdateJob
  eqiad.mediawiki.job.CognateLocalJobSubmitJob
  eqiad.mediawiki.job.DeleteTranslatableBundleJob
  eqiad.mediawiki.job.DispatchChangeDeletionNotification
  eqiad.mediawiki.job.DispatchChangeVisibilityNotification
  eqiad.mediawiki.job.DispatchChanges
  eqiad.mediawiki.job.EchoNotificationDeleteJob
  eqiad.mediawiki.job.EchoNotificationJob
  eqiad.mediawiki.job.EchoPushNotificationRequest
  eqiad.mediawiki.job.EntityChangeNotification
  eqiad.mediawiki.job.GlobalUserPageLocalJobSubmitJob
  eqiad.mediawiki.job.ImageSuggestionsNotifications
  eqiad.mediawiki.job.LocalGlobalUserPageCacheUpdateJob
  eqiad.mediawiki.job.LocalPageMoveJob
  eqiad.mediawiki.job.LocalRenameUserJob
  eqiad.mediawiki.job.LoginNotifyChecks
  eqiad.mediawiki.job.MassMessageJob

Broker change summary:
  Broker 1003 marked for removal
  Broker 1005 marked for removal
  Broker 1006 marked for removal
  Broker 1002 marked for removal
  Broker 1004 marked for removal
  Broker 1001 marked for removal
  New broker 1015
  New broker 1007
  New broker 1014
  New broker 1011
  New broker 1010
  New broker 1013
  New broker 1012
  -
  Replacing 6, added 7, missing 0, total count changed by 1

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.ChangeDeletionNotification p0: [1002 1004 1008] -> [1007 1013 1008] replaced broker
  eqiad.mediawiki.job.ChangeNotification p0: [1001 1003 1004] -> [1015 1009 1011] replaced broker
  eqiad.mediawiki.job.ChangeVisibilityNotification p0: [1001 1005 1008] -> [1014 1012 1008] replaced broker
  eqiad.mediawiki.job.CleanTermsIfUnused p0: [1006 1003 1004] -> [1007 1010 1014] replaced broker
  eqiad.mediawiki.job.CognateCacheUpdateJob p0: [1005 1006 1002] -> [1014 1011 1007] replaced broker
  eqiad.mediawiki.job.CognateLocalJobSubmitJob p0: [1004 1006 1002] -> [1012 1013 1008] replaced broker
  eqiad.mediawiki.job.DeleteTranslatableBundleJob p0: [1001 1003 1004] -> [1010 1009 1014] replaced broker
  eqiad.mediawiki.job.DispatchChangeDeletionNotification p0: [1004 1009 1001] -> [1009 1007 1015] replaced broker
  eqiad.mediawiki.job.DispatchChangeVisibilityNotification p0: [1002 1005 1008] -> [1008 1010 1015] replaced broker
  eqiad.mediawiki.job.DispatchChanges p0: [1009 1001 1003] -> [1010 1009 1015] replaced broker
  eqiad.mediawiki.job.EchoNotificationDeleteJob p0: [1006 1001 1003] -> [1013 1012 1008] replaced broker
  eqiad.mediawiki.job.EchoNotificationJob p0: [1001 1006 1005] -> [1008 1013 1010] replaced broker
  eqiad.mediawiki.job.EchoPushNotificationRequest p0: [1009 1001 1003] -> [1011 1009 1014] replaced broker
  eqiad.mediawiki.job.EntityChangeNotification p0: [1003 1004 1006] -> [1013 1011 1009] replaced broker
  eqiad.mediawiki.job.GlobalUserPageLocalJobSubmitJob p0: [1005 1001 1003] -> [1007 1011 1013] replaced broker
  eqiad.mediawiki.job.ImageSuggestionsNotifications p0: [1004 1006 1002] -> [1014 1012 1007] replaced broker
  eqiad.mediawiki.job.LocalGlobalUserPageCacheUpdateJob p0: [1003 1002 1005] -> [1015 1007 1010] replaced broker
  eqiad.mediawiki.job.LocalPageMoveJob p0: [1004 1003 1006] -> [1012 1007 1015] replaced broker
  eqiad.mediawiki.job.LocalRenameUserJob p0: [1002 1004 1006] -> [1015 1012 1008] replaced broker
  eqiad.mediawiki.job.LoginNotifyChecks p0: [1004 1006 1002] -> [1009 1010 1013] replaced broker
  eqiad.mediawiki.job.MassMessageJob p0: [1005 1006 1002] -> [1009 1011 1014] replaced broker

Broker distribution:
  degree [min/max/avg]: 3/6/5.00 -> 5/8/6.00
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 2, follower: 5, total: 7
  Broker 1009 - leader: 3, follower: 5, total: 8
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 1, follower: 5, total: 6
  Broker 1012 - leader: 2, follower: 4, total: 6
  Broker 1013 - leader: 2, follower: 5, total: 7
  Broker 1014 - leader: 3, follower: 4, total: 7
  Broker 1015 - leader: 3, follower: 4, total: 7

WARN:
  [none]

New partition maps:
  e-batch9.json [combined map]
  eqiad.mediawiki.job.CleanTermsIfUnused.json
  eqiad.mediawiki.job.DispatchChangeVisibilityNotification.json
  eqiad.mediawiki.job.EchoPushNotificationRequest.json
  eqiad.mediawiki.job.GlobalUserPageLocalJobSubmitJob.json
  eqiad.mediawiki.job.EchoNotificationJob.json
  eqiad.mediawiki.job.EntityChangeNotification.json
  eqiad.mediawiki.job.LocalGlobalUserPageCacheUpdateJob.json
  eqiad.mediawiki.job.LocalRenameUserJob.json
  eqiad.mediawiki.job.ChangeDeletionNotification.json
  eqiad.mediawiki.job.ChangeNotification.json
  eqiad.mediawiki.job.CognateLocalJobSubmitJob.json
  eqiad.mediawiki.job.DeleteTranslatableBundleJob.json
  eqiad.mediawiki.job.MassMessageJob.json
  eqiad.mediawiki.job.ChangeVisibilityNotification.json
  eqiad.mediawiki.job.DispatchChangeDeletionNotification.json
  eqiad.mediawiki.job.DispatchChanges.json
  eqiad.mediawiki.job.ImageSuggestionsNotifications.json
  eqiad.mediawiki.job.CognateCacheUpdateJob.json
  eqiad.mediawiki.job.EchoNotificationDeleteJob.json
  eqiad.mediawiki.job.LocalPageMoveJob.json
  eqiad.mediawiki.job.LoginNotifyChecks.json

Next batch. All topics are ~ empty

brouberol@kafka-jumbo1010:~/topicmappr$ ./generate_topicmappr_command.py eqiad.mediawiki.job.MassMessageJob 20 e-batch10
topicmappr rebuild  --topics '^(eqiad.mediawiki.job.MassMessageServerSideJob|eqiad.mediawiki.job.MassMessageSubmitJob|eqiad.mediawiki.job.MessageGroupStatesUpdaterJob|eqiad.mediawiki.job.MessageGroupStatsRebuildJob|eqiad.mediawiki.job.MessageIndexRebuildJob|eqiad.mediawiki.job.MessageUpdateJob|eqiad.mediawiki.job.MoveTranslatableBundleJob|eqiad.mediawiki.job.ORESFetchScoreJob|eqiad.mediawiki.job.PublishStashedFile|eqiad.mediawiki.job.PurgeEntityData|eqiad.mediawiki.job.RecordLintJob|eqiad.mediawiki.job.RefreshLinksJob|eqiad.mediawiki.job.RenderTranslationPageJob|eqiad.mediawiki.job.TTMServerMessageUpdateJob|eqiad.mediawiki.job.ThumbnailRender|eqiad.mediawiki.job.TranslatableBundleDeleteJob|eqiad.mediawiki.job.TranslatableBundleMoveJob|eqiad.mediawiki.job.TranslatablePageMoveJob|eqiad.mediawiki.job.TranslateDeleteJob|eqiad.mediawiki.job.TranslateRenderJob|eqiad.mediawiki.job.TranslationNotificationEmailJob)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch10
brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.mediawiki.job.MassMessageServerSideJob|eqiad.mediawiki.job.MassMessageSubmitJob|eqiad.mediawiki.job.MessageGroupStatesUpdaterJob|eqiad.mediawiki.job.MessageGroupStatsRebuildJob|eqiad.mediawiki.job.MessageIndexRebuildJob|eqiad.mediawiki.job.MessageUpdateJob|eqiad.mediawiki.job.MoveTranslatableBundleJob|eqiad.mediawiki.job.ORESFetchScoreJob|eqiad.mediawiki.job.PublishStashedFile|eqiad.mediawiki.job.PurgeEntityData|eqiad.mediawiki.job.RecordLintJob|eqiad.mediawiki.job.RefreshLinksJob|eqiad.mediawiki.job.RenderTranslationPageJob|eqiad.mediawiki.job.TTMServerMessageUpdateJob|eqiad.mediawiki.job.ThumbnailRender|eqiad.mediawiki.job.TranslatableBundleDeleteJob|eqiad.mediawiki.job.TranslatableBundleMoveJob|eqiad.mediawiki.job.TranslatablePageMoveJob|eqiad.mediawiki.job.TranslateDeleteJob|eqiad.mediawiki.job.TranslateRenderJob|eqiad.mediawiki.job.TranslationNotificationEmailJob)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch10

Topics:
  eqiad.mediawiki.job.MassMessageServerSideJob
  eqiad.mediawiki.job.MassMessageSubmitJob
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob
  eqiad.mediawiki.job.MessageGroupStatsRebuildJob
  eqiad.mediawiki.job.MessageIndexRebuildJob
  eqiad.mediawiki.job.MessageUpdateJob
  eqiad.mediawiki.job.MoveTranslatableBundleJob
  eqiad.mediawiki.job.ORESFetchScoreJob
  eqiad.mediawiki.job.PublishStashedFile
  eqiad.mediawiki.job.PurgeEntityData
  eqiad.mediawiki.job.RecordLintJob
  eqiad.mediawiki.job.RefreshLinksJob
  eqiad.mediawiki.job.RenderTranslationPageJob
  eqiad.mediawiki.job.TTMServerMessageUpdateJob
  eqiad.mediawiki.job.ThumbnailRender
  eqiad.mediawiki.job.TranslatableBundleDeleteJob
  eqiad.mediawiki.job.TranslatableBundleMoveJob
  eqiad.mediawiki.job.TranslatablePageMoveJob
  eqiad.mediawiki.job.TranslateDeleteJob
  eqiad.mediawiki.job.TranslateRenderJob
  eqiad.mediawiki.job.TranslationNotificationEmailJob

Broker change summary:
  Broker 1001 marked for removal
  Broker 1004 marked for removal
  Broker 1006 marked for removal
  Broker 1005 marked for removal
  Broker 1003 marked for removal
  Broker 1002 marked for removal
  New broker 1015
  New broker 1010
  New broker 1014
  New broker 1011
  New broker 1012
  New broker 1013
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.MassMessageServerSideJob p0: [1001 1006 1005] -> [1010 1015 1008] replaced broker
  eqiad.mediawiki.job.MassMessageSubmitJob p0: [1003 1004 1006] -> [1012 1014 1008] replaced broker
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob p0: [1006 1005 1001] -> [1007 1013 1010] replaced broker
  eqiad.mediawiki.job.MessageGroupStatsRebuildJob p0: [1002 1004 1006] -> [1009 1013 1011] replaced broker
  eqiad.mediawiki.job.MessageIndexRebuildJob p0: [1004 1003 1006] -> [1009 1013 1012] replaced broker
  eqiad.mediawiki.job.MessageUpdateJob p0: [1006 1001 1003] -> [1014 1008 1012] replaced broker
  eqiad.mediawiki.job.MoveTranslatableBundleJob p0: [1002 1003 1004] -> [1011 1015 1007] replaced broker
  eqiad.mediawiki.job.ORESFetchScoreJob p0: [1007 1003 1009] -> [1007 1009 1010] replaced broker
  eqiad.mediawiki.job.PublishStashedFile p0: [1003 1006 1002] -> [1009 1012 1015] replaced broker
  eqiad.mediawiki.job.PurgeEntityData p0: [1009 1002 1005] -> [1009 1011 1014] replaced broker
  eqiad.mediawiki.job.RecordLintJob p0: [1001 1006 1007] -> [1007 1010 1008] replaced broker
  eqiad.mediawiki.job.RefreshLinksJob p0: [1009 1002 1005] -> [1012 1013 1009] replaced broker
  eqiad.mediawiki.job.RenderTranslationPageJob p0: [1003 1008 1007] -> [1008 1011 1007] replaced broker
  eqiad.mediawiki.job.TTMServerMessageUpdateJob p0: [1004 1006 1002] -> [1013 1011 1007] replaced broker
  eqiad.mediawiki.job.ThumbnailRender p0: [1003 1001 1004] -> [1011 1007 1014] replaced broker
  eqiad.mediawiki.job.TranslatableBundleDeleteJob p0: [1007 1006 1002] -> [1010 1015 1007] replaced broker
  eqiad.mediawiki.job.TranslatableBundleMoveJob p0: [1005 1008 1001] -> [1008 1014 1011] replaced broker
  eqiad.mediawiki.job.TranslatablePageMoveJob p0: [1001 1006 1005] -> [1015 1010 1009] replaced broker
  eqiad.mediawiki.job.TranslateDeleteJob p0: [1006 1005 1001] -> [1013 1012 1008] replaced broker
  eqiad.mediawiki.job.TranslateRenderJob p0: [1006 1003 1004] -> [1008 1012 1015] replaced broker
  eqiad.mediawiki.job.TranslationNotificationEmailJob p0: [1004 1003 1006] -> [1010 1014 1009] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/7/5.33 -> 5/7/6.22
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 3, follower: 5, total: 8
  Broker 1009 - leader: 4, follower: 4, total: 8
  Broker 1010 - leader: 3, follower: 4, total: 7
  Broker 1011 - leader: 2, follower: 5, total: 7
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 2, follower: 4, total: 6
  Broker 1014 - leader: 1, follower: 5, total: 6
  Broker 1015 - leader: 1, follower: 5, total: 6

WARN:
  [none]

New partition maps:
  e-batch10.json [combined map]
  eqiad.mediawiki.job.MessageUpdateJob.json
  eqiad.mediawiki.job.PublishStashedFile.json
  eqiad.mediawiki.job.RefreshLinksJob.json
  eqiad.mediawiki.job.TranslatableBundleDeleteJob.json
  eqiad.mediawiki.job.TranslatableBundleMoveJob.json
  eqiad.mediawiki.job.MassMessageServerSideJob.json
  eqiad.mediawiki.job.MessageIndexRebuildJob.json
  eqiad.mediawiki.job.ORESFetchScoreJob.json
  eqiad.mediawiki.job.TTMServerMessageUpdateJob.json
  eqiad.mediawiki.job.TranslateRenderJob.json
  eqiad.mediawiki.job.MassMessageSubmitJob.json
  eqiad.mediawiki.job.MoveTranslatableBundleJob.json
  eqiad.mediawiki.job.ThumbnailRender.json
  eqiad.mediawiki.job.TranslationNotificationEmailJob.json
  eqiad.mediawiki.job.MessageGroupStatesUpdaterJob.json
  eqiad.mediawiki.job.PurgeEntityData.json
  eqiad.mediawiki.job.RenderTranslationPageJob.json
  eqiad.mediawiki.job.TranslatablePageMoveJob.json
  eqiad.mediawiki.job.TranslateDeleteJob.json
  eqiad.mediawiki.job.MessageGroupStatsRebuildJob.json
  eqiad.mediawiki.job.RecordLintJob.json

Next batch. All topics are ~ empty

brouberol@kafka-jumbo1010:~/topicmappr$ ./generate_topicmappr_command.py eqiad.mediawiki.job.TranslationNotificationEmailJob 20 e-batch11
topicmappr rebuild  --topics '^(eqiad.mediawiki.job.TranslationNotificationJob|eqiad.mediawiki.job.TranslationNotificationSubmitJob|eqiad.mediawiki.job.TranslationNotificationsEmailJob|eqiad.mediawiki.job.TranslationNotificationsSubmitJob|eqiad.mediawiki.job.TranslationNotificationsTalkPageJob|eqiad.mediawiki.job.TranslationsUpdateJob|eqiad.mediawiki.job.UpdateMessageBundle|eqiad.mediawiki.job.UpdateRepoOnDelete|eqiad.mediawiki.job.UpdateRepoOnMove|eqiad.mediawiki.job.UpdateTranslatablePageJob|eqiad.mediawiki.job.UpdateTranslatorActivity|eqiad.mediawiki.job.activityUpdateJob|eqiad.mediawiki.job.cacheTesterResults|eqiad.mediawiki.job.categoryMembershipChange|eqiad.mediawiki.job.cdnPurge|eqiad.mediawiki.job.changeNotificationJo|eqiad.mediawiki.job.changeNotificationJob|eqiad.mediawiki.job.checkuserPruneCheckUserDataJob|eqiad.mediawiki.job.cirrusSearchCheckerJob|eqiad.mediawiki.job.cirrusSearchDeleteArchive|eqiad.mediawiki.job.cirrusSearchDeletePages)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch11
brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^(eqiad.mediawiki.job.TranslationNotificationJob|eqiad.mediawiki.job.TranslationNotificationSubmitJob|eqiad.mediawiki.job.TranslationNotificationsEmailJob|eqiad.mediawiki.job.TranslationNotificationsSubmitJob|eqiad.mediawiki.job.TranslationNotificationsTalkPageJob|eqiad.mediawiki.job.TranslationsUpdateJob|eqiad.mediawiki.job.UpdateMessageBundle|eqiad.mediawiki.job.UpdateRepoOnDelete|eqiad.mediawiki.job.UpdateRepoOnMove|eqiad.mediawiki.job.UpdateTranslatablePageJob|eqiad.mediawiki.job.UpdateTranslatorActivity|eqiad.mediawiki.job.activityUpdateJob|eqiad.mediawiki.job.cacheTesterResults|eqiad.mediawiki.job.categoryMembershipChange|eqiad.mediawiki.job.cdnPurge|eqiad.mediawiki.job.changeNotificationJo|eqiad.mediawiki.job.changeNotificationJob|eqiad.mediawiki.job.checkuserPruneCheckUserDataJob|eqiad.mediawiki.job.cirrusSearchCheckerJob|eqiad.mediawiki.job.cirrusSearchDeleteArchive|eqiad.mediawiki.job.cirrusSearchDeletePages)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch11

Topics:
  eqiad.mediawiki.job.TranslationNotificationJob
  eqiad.mediawiki.job.TranslationNotificationSubmitJob
  eqiad.mediawiki.job.TranslationNotificationsEmailJob
  eqiad.mediawiki.job.TranslationNotificationsSubmitJob
  eqiad.mediawiki.job.TranslationNotificationsTalkPageJob
  eqiad.mediawiki.job.TranslationsUpdateJob
  eqiad.mediawiki.job.UpdateMessageBundle
  eqiad.mediawiki.job.UpdateRepoOnDelete
  eqiad.mediawiki.job.UpdateRepoOnMove
  eqiad.mediawiki.job.UpdateTranslatablePageJob
  eqiad.mediawiki.job.UpdateTranslatorActivity
  eqiad.mediawiki.job.activityUpdateJob
  eqiad.mediawiki.job.cacheTesterResults
  eqiad.mediawiki.job.categoryMembershipChange
  eqiad.mediawiki.job.cdnPurge
  eqiad.mediawiki.job.changeNotificationJo
  eqiad.mediawiki.job.changeNotificationJob
  eqiad.mediawiki.job.checkuserPruneCheckUserDataJob
  eqiad.mediawiki.job.cirrusSearchCheckerJob
  eqiad.mediawiki.job.cirrusSearchDeleteArchive
  eqiad.mediawiki.job.cirrusSearchDeletePages

Broker change summary:
  Broker 1001 marked for removal
  Broker 1006 marked for removal
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  Broker 1005 marked for removal
  Broker 1002 marked for removal
  New broker 1014
  New broker 1015
  New broker 1013
  New broker 1007
  New broker 1012
  New broker 1010
  New broker 1011
  -
  Replacing 6, added 7, missing 0, total count changed by 1

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.TranslationNotificationJob p0: [1001 1003 1004] -> [1015 1011 1007] replaced broker
  eqiad.mediawiki.job.TranslationNotificationSubmitJob p0: [1001 1005 1003] -> [1010 1008 1015] replaced broker
  eqiad.mediawiki.job.TranslationNotificationsEmailJob p0: [1003 1005 1001] -> [1013 1012 1009] replaced broker
  eqiad.mediawiki.job.TranslationNotificationsSubmitJob p0: [1002 1005 1003] -> [1010 1007 1014] replaced broker
  eqiad.mediawiki.job.TranslationNotificationsTalkPageJob p0: [1006 1002 1005] -> [1011 1009 1015] replaced broker
  eqiad.mediawiki.job.TranslationsUpdateJob p0: [1005 1006 1002] -> [1012 1013 1008] replaced broker
  eqiad.mediawiki.job.UpdateMessageBundle p0: [1001 1003 1004] -> [1014 1007 1012] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnDelete p0: [1004 1002 1003] -> [1013 1008 1010] replaced broker
  eqiad.mediawiki.job.UpdateRepoOnMove p0: [1002 1004 1006] -> [1009 1007 1015] replaced broker
  eqiad.mediawiki.job.UpdateTranslatablePageJob p0: [1002 1003 1004] -> [1007 1015 1010] replaced broker
  eqiad.mediawiki.job.UpdateTranslatorActivity p0: [1001 1006 1005] -> [1014 1009 1011] replaced broker
  eqiad.mediawiki.job.activityUpdateJob p0: [1004 1003 1006] -> [1008 1011 1013] replaced broker
  eqiad.mediawiki.job.cacheTesterResults p0: [1009 1001 1003] -> [1009 1011 1013] replaced broker
  eqiad.mediawiki.job.categoryMembershipChange p0: [1009 1002 1004] -> [1010 1009 1014] replaced broker
  eqiad.mediawiki.job.cdnPurge p0: [1005 1008 1002] -> [1015 1008 1010] replaced broker
  eqiad.mediawiki.job.changeNotificationJo p0: [1006 1002 1005] -> [1007 1012 1014] replaced broker
  eqiad.mediawiki.job.changeNotificationJob p0: [1002 1003 1004] -> [1011 1009 1014] replaced broker
  eqiad.mediawiki.job.checkuserPruneCheckUserDataJob p0: [1006 1002 1005] -> [1008 1014 1012] replaced broker
  eqiad.mediawiki.job.cirrusSearchCheckerJob p0: [1006 1003 1004] -> [1009 1011 1013] replaced broker
  eqiad.mediawiki.job.cirrusSearchDeleteArchive p0: [1001 1004 1006] -> [1013 1007 1012] replaced broker
  eqiad.mediawiki.job.cirrusSearchDeletePages p0: [1006 1001 1003] -> [1008 1015 1010] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/6/4.75 -> 5/7/6.00
  -
  Broker 1007 - leader: 2, follower: 5, total: 7
  Broker 1008 - leader: 3, follower: 4, total: 7
  Broker 1009 - leader: 3, follower: 5, total: 8
  Broker 1010 - leader: 3, follower: 4, total: 7
  Broker 1011 - leader: 2, follower: 5, total: 7
  Broker 1012 - leader: 1, follower: 5, total: 6
  Broker 1013 - leader: 3, follower: 4, total: 7
  Broker 1014 - leader: 2, follower: 5, total: 7
  Broker 1015 - leader: 2, follower: 5, total: 7

WARN:
  [none]

New partition maps:
  e-batch11.json [combined map]
  eqiad.mediawiki.job.UpdateMessageBundle.json
  eqiad.mediawiki.job.UpdateRepoOnMove.json
  eqiad.mediawiki.job.UpdateTranslatablePageJob.json
  eqiad.mediawiki.job.activityUpdateJob.json
  eqiad.mediawiki.job.cacheTesterResults.json
  eqiad.mediawiki.job.cdnPurge.json
  eqiad.mediawiki.job.changeNotificationJob.json
  eqiad.mediawiki.job.TranslationsUpdateJob.json
  eqiad.mediawiki.job.cirrusSearchDeleteArchive.json
  eqiad.mediawiki.job.cirrusSearchCheckerJob.json
  eqiad.mediawiki.job.TranslationNotificationsEmailJob.json
  eqiad.mediawiki.job.TranslationNotificationsSubmitJob.json
  eqiad.mediawiki.job.UpdateRepoOnDelete.json
  eqiad.mediawiki.job.categoryMembershipChange.json
  eqiad.mediawiki.job.cirrusSearchDeletePages.json
  eqiad.mediawiki.job.TranslationNotificationJob.json
  eqiad.mediawiki.job.UpdateTranslatorActivity.json
  eqiad.mediawiki.job.checkuserPruneCheckUserDataJob.json
  eqiad.mediawiki.job.TranslationNotificationSubmitJob.json
  eqiad.mediawiki.job.changeNotificationJo.json
  eqiad.mediawiki.job.TranslationNotificationsTalkPageJob.json

Next batch. All topics are ~empty:

topicmappr rebuild  --topics '^(eqiad.mediawiki.job.cirrusSearchElasticaWrite|eqiad.mediawiki.job.cirrusSearchIncomingLinkCount|eqiad.mediawiki.job.cirrusSearchLinksUpdate|eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized|eqiad.mediawiki.job.cirrusSearchMassIndex|eqiad.mediawiki.job.cirrusSearchOtherIndex|eqiad.mediawiki.job.cleanTer|eqiad.mediawiki.job.clearUserWatchlist|eqiad.mediawiki.job.compileArticleMetadata|eqiad.mediawiki.job.constraintsRunCheck|eqiad.mediawiki.job.constraintsTableUpdate|eqiad.mediawiki.job.crosswikiSuppressUser|eqiad.mediawiki.job.deleteLinks|eqiad.mediawiki.job.deletePage|eqiad.mediawiki.job.enotifNotify|eqiad.mediawiki.job.enqueue|eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations|eqiad.mediawiki.job.fixDoubleRedirect|eqiad.mediawiki.job.flaggedrevs_CacheUpdate|eqiad.mediawiki.job.globalUsageCachePurge|eqiad.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch12

Topics:
  eqiad.mediawiki.job.cirrusSearchElasticaWrite
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount
  eqiad.mediawiki.job.cirrusSearchLinksUpdate
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized
  eqiad.mediawiki.job.cirrusSearchMassIndex
  eqiad.mediawiki.job.cirrusSearchOtherIndex
  eqiad.mediawiki.job.cleanTer
  eqiad.mediawiki.job.clearUserWatchlist
  eqiad.mediawiki.job.compileArticleMetadata
  eqiad.mediawiki.job.constraintsRunCheck
  eqiad.mediawiki.job.constraintsTableUpdate
  eqiad.mediawiki.job.crosswikiSuppressUser
  eqiad.mediawiki.job.deleteLinks
  eqiad.mediawiki.job.deletePage
  eqiad.mediawiki.job.enotifNotify
  eqiad.mediawiki.job.enqueue
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations
  eqiad.mediawiki.job.fixDoubleRedirect
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate
  eqiad.mediawiki.job.globalUsageCachePurge
  eqiad.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob

Broker change summary:
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  Broker 1006 marked for removal
  Broker 1004 marked for removal
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  New broker 1015
  New broker 1010
  New broker 1011
  New broker 1012
  New broker 1014
  New broker 1008
  New broker 1013
  New broker 1009
  -
  Replacing 6, added 8, missing 0, total count changed by 2

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p0: [1004 1001 1003] -> [1007 1009 1013] replaced broker
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p1: [1002 1003 1004] -> [1015 1008 1012] replaced broker
  eqiad.mediawiki.job.cirrusSearchElasticaWrite p2: [1005 1006 1002] -> [1008 1015 1010] replaced broker
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount p0: [1003 1006 1002] -> [1009 1011 1014] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdate p0: [1005 1001 1003] -> [1007 1013 1009] replaced broker
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized p0: [1002 1003 1004] -> [1014 1010 1008] replaced broker
  eqiad.mediawiki.job.cirrusSearchMassIndex p0: [1005 1006 1002] -> [1013 1012 1009] replaced broker
  eqiad.mediawiki.job.cirrusSearchOtherIndex p0: [1004 1003 1006] -> [1010 1008 1014] replaced broker
  eqiad.mediawiki.job.cleanTer p0: [1002 1003 1004] -> [1010 1008 1014] replaced broker
  eqiad.mediawiki.job.clearUserWatchlist p0: [1002 1005 1003] -> [1012 1009 1013] replaced broker
  eqiad.mediawiki.job.compileArticleMetadata p0: [1002 1005 1003] -> [1015 1007 1010] replaced broker
  eqiad.mediawiki.job.constraintsRunCheck p0: [1005 1006 1002] -> [1011 1007 1015] replaced broker
  eqiad.mediawiki.job.constraintsTableUpdate p0: [1002 1006 1005] -> [1009 1013 1011] replaced broker
  eqiad.mediawiki.job.crosswikiSuppressUser p0: [1006 1002 1005] -> [1012 1014 1009] replaced broker
  eqiad.mediawiki.job.deleteLinks p0: [1006 1005 1001] -> [1014 1010 1009] replaced broker
  eqiad.mediawiki.job.deletePage p0: [1001 1003 1004] -> [1012 1007 1015] replaced broker
  eqiad.mediawiki.job.enotifNotify p0: [1002 1003 1004] -> [1007 1011 1014] replaced broker
  eqiad.mediawiki.job.enqueue p0: [1003 1004 1006] -> [1010 1008 1013] replaced broker
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations p0: [1002 1003 1004] -> [1013 1011 1008] replaced broker
  eqiad.mediawiki.job.fixDoubleRedirect p0: [1007 1006 1002] -> [1013 1007 1012] replaced broker
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate p0: [1003 1005 1001] -> [1015 1007 1012] replaced broker
  eqiad.mediawiki.job.globalUsageCachePurge p0: [1003 1006 1002] -> [1008 1011 1015] replaced broker
  eqiad.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob p0: [1003 1001 1004] -> [1008 1011 1015] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/6/4.29 -> 5/7/6.00
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 3, follower: 6, total: 9
  Broker 1009 - leader: 2, follower: 6, total: 8
  Broker 1010 - leader: 3, follower: 4, total: 7
  Broker 1011 - leader: 1, follower: 6, total: 7
  Broker 1012 - leader: 3, follower: 4, total: 7
  Broker 1013 - leader: 3, follower: 5, total: 8
  Broker 1014 - leader: 2, follower: 5, total: 7
  Broker 1015 - leader: 3, follower: 5, total: 8

WARN:
  [none]

New partition maps:
  e-batch12.json [combined map]
  eqiad.mediawiki.job.deletePage.json
  eqiad.mediawiki.job.fetchGoogleCloudVisionAnnotations.json
  eqiad.mediawiki.job.flaggedrevs_CacheUpdate.json
  eqiad.mediawiki.job.gwtoolsetGWTFileBackendCleanupJob.json
  eqiad.mediawiki.job.cirrusSearchMassIndex.json
  eqiad.mediawiki.job.cleanTer.json
  eqiad.mediawiki.job.crosswikiSuppressUser.json
  eqiad.mediawiki.job.deleteLinks.json
  eqiad.mediawiki.job.enotifNotify.json
  eqiad.mediawiki.job.cirrusSearchElasticaWrite.json
  eqiad.mediawiki.job.constraintsRunCheck.json
  eqiad.mediawiki.job.constraintsTableUpdate.json
  eqiad.mediawiki.job.compileArticleMetadata.json
  eqiad.mediawiki.job.fixDoubleRedirect.json
  eqiad.mediawiki.job.globalUsageCachePurge.json
  eqiad.mediawiki.job.cirrusSearchLinksUpdatePrioritized.json
  eqiad.mediawiki.job.cirrusSearchOtherIndex.json
  eqiad.mediawiki.job.clearUserWatchlist.json
  eqiad.mediawiki.job.cirrusSearchIncomingLinkCount.json
  eqiad.mediawiki.job.cirrusSearchLinksUpdate.json
  eqiad.mediawiki.job.enqueue.json

Next batch. All topics are ~empty except for eqiad.mediawiki.job.parsoidCachePrewarm that has a single partition of ~16GB.

topicmappr rebuild  --topics '^(eqiad.mediawiki.job.gwtoolsetUploadMediafileJob|eqiad.mediawiki.job.gwtoolsetUploadMetadataJob|eqiad.mediawiki.job.htmlCacheUpdate|eqiad.mediawiki.job.ipinfoLogIPInfoAccess|eqiad.mediawiki.job.menteeOverviewUpdateDataForMentor|eqiad.mediawiki.job.newUserMessageJob|eqiad.mediawiki.job.newcomerTasksCacheRefreshJob|eqiad.mediawiki.job.notificationGetStartedJob|eqiad.mediawiki.job.notificationKeepGoingJob|eqiad.mediawiki.job.null|eqiad.mediawiki.job.nullJob|eqiad.mediawiki.job.parsoidCachePrewarm|eqiad.mediawiki.job.processMediaModeration|eqiad.mediawiki.job.reassignMenteesJob|eqiad.mediawiki.job.recentChangesUpdate|eqiad.mediawiki.job.refreshLinks|eqiad.mediawiki.job.refreshLinksPrioritized|eqiad.mediawiki.job.refreshUserImpactJob|eqiad.mediawiki.job.renameUser|eqiad.mediawiki.job.revertedTagUpdate|eqiad.mediawiki.job.saveMentor)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch13

Topics:
  eqiad.mediawiki.job.gwtoolsetUploadMediafileJob
  eqiad.mediawiki.job.gwtoolsetUploadMetadataJob
  eqiad.mediawiki.job.htmlCacheUpdate
  eqiad.mediawiki.job.ipinfoLogIPInfoAccess
  eqiad.mediawiki.job.menteeOverviewUpdateDataForMentor
  eqiad.mediawiki.job.newUserMessageJob
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob
  eqiad.mediawiki.job.notificationGetStartedJob
  eqiad.mediawiki.job.notificationKeepGoingJob
  eqiad.mediawiki.job.null
  eqiad.mediawiki.job.nullJob
  eqiad.mediawiki.job.parsoidCachePrewarm
  eqiad.mediawiki.job.processMediaModeration
  eqiad.mediawiki.job.reassignMenteesJob
  eqiad.mediawiki.job.recentChangesUpdate
  eqiad.mediawiki.job.refreshLinks
  eqiad.mediawiki.job.refreshLinksPrioritized
  eqiad.mediawiki.job.refreshUserImpactJob
  eqiad.mediawiki.job.renameUser
  eqiad.mediawiki.job.revertedTagUpdate
  eqiad.mediawiki.job.saveMentor

Broker change summary:
  Broker 1006 marked for removal
  Broker 1001 marked for removal
  Broker 1005 marked for removal
  Broker 1002 marked for removal
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  New broker 1014
  New broker 1011
  New broker 1013
  New broker 1010
  New broker 1012
  New broker 1015
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.gwtoolsetUploadMediafileJob p0: [1003 1004 1006] -> [1007 1015 1011] replaced broker
  eqiad.mediawiki.job.gwtoolsetUploadMetadataJob p0: [1001 1005 1003] -> [1012 1009 1014] replaced broker
  eqiad.mediawiki.job.htmlCacheUpdate p0: [1003 1005 1006] -> [1010 1013 1007] replaced broker
  eqiad.mediawiki.job.ipinfoLogIPInfoAccess p0: [1001 1003 1004] -> [1013 1008 1011] replaced broker
  eqiad.mediawiki.job.menteeOverviewUpdateDataForMentor p0: [1009 1002 1005] -> [1015 1009 1010] replaced broker
  eqiad.mediawiki.job.newUserMessageJob p0: [1002 1006 1005] -> [1009 1011 1013] replaced broker
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob p0: [1005 1003 1006] -> [1007 1012 1014] replaced broker
  eqiad.mediawiki.job.notificationGetStartedJob p0: [1008 1007 1001] -> [1012 1008 1007] replaced broker
  eqiad.mediawiki.job.notificationKeepGoingJob p0: [1002 1005 1008] -> [1008 1010 1015] replaced broker
  eqiad.mediawiki.job.null p0: [1009 1001 1003] -> [1011 1014 1009] replaced broker
  eqiad.mediawiki.job.nullJob p0: [1007 1009 1001] -> [1011 1009 1007] replaced broker
  eqiad.mediawiki.job.parsoidCachePrewarm p0: [1007 1009 1001] -> [1009 1013 1007] replaced broker
  eqiad.mediawiki.job.processMediaModeration p0: [1001 1005 1008] -> [1008 1012 1015] replaced broker
  eqiad.mediawiki.job.reassignMenteesJob p0: [1005 1008 1001] -> [1014 1008 1010] replaced broker
  eqiad.mediawiki.job.recentChangesUpdate p0: [1004 1003 1006] -> [1014 1011 1009] replaced broker
  eqiad.mediawiki.job.refreshLinks p0: [1006 1007 1003] -> [1013 1007 1012] replaced broker
  eqiad.mediawiki.job.refreshLinksPrioritized p0: [1001 1006 1005] -> [1010 1014 1009] replaced broker
  eqiad.mediawiki.job.refreshUserImpactJob p0: [1005 1008 1001] -> [1015 1008 1010] replaced broker
  eqiad.mediawiki.job.renameUser p0: [1006 1005 1001] -> [1008 1015 1012] replaced broker
  eqiad.mediawiki.job.revertedTagUpdate p0: [1008 1007 1001] -> [1007 1008 1012] replaced broker
  eqiad.mediawiki.job.saveMentor p0: [1003 1004 1006] -> [1008 1013 1010] replaced broker

Broker distribution:
  degree [min/max/avg]: 3/7/5.11 -> 6/8/6.44
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 4, follower: 5, total: 9
  Broker 1009 - leader: 2, follower: 6, total: 8
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 2, follower: 4, total: 6
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 2, follower: 4, total: 6
  Broker 1014 - leader: 2, follower: 4, total: 6
  Broker 1015 - leader: 2, follower: 4, total: 6

WARN:
  [none]

New partition maps:
  e-batch13.json [combined map]
  eqiad.mediawiki.job.notificationGetStartedJob.json
  eqiad.mediawiki.job.nullJob.json
  eqiad.mediawiki.job.newUserMessageJob.json
  eqiad.mediawiki.job.null.json
  eqiad.mediawiki.job.refreshLinks.json
  eqiad.mediawiki.job.saveMentor.json
  eqiad.mediawiki.job.menteeOverviewUpdateDataForMentor.json
  eqiad.mediawiki.job.htmlCacheUpdate.json
  eqiad.mediawiki.job.newcomerTasksCacheRefreshJob.json
  eqiad.mediawiki.job.notificationKeepGoingJob.json
  eqiad.mediawiki.job.processMediaModeration.json
  eqiad.mediawiki.job.reassignMenteesJob.json
  eqiad.mediawiki.job.gwtoolsetUploadMetadataJob.json
  eqiad.mediawiki.job.ipinfoLogIPInfoAccess.json
  eqiad.mediawiki.job.parsoidCachePrewarm.json
  eqiad.mediawiki.job.recentChangesUpdate.json
  eqiad.mediawiki.job.refreshLinksPrioritized.json
  eqiad.mediawiki.job.refreshUserImpactJob.json
  eqiad.mediawiki.job.renameUser.json
  eqiad.mediawiki.job.revertedTagUpdate.json
  eqiad.mediawiki.job.gwtoolsetUploadMediafileJob.json

Next batch.

topicmappr rebuild  --topics '^(eqiad.mediawiki.job.securePollArchiveElection|eqiad.mediawiki.job.securePollLogAdminAction|eqiad.mediawiki.job.securePollTallyElection|eqiad.mediawiki.job.securePollUnarchiveElection|eqiad.mediawiki.job.sendMail|eqiad.mediawiki.job.setUserMentorDatabaseJob|eqiad.mediawiki.job.synchroniseThreadArticleData|eqiad.mediawiki.job.translationNotificationJob|eqiad.mediawiki.job.tutorialVisit|eqiad.mediawiki.job.updateBetaFeaturesUserCounts|eqiad.mediawiki.job.updateImplementations|eqiad.mediawiki.job.userGroupExpiry|eqiad.mediawiki.job.userOptionsUpdate|eqiad.mediawiki.job.watchlistExpiry|eqiad.mediawiki.job.webVideoTranscode|eqiad.mediawiki.job.webVideoTranscodePrioritized|eqiad.mediawiki.job.wikibase-InjectRCRecords|eqiad.mediawiki.job.wikibase-addUsagesForPage|eqiad.mediawiki.jobrefreshLinks|eqiad.mediawiki.maps_interaction|eqiad.mediawiki.mediasearch_interaction)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch14

Topics:
  eqiad.mediawiki.job.securePollArchiveElection
  eqiad.mediawiki.job.securePollLogAdminAction
  eqiad.mediawiki.job.securePollTallyElection
  eqiad.mediawiki.job.securePollUnarchiveElection
  eqiad.mediawiki.job.sendMail
  eqiad.mediawiki.job.setUserMentorDatabaseJob
  eqiad.mediawiki.job.synchroniseThreadArticleData
  eqiad.mediawiki.job.translationNotificationJob
  eqiad.mediawiki.job.tutorialVisit
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts
  eqiad.mediawiki.job.updateImplementations
  eqiad.mediawiki.job.userGroupExpiry
  eqiad.mediawiki.job.userOptionsUpdate
  eqiad.mediawiki.job.watchlistExpiry
  eqiad.mediawiki.job.webVideoTranscode
  eqiad.mediawiki.job.webVideoTranscodePrioritized
  eqiad.mediawiki.job.wikibase-InjectRCRecords
  eqiad.mediawiki.job.wikibase-addUsagesForPage
  eqiad.mediawiki.jobrefreshLinks
  eqiad.mediawiki.maps_interaction
  eqiad.mediawiki.mediasearch_interaction

Broker change summary:
  Broker 1003 marked for removal
  Broker 1005 marked for removal
  Broker 1004 marked for removal
  Broker 1001 marked for removal
  Broker 1002 marked for removal
  Broker 1006 marked for removal
  New broker 1011
  New broker 1010
  New broker 1012
  New broker 1013
  New broker 1014
  New broker 1015
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.job.securePollArchiveElection p0: [1002 1005 1008] -> [1008 1007 1015] replaced broker
  eqiad.mediawiki.job.securePollLogAdminAction p0: [1009 1002 1005] -> [1010 1014 1009] replaced broker
  eqiad.mediawiki.job.securePollTallyElection p0: [1004 1009 1001] -> [1015 1009 1012] replaced broker
  eqiad.mediawiki.job.securePollUnarchiveElection p0: [1005 1008 1001] -> [1010 1007 1008] replaced broker
  eqiad.mediawiki.job.sendMail p0: [1006 1001 1003] -> [1013 1011 1009] replaced broker
  eqiad.mediawiki.job.setUserMentorDatabaseJob p0: [1007 1009 1001] -> [1009 1014 1007] replaced broker
  eqiad.mediawiki.job.synchroniseThreadArticleData p0: [1003 1004 1006] -> [1008 1011 1013] replaced broker
  eqiad.mediawiki.job.translationNotificationJob p0: [1004 1002 1003] -> [1014 1011 1008] replaced broker
  eqiad.mediawiki.job.tutorialVisit p0: [1005 1001 1003] -> [1012 1007 1015] replaced broker
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts p0: [1004 1001 1003] -> [1009 1012 1015] replaced broker
  eqiad.mediawiki.job.updateImplementations p0: [1002 1003 1004] -> [1008 1010 1014] replaced broker
  eqiad.mediawiki.job.userGroupExpiry p0: [1002 1005 1003] -> [1007 1011 1013] replaced broker
  eqiad.mediawiki.job.userOptionsUpdate p0: [1004 1003 1006] -> [1013 1008 1010] replaced broker
  eqiad.mediawiki.job.watchlistExpiry p0: [1009 1002 1005] -> [1009 1011 1013] replaced broker
  eqiad.mediawiki.job.webVideoTranscode p0: [1006 1005 1001] -> [1009 1012 1014] replaced broker
  eqiad.mediawiki.job.webVideoTranscodePrioritized p0: [1002 1003 1004] -> [1007 1010 1015] replaced broker
  eqiad.mediawiki.job.wikibase-InjectRCRecords p0: [1008 1002 1007] -> [1015 1007 1008] replaced broker
  eqiad.mediawiki.job.wikibase-addUsagesForPage p0: [1002 1005 1009] -> [1015 1007 1009] replaced broker
  eqiad.mediawiki.jobrefreshLinks p0: [1002 1005 1003] -> [1014 1008 1012] replaced broker
  eqiad.mediawiki.maps_interaction p0: [1001 1003 1004] -> [1010 1008 1014] replaced broker
  eqiad.mediawiki.mediasearch_interaction p0: [1003 1004 1006] -> [1011 1007 1013] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/7/5.11 -> 5/8/6.00
  -
  Broker 1007 - leader: 2, follower: 7, total: 9
  Broker 1008 - leader: 3, follower: 6, total: 9
  Broker 1009 - leader: 4, follower: 4, total: 8
  Broker 1010 - leader: 3, follower: 3, total: 6
  Broker 1011 - leader: 1, follower: 5, total: 6
  Broker 1012 - leader: 1, follower: 4, total: 5
  Broker 1013 - leader: 2, follower: 4, total: 6
  Broker 1014 - leader: 2, follower: 5, total: 7
  Broker 1015 - leader: 3, follower: 4, total: 7

WARN:
  [none]

New partition maps:
  e-batch14.json [combined map]
  eqiad.mediawiki.job.securePollLogAdminAction.json
  eqiad.mediawiki.job.tutorialVisit.json
  eqiad.mediawiki.job.updateBetaFeaturesUserCounts.json
  eqiad.mediawiki.job.updateImplementations.json
  eqiad.mediawiki.jobrefreshLinks.json
  eqiad.mediawiki.job.securePollTallyElection.json
  eqiad.mediawiki.job.securePollUnarchiveElection.json
  eqiad.mediawiki.job.userOptionsUpdate.json
  eqiad.mediawiki.job.watchlistExpiry.json
  eqiad.mediawiki.job.translationNotificationJob.json
  eqiad.mediawiki.job.userGroupExpiry.json
  eqiad.mediawiki.job.webVideoTranscode.json
  eqiad.mediawiki.job.webVideoTranscodePrioritized.json
  eqiad.mediawiki.job.securePollArchiveElection.json
  eqiad.mediawiki.job.sendMail.json
  eqiad.mediawiki.job.setUserMentorDatabaseJob.json
  eqiad.mediawiki.job.synchroniseThreadArticleData.json
  eqiad.mediawiki.job.wikibase-InjectRCRecords.json
  eqiad.mediawiki.job.wikibase-addUsagesForPage.json
  eqiad.mediawiki.maps_interaction.json
  eqiad.mediawiki.mediasearch_interaction.json

Next batch

topicmappr rebuild  --topics '^(eqiad.mediawiki.pref_diff|eqiad.mediawiki.reading_depth|eqiad.mediawiki.recentchange|eqiad.mediawiki.resource-change|eqiad.mediawiki.revision-crate|eqiad.mediawiki.revision-create|eqiad.mediawiki.revision-create-database|eqiad.mediawiki.revision-delete|eqiad.mediawiki.revision-recommendation-create|eqiad.mediawiki.revision-score|eqiad.mediawiki.revision-score-test|eqiad.mediawiki.revision-tags-change|eqiad.mediawiki.revision-visibility-change|eqiad.mediawiki.revision_create|eqiad.mediawiki.revision_score|eqiad.mediawiki.revision_score_articlequality|eqiad.mediawiki.revision_score_articletopic|eqiad.mediawiki.revision_score_damaging|eqiad.mediawiki.revision_score_draftquality|eqiad.mediawiki.revision_score_drafttopic|eqiad.mediawiki.revision_score_goodfaith)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch16

Topics:
  eqiad.mediawiki-revision-create
  eqiad.mediawiki-revision-delete
  eqiad.mediawiki.pref_diff
  eqiad.mediawiki.reading_depth
  eqiad.mediawiki.recentchange
  eqiad.mediawiki.resource-change
  eqiad.mediawiki.revision-crate
  eqiad.mediawiki.revision-create
  eqiad.mediawiki.revision-create-database
  eqiad.mediawiki.revision-delete
  eqiad.mediawiki.revision-recommendation-create
  eqiad.mediawiki.revision-score
  eqiad.mediawiki.revision-score-test
  eqiad.mediawiki.revision-tags-change
  eqiad.mediawiki.revision-visibility-change
  eqiad.mediawiki.revision_create
  eqiad.mediawiki.revision_score
  eqiad.mediawiki.revision_score_articlequality
  eqiad.mediawiki.revision_score_articletopic
  eqiad.mediawiki.revision_score_damaging
  eqiad.mediawiki.revision_score_draftquality
  eqiad.mediawiki.revision_score_drafttopic
  eqiad.mediawiki.revision_score_goodfaith

Broker change summary:
  Broker 1006 marked for removal
  Broker 1001 marked for removal
  Broker 1005 marked for removal
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  Broker 1002 marked for removal
  New broker 1015
  New broker 1013
  New broker 1010
  -
  Replacing 6, added 3, missing 0, total count changed by -3

Action:
  Shrinking topic by 3 broker(s)
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki-revision-create p0: [1014 1007 1012] -> [1012 1007 1014] preferred leader
  eqiad.mediawiki-revision-delete p0: [1014 1011 1009] -> [1014 1011 1009] no-op
  eqiad.mediawiki.pref_diff p0: [1001 1003 1004] -> [1007 1014 1010] replaced broker
  eqiad.mediawiki.reading_depth p0: [1005 1008 1001] -> [1010 1013 1008] replaced broker
  eqiad.mediawiki.recentchange p0: [1009 1001 1005] -> [1014 1009 1010] replaced broker
  eqiad.mediawiki.resource-change p0: [1005 1001 1003] -> [1015 1011 1008] replaced broker
  eqiad.mediawiki.revision-crate p0: [1004 1002 1003] -> [1009 1012 1013] replaced broker
  eqiad.mediawiki.revision-create p0: [1005 1009 1003] -> [1014 1009 1010] replaced broker
  eqiad.mediawiki.revision-create-database p0: [1004 1009 1001] -> [1011 1015 1009] replaced broker
  eqiad.mediawiki.revision-delete p0: [1005 1006 1002] -> [1008 1013 1011] replaced broker
  eqiad.mediawiki.revision-recommendation-create p0: [1001 1003 1004] -> [1015 1008 1010] replaced broker
  eqiad.mediawiki.revision-score p0: [1003 1006 1004] -> [1012 1013 1007] replaced broker
  eqiad.mediawiki.revision-score-test p0: [1009 1001 1003] -> [1009 1014 1011] replaced broker
  eqiad.mediawiki.revision-tags-change p0: [1004 1008 1001] -> [1008 1012 1015] replaced broker
  eqiad.mediawiki.revision-visibility-change p0: [1003 1004 1006] -> [1013 1012 1007] replaced broker
  eqiad.mediawiki.revision_create p0: [1005 1002 1003] -> [1007 1010 1014] replaced broker
  eqiad.mediawiki.revision_score p0: [1003 1004 1006] -> [1011 1007 1015] replaced broker
  eqiad.mediawiki.revision_score_articlequality p0: [1003 1008 1007] -> [1007 1008 1013] replaced broker
  eqiad.mediawiki.revision_score_articletopic p0: [1001 1003 1004] -> [1010 1009 1014] replaced broker
  eqiad.mediawiki.revision_score_damaging p0: [1002 1003 1004] -> [1010 1007 1015] replaced broker
  eqiad.mediawiki.revision_score_draftquality p0: [1001 1005 1008] -> [1008 1015 1011] replaced broker
  eqiad.mediawiki.revision_score_drafttopic p0: [1003 1004 1006] -> [1013 1012 1008] replaced broker
  eqiad.mediawiki.revision_score_goodfaith p0: [1009 1001 1003] -> [1013 1012 1009] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/8/4.67 -> 5/7/6.00
  -
  Broker 1007 - leader: 3, follower: 5, total: 8
  Broker 1008 - leader: 3, follower: 5, total: 8
  Broker 1009 - leader: 2, follower: 6, total: 8
  Broker 1010 - leader: 3, follower: 5, total: 8
  Broker 1011 - leader: 2, follower: 5, total: 7
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 3, follower: 5, total: 8
  Broker 1014 - leader: 3, follower: 5, total: 8
  Broker 1015 - leader: 2, follower: 5, total: 7

WARN:
  [none]

New partition maps:
  e-batch16.json [combined map]
  eqiad.mediawiki.revision-score.json
  eqiad.mediawiki.revision-visibility-change.json
  eqiad.mediawiki.revision_create.json
  eqiad.mediawiki.revision_score.json
  eqiad.mediawiki.revision_score_articlequality.json
  eqiad.mediawiki.revision_score_draftquality.json
  eqiad.mediawiki.reading_depth.json
  eqiad.mediawiki.revision-create.json
  eqiad.mediawiki.revision_score_drafttopic.json
  eqiad.mediawiki.revision_score_goodfaith.json
  eqiad.mediawiki.resource-change.json
  eqiad.mediawiki.revision-delete.json
  eqiad.mediawiki.revision-crate.json
  eqiad.mediawiki.revision-recommendation-create.json
  eqiad.mediawiki.revision-score-test.json
  eqiad.mediawiki.revision-tags-change.json
  eqiad.mediawiki.revision_score_articletopic.json
  eqiad.mediawiki.pref_diff.json
  eqiad.mediawiki.recentchange.json
  eqiad.mediawiki.revision_score_damaging.json
  eqiad.mediawiki-revision-create.json
  eqiad.mediawiki.revision-create-database.json

Next batch

topicmappr rebuild  --topics '^(eqiad.mediawiki.revision_score_reverted|eqiad.mediawiki.revisioni-score|eqiad.mediawiki.searchpreview|eqiad.mediawiki.skin_diff|eqiad.mediawiki.special_diff_interactions|eqiad.mediawiki.structured_task.article.image_suggestion_interaction|eqiad.mediawiki.structured_task.article.link_suggestion_interaction|eqiad.mediawiki.talk_page_edit|eqiad.mediawiki.test|eqiad.mediawiki.test-event|eqiad.mediawiki.user-blocks-change|eqiad.mediawiki.visual_editor_feature_use|eqiad.mediawiki.web_ab_test_enrollment|eqiad.mediawiki.web_ui.interactions|eqiad.mediawiki.web_ui_scroll|eqiad.mediawiki.welcomesurvey.interaction|eqiad.mediawiki.wikistories_consumption_event|eqiad.mediawiki.wikistories_contribution_event|eqiad.mediawiki.wikistories_creation_event|eqiad.mediawiki_page_content_change_enrichment.error|eqiad.mw_page_content_change_enrich.error)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch17

Topics:
  eqiad.mediawiki.revision_score_reverted
  eqiad.mediawiki.revisioni-score
  eqiad.mediawiki.searchpreview
  eqiad.mediawiki.skin_diff
  eqiad.mediawiki.special_diff_interactions
  eqiad.mediawiki.structured_task.article.image_suggestion_interaction
  eqiad.mediawiki.structured_task.article.link_suggestion_interaction
  eqiad.mediawiki.talk_page_edit
  eqiad.mediawiki.test
  eqiad.mediawiki.test-event
  eqiad.mediawiki.user-blocks-change
  eqiad.mediawiki.visual_editor_feature_use
  eqiad.mediawiki.web_ab_test_enrollment
  eqiad.mediawiki.web_ui.interactions
  eqiad.mediawiki.web_ui_scroll
  eqiad.mediawiki.welcomesurvey.interaction
  eqiad.mediawiki.wikistories_consumption_event
  eqiad.mediawiki.wikistories_contribution_event
  eqiad.mediawiki.wikistories_creation_event
  eqiad.mediawiki_page_content_change_enrichment.error
  eqiad.mw_page_content_change_enrich.error

Broker change summary:
  Broker 1004 marked for removal
  Broker 1005 marked for removal
  Broker 1002 marked for removal
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  Broker 1006 marked for removal
  New broker 1015
  New broker 1012
  New broker 1014
  New broker 1011
  New broker 1010
  New broker 1013
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mediawiki.revision_score_reverted p0: [1006 1001 1003] -> [1015 1012 1009] replaced broker
  eqiad.mediawiki.revisioni-score p0: [1004 1001 1003] -> [1012 1014 1009] replaced broker
  eqiad.mediawiki.searchpreview p0: [1001 1005 1008] -> [1013 1008 1010] replaced broker
  eqiad.mediawiki.skin_diff p0: [1002 1005 1008] -> [1008 1011 1013] replaced broker
  eqiad.mediawiki.special_diff_interactions p0: [1005 1008 1001] -> [1010 1008 1013] replaced broker
  eqiad.mediawiki.structured_task.article.image_suggestion_interaction p0: [1008 1007 1001] -> [1007 1008 1015] replaced broker
  eqiad.mediawiki.structured_task.article.link_suggestion_interaction p0: [1003 1008 1007] -> [1007 1014 1008] replaced broker
  eqiad.mediawiki.talk_page_edit p0: [1006 1002 1005] -> [1007 1010 1015] replaced broker
  eqiad.mediawiki.test p0: [1006 1001 1003] -> [1009 1014 1012] replaced broker
  eqiad.mediawiki.test-event p0: [1006 1002 1005] -> [1007 1010 1015] replaced broker
  eqiad.mediawiki.user-blocks-change p0: [1003 1005 1001] -> [1009 1011 1013] replaced broker
  eqiad.mediawiki.visual_editor_feature_use p0: [1009 1001 1003] -> [1011 1014 1009] replaced broker
  eqiad.mediawiki.web_ab_test_enrollment p0: [1006 1001 1003] -> [1011 1013 1007] replaced broker
  eqiad.mediawiki.web_ui.interactions p0: [1005 1003 1006] -> [1014 1012 1009] replaced broker
  eqiad.mediawiki.web_ui_scroll p0: [1008 1007 1001] -> [1015 1008 1007] replaced broker
  eqiad.mediawiki.welcomesurvey.interaction p0: [1002 1003 1004] -> [1015 1010 1007] replaced broker
  eqiad.mediawiki.wikistories_consumption_event p0: [1005 1008 1001] -> [1013 1008 1012] replaced broker
  eqiad.mediawiki.wikistories_contribution_event p0: [1005 1003 1006] -> [1010 1014 1008] replaced broker
  eqiad.mediawiki.wikistories_creation_event p0: [1008 1007 1001] -> [1008 1011 1007] replaced broker
  eqiad.mediawiki_page_content_change_enrichment.error p0: [1007 1009 1001] -> [1012 1009 1007] replaced broker
  eqiad.mw_page_content_change_enrich.error p0: [1004 1009 1001] -> [1011 1009 1015] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/8/5.11 -> 5/8/6.22
  -
  Broker 1007 - leader: 4, follower: 5, total: 9
  Broker 1008 - leader: 2, follower: 7, total: 9
  Broker 1009 - leader: 2, follower: 6, total: 8
  Broker 1010 - leader: 2, follower: 4, total: 6
  Broker 1011 - leader: 3, follower: 3, total: 6
  Broker 1012 - leader: 2, follower: 4, total: 6
  Broker 1013 - leader: 2, follower: 4, total: 6
  Broker 1014 - leader: 1, follower: 5, total: 6
  Broker 1015 - leader: 3, follower: 4, total: 7

WARN:
  [none]

New partition maps:
  e-batch17.json [combined map]
  eqiad.mediawiki.web_ab_test_enrollment.json
  eqiad.mediawiki.wikistories_consumption_event.json
  eqiad.mediawiki.wikistories_contribution_event.json
  eqiad.mediawiki.skin_diff.json
  eqiad.mediawiki.special_diff_interactions.json
  eqiad.mediawiki.structured_task.article.link_suggestion_interaction.json
  eqiad.mediawiki.test-event.json
  eqiad.mediawiki.user-blocks-change.json
  eqiad.mediawiki_page_content_change_enrichment.error.json
  eqiad.mediawiki.revision_score_reverted.json
  eqiad.mediawiki.structured_task.article.image_suggestion_interaction.json
  eqiad.mediawiki.visual_editor_feature_use.json
  eqiad.mw_page_content_change_enrich.error.json
  eqiad.mediawiki.wikistories_creation_event.json
  eqiad.mediawiki.searchpreview.json
  eqiad.mediawiki.talk_page_edit.json
  eqiad.mediawiki.test.json
  eqiad.mediawiki.web_ui_scroll.json
  eqiad.mediawiki.welcomesurvey.interaction.json
  eqiad.mediawiki.revisioni-score.json
  eqiad.mediawiki.web_ui.interactions.json

Next batch. Most of the topics are ~empty.

topicmappr rebuild  --topics '^(eqiad.mwcli.command_execute|eqiad.null|eqiad.page_content_change|eqiad.page_content_change.v1|eqiad.rc0.mediawiki.page_change|eqiad.rc0.mediawiki.page_change_content|eqiad.rc0.mediawiki.page_content_change|eqiad.rc01mediawiki.page_change|eqiad.rc1.enrichment.mediawiki_page_content_change.error|eqiad.rc1.mediawiki.page_change|eqiad.rc1.mediawiki.page_content_change|eqiad.rc1.mediawiki.page_content_changel|eqiad.rdf-streaming-updater.failed-operation|eqiad.rdf-streaming-updater.failed-ops|eqiad.rdf-streaming-updater.fetch-failure|eqiad.rdf-streaming-updater.lapsed-action|eqiad.rdf-streaming-updater.lapsed_action|eqiad.rdf-streaming-updater.mutation|eqiad.rdf-streaming-updater.mutation-staging|eqiad.rdf-streaming-updater.reconcile|eqiad.rdf-streaming-updater.state-inconsistency)$' --brokers 1007,1008,1009,1010,1011,1012,1013,1014,1015 --skip-no-ops  --optimize-leadership --out-file e-batch18

Topics:
  eqiad.mwcli.command_execute
  eqiad.null
  eqiad.page_content_change
  eqiad.page_content_change.v1
  eqiad.rc0.mediawiki.page_change
  eqiad.rc0.mediawiki.page_change_content
  eqiad.rc0.mediawiki.page_content_change
  eqiad.rc01mediawiki.page_change
  eqiad.rc1.enrichment.mediawiki_page_content_change.error
  eqiad.rc1.mediawiki.page_change
  eqiad.rc1.mediawiki.page_content_change
  eqiad.rc1.mediawiki.page_content_changel
  eqiad.rdf-streaming-updater.failed-operation
  eqiad.rdf-streaming-updater.failed-ops
  eqiad.rdf-streaming-updater.fetch-failure
  eqiad.rdf-streaming-updater.lapsed-action
  eqiad.rdf-streaming-updater.lapsed_action
  eqiad.rdf-streaming-updater.mutation
  eqiad.rdf-streaming-updater.mutation-staging
  eqiad.rdf-streaming-updater.reconcile
  eqiad.rdf-streaming-updater.state-inconsistency

Broker change summary:
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  Broker 1006 marked for removal
  Broker 1004 marked for removal
  New broker 1012
  New broker 1014
  New broker 1011
  New broker 1013
  -
  Replacing 6, added 4, missing 0, total count changed by -2

Action:
  Shrinking topic by 2 broker(s)
  Optimizing leader/follower ratios

Partition map changes:
  eqiad.mwcli.command_execute p0: [1003 1008 1007] -> [1007 1008 1011] replaced broker
  eqiad.null p0: [1009 1002 1005] -> [1009 1014 1010] replaced broker
  eqiad.page_content_change p0: [1003 1004 1006] -> [1012 1013 1009] replaced broker
  eqiad.page_content_change.v1 p0: [1008 1007 1001] -> [1014 1008 1007] replaced broker
  eqiad.rc0.mediawiki.page_change p0: [1002 1003 1004] -> [1013 1009 1010] replaced broker
  eqiad.rc0.mediawiki.page_change_content p0: [1005 1008 1001] -> [1012 1014 1008] replaced broker
  eqiad.rc0.mediawiki.page_content_change p0: [1004 1009 1001] -> [1009 1010 1015] replaced broker
  eqiad.rc01mediawiki.page_change p0: [1004 1006 1002] -> [1007 1014 1012] replaced broker
  eqiad.rc1.enrichment.mediawiki_page_content_change.error p0: [1007 1009 1001] -> [1009 1015 1007] replaced broker
  eqiad.rc1.mediawiki.page_change p0: [1008 1007 1001] -> [1008 1011 1007] replaced broker
  eqiad.rc1.mediawiki.page_content_change p0: [1002 1005 1008] -> [1015 1010 1008] replaced broker
  eqiad.rc1.mediawiki.page_content_changel p0: [1004 1009 1001] -> [1011 1009 1014] replaced broker
  eqiad.rdf-streaming-updater.failed-operation p0: [1008 1004 1002] -> [1008 1012 1013] replaced broker
  eqiad.rdf-streaming-updater.failed-ops p0: [1004 1006 1002] -> [1011 1009 1013] replaced broker
  eqiad.rdf-streaming-updater.fetch-failure p0: [1006 1002 1005] -> [1014 1012 1009] replaced broker
  eqiad.rdf-streaming-updater.lapsed-action p0: [1007 1009 1001] -> [1007 1009 1015] replaced broker
  eqiad.rdf-streaming-updater.lapsed_action p0: [1010 1015 1001] -> [1010 1015 1007] replaced broker
  eqiad.rdf-streaming-updater.mutation p0: [1002 1005 1008] -> [1008 1015 1011] replaced broker
  eqiad.rdf-streaming-updater.mutation-staging p0: [1003 1008 1007] -> [1012 1007 1008] replaced broker
  eqiad.rdf-streaming-updater.reconcile p0: [1003 1008 1007] -> [1008 1010 1007] replaced broker
  eqiad.rdf-streaming-updater.state-inconsistency p0: [1005 1003 1006] -> [1011 1013 1008] replaced broker

Broker distribution:
  degree [min/max/avg]: 2/7/4.91 -> 5/7/6.00
  -
  Broker 1007 - leader: 3, follower: 6, total: 9
  Broker 1008 - leader: 4, follower: 6, total: 10
  Broker 1009 - leader: 3, follower: 6, total: 9
  Broker 1010 - leader: 1, follower: 5, total: 6
  Broker 1011 - leader: 3, follower: 3, total: 6
  Broker 1012 - leader: 3, follower: 3, total: 6
  Broker 1013 - leader: 1, follower: 4, total: 5
  Broker 1014 - leader: 2, follower: 4, total: 6
  Broker 1015 - leader: 1, follower: 5, total: 6

WARN:
  [none]

New partition maps:
  e-batch18.json [combined map]
  eqiad.page_content_change.v1.json
  eqiad.rdf-streaming-updater.failed-ops.json
  eqiad.rdf-streaming-updater.lapsed-action.json
  eqiad.rdf-streaming-updater.lapsed_action.json
  eqiad.page_content_change.json
  eqiad.rc01mediawiki.page_change.json
  eqiad.rc1.mediawiki.page_content_change.json
  eqiad.rc1.mediawiki.page_content_changel.json
  eqiad.rdf-streaming-updater.mutation.json
  eqiad.rdf-streaming-updater.state-inconsistency.json
  eqiad.rc0.mediawiki.page_change_content.json
  eqiad.rdf-streaming-updater.failed-operation.json
  eqiad.rdf-streaming-updater.fetch-failure.json
  eqiad.rdf-streaming-updater.mutation-staging.json
  eqiad.rc0.mediawiki.page_change.json
  eqiad.null.json
  eqiad.rc0.mediawiki.page_content_change.json
  eqiad.rc1.enrichment.mediawiki_page_content_change.error.json
  eqiad.rc1.mediawiki.page_change.json
  eqiad.rdf-streaming-updater.reconcile.json
  eqiad.mwcli.command_execute.json

After many (many) batches, we're down to the last batch before the webrequest* topics:

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild  --topics '^[s-v].*$' --brokers '1007,1008,1009,1010,1011,1012,1013,1014,1015' --skip-no-ops  --optimize-leadership --out-file s-to-v

Topics:
  staging.api-gateway.request
  staging.changeprop-staging.error
  staging.changeprop-staging.retry.mediawiki.job.ThumbnailRender
  staging.changeprop-staging.retry.mediawiki.revision-create
  staging.cqs-external.sparql-query
  staging.eventgate-analytics-external.error.validation
  staging.eventgate-analytics-external.test.event
  staging.eventgate-analytics.error.validation
  staging.eventgate-analytics.test.event
  staging.eventgate-main.test.event
  staging.liftwing.test-events
  staging.liftwing.test-outlink-events
  staging.mediawiki.api-request
  staging.mediawiki.cirrussearch-request
  staging.mediawiki.job.ThumbnailRender
  staging.mediawiki.job.renameUser
  staging.mediawiki.page-create
  staging.mediawiki.page-delete
  staging.mediawiki.page-links-change
  staging.mediawiki.page-move
  staging.mediawiki.page-properties-change
  staging.mediawiki.page-restrictions-change
  staging.mediawiki.page-undelete
  staging.mediawiki.recentchange
  staging.mediawiki.revision-create
  staging.mediawiki.revision-score
  staging.mediawiki.revision-tags-change
  staging.mediawiki.revision-visibility-change
  staging.mediawiki.user-blocks-change
  staging.rdf-streaming-updater.fetch-failure
  staging.rdf-streaming-updater.mutation
  staging.rdf-streaming-updater.state-inconsistency
  staging.resource-purge
  staging.resource_change
  staging.swift.example_container.upload-complete
  staging.test.event
  staging.test_purge
  staging.wdqs-external.sparql-query
  staging.wdqs-internal.sparql-query
  statsv
  tchin_test0
  temp_NavigationTiming
  temp_NavigationTiming_replay
  test
  test.instrumentation
  test_elukey
  test_flink0_page_content_change
  test_wdqs_streaming_updater
  test_wdqs_streaming_updater_maryum
  translationrecommendationuirequests
  virtualpageview

Broker change summary:
  Broker 1001 marked for removal
  Broker 1006 marked for removal
  Broker 1004 marked for removal
  Broker 1003 marked for removal
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  -
  Replacing 6, added 0, missing 0, total count changed by -6

Action:
  Shrinking topic by 6 broker(s)
  Optimizing leader/follower ratios

Partition map changes:
  staging.api-gateway.request p0: [1008 1014 1007] -> [1007 1014 1008] preferred leader
  staging.changeprop-staging.error p0: [1008 1012 1015] -> [1008 1015 1012] preferred leader
  staging.changeprop-staging.retry.mediawiki.job.ThumbnailRender p0: [1009 1007 1011] -> [1009 1007 1011] no-op
  staging.changeprop-staging.retry.mediawiki.revision-create p0: [1014 1011 1009] -> [1014 1011 1009] no-op
  staging.cqs-external.sparql-query p0: [1012 1014 1008] -> [1014 1012 1008] preferred leader
  staging.eventgate-analytics-external.error.validation p0: [1015 1010 1009] -> [1009 1015 1010] preferred leader
  staging.eventgate-analytics-external.test.event p0: [1008 1007 1015] -> [1008 1007 1015] no-op
  staging.eventgate-analytics.error.validation p0: [1012 1009 1013] -> [1012 1013 1009] preferred leader
  staging.eventgate-analytics.test.event p0: [1014 1012 1009] -> [1009 1012 1014] preferred leader
  staging.eventgate-main.test.event p0: [1009 1011 1013] -> [1011 1013 1009] preferred leader
  staging.liftwing.test-events p0: [1008 1010 1014] -> [1008 1014 1010] preferred leader
  staging.liftwing.test-outlink-events p0: [1013 1012 1008] -> [1012 1013 1008] preferred leader
  staging.mediawiki.api-request p0: [1007 1011 1015] -> [1011 1007 1015] preferred leader
  staging.mediawiki.cirrussearch-request p0: [1007 1010 1015] -> [1015 1007 1010] preferred leader
  staging.mediawiki.job.ThumbnailRender p0: [1009 1001 1003] -> [1013 1009 1012] replaced broker
  staging.mediawiki.job.renameUser p0: [1002 1006 1005] -> [1014 1009 1010] replaced broker
  staging.mediawiki.page-create p0: [1003 1004 1006] -> [1007 1010 1013] replaced broker
  staging.mediawiki.page-delete p0: [1001 1003 1004] -> [1010 1008 1015] replaced broker
  staging.mediawiki.page-links-change p0: [1003 1001 1004] -> [1007 1009 1011] replaced broker
  staging.mediawiki.page-move p0: [1001 1003 1004] -> [1008 1013 1011] replaced broker
  staging.mediawiki.page-properties-change p0: [1005 1006 1002] -> [1007 1015 1010] replaced broker
  staging.mediawiki.page-restrictions-change p0: [1001 1006 1005] -> [1012 1007 1014] replaced broker
  staging.mediawiki.page-undelete p0: [1004 1001 1003] -> [1008 1010 1013] replaced broker
  staging.mediawiki.recentchange p0: [1001 1004 1006] -> [1009 1015 1011] replaced broker
  staging.mediawiki.revision-create p0: [1002 1003 1004] -> [1008 1013 1012] replaced broker
  staging.mediawiki.revision-score p0: [1001 1003 1004] -> [1009 1011 1014] replaced broker
  staging.mediawiki.revision-tags-change p0: [1006 1004 1002] -> [1015 1009 1011] replaced broker
  staging.mediawiki.revision-visibility-change p0: [1005 1002 1003] -> [1012 1014 1008] replaced broker
  staging.mediawiki.user-blocks-change p0: [1004 1003 1006] -> [1007 1015 1010] replaced broker
  staging.rdf-streaming-updater.fetch-failure p0: [1007 1006 1002] -> [1013 1007 1012] replaced broker
  staging.rdf-streaming-updater.mutation p0: [1002 1005 1008] -> [1013 1011 1008] replaced broker
  staging.rdf-streaming-updater.state-inconsistency p0: [1004 1006 1002] -> [1010 1013 1008] replaced broker
  staging.resource-purge p0: [1005 1006 1002] -> [1007 1014 1012] replaced broker
  staging.resource_change p0: [1001 1005 1003] -> [1015 1011 1009] replaced broker
  staging.swift.example_container.upload-complete p0: [1009 1001 1003] -> [1015 1010 1009] replaced broker
  staging.test.event p0: [1003 1005 1001] -> [1013 1012 1007] replaced broker
  staging.test_purge p0: [1004 1003 1006] -> [1010 1014 1007] replaced broker
  staging.wdqs-external.sparql-query p0: [1003 1004 1006] -> [1014 1011 1009] replaced broker
  staging.wdqs-internal.sparql-query p0: [1005 1001 1003] -> [1011 1013 1008] replaced broker
  statsv p0: [1006 1004 1003] -> [1014 1012 1008] replaced broker
  tchin_test0 p0: [1007 1006 1002] -> [1010 1015 1007] replaced broker
  temp_NavigationTiming p0: [1004 1006 1002] -> [1015 1008 1010] replaced broker
  temp_NavigationTiming_replay p0: [1001 1003 1004] -> [1012 1009 1015] replaced broker
  test p0: [1001 1003 1004] -> [1015 1009 1012] replaced broker
  test.instrumentation p0: [1002 1005 1008] -> [1010 1008 1014] replaced broker
  test_elukey p0: [1009 1002 1005] -> [1011 1009 1013] replaced broker
  test_flink0_page_content_change p0: [1006 1002 1005] -> [1011 1008 1013] replaced broker
  test_wdqs_streaming_updater p0: [1007 1006 1002] -> [1014 1010 1007] replaced broker
  test_wdqs_streaming_updater_maryum p0: [1003] -> [1013] replaced broker
  translationrecommendationuirequests p0: [1007 1006 1002] -> [1015 1011 1007] replaced broker
  virtualpageview p0: [1001 1005 1003] -> [1012 1014 1007] replaced broker

Broker distribution:
  degree [min/max/avg]: 4/11/6.13 -> 6/8/6.44
  -
  Broker 1007 - leader: 6, follower: 12, total: 18
  Broker 1008 - leader: 6, follower: 12, total: 18
  Broker 1009 - leader: 5, follower: 13, total: 18
  Broker 1010 - leader: 5, follower: 11, total: 16
  Broker 1011 - leader: 5, follower: 11, total: 16
  Broker 1012 - leader: 6, follower: 10, total: 16
  Broker 1013 - leader: 5, follower: 11, total: 16
  Broker 1014 - leader: 6, follower: 10, total: 16
  Broker 1015 - leader: 7, follower: 10, total: 17

WARN:
  [none]

New partition maps:
  s-to-v.json [combined map]
  staging.mediawiki.job.renameUser.json
  staging.mediawiki.page-restrictions-change.json
  staging.mediawiki.revision-visibility-change.json
  test_wdqs_streaming_updater.json
  staging.rdf-streaming-updater.mutation.json
  staging.resource-purge.json
  tchin_test0.json
  staging.eventgate-analytics.test.event.json
  staging.eventgate-main.test.event.json
  staging.liftwing.test-events.json
  staging.mediawiki.page-undelete.json
  staging.mediawiki.revision-create.json
  temp_NavigationTiming.json
  test_elukey.json
  test_wdqs_streaming_updater_maryum.json
  staging.rdf-streaming-updater.state-inconsistency.json
  temp_NavigationTiming_replay.json
  staging.eventgate-analytics-external.error.validation.json
  staging.eventgate-analytics.error.validation.json
  staging.mediawiki.cirrussearch-request.json
  staging.mediawiki.page-delete.json
  staging.mediawiki.page-links-change.json
  staging.mediawiki.recentchange.json
  staging.rdf-streaming-updater.fetch-failure.json
  staging.wdqs-external.sparql-query.json
  staging.mediawiki.page-move.json
  virtualpageview.json
  staging.mediawiki.job.ThumbnailRender.json
  staging.mediawiki.page-create.json
  staging.mediawiki.revision-score.json
  staging.resource_change.json
  staging.test_purge.json
  translationrecommendationuirequests.json
  staging.mediawiki.api-request.json
  staging.mediawiki.page-properties-change.json
  staging.mediawiki.user-blocks-change.json
  staging.test.event.json
  test.json
  staging.swift.example_container.upload-complete.json
  staging.wdqs-internal.sparql-query.json
  statsv.json
  staging.api-gateway.request.json
  staging.changeprop-staging.error.json
  staging.cqs-external.sparql-query.json
  staging.liftwing.test-outlink-events.json
  staging.mediawiki.revision-tags-change.json
  test.instrumentation.json
  test_flink0_page_content_change.json

Final batch before starting on the large`webrequest_(text|uploads)` movements.

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild --topics '^(wdqs_streaming_updater_15_test|wdqs_streaming_updater_test|wdqs_streaming_updater_test_T289836|wdqs_streaming_updater_test_cross_dc|wdqs_streaming_updater_test_zbyszko|webrequest|webrequest-test-text|webrequest_canary|webrequest_canary_test|webrequest_live_sampled|webrequest_misc|webrequest_misc_test|webrequest_sampled|webrequest_sampled_live|webrequest_test|webrequest_test_text|with|wmf_netflow|__consumer_offsets|__transaction_state|_schemas)$' --brokers '1007,1008,1009,1010,1011,1012,1013,1014,1015' --skip-no-ops --optimize-leadership --out-file anything-but-large-webrequests

Topics:
  __consumer_offsets
  __transaction_state
  _schemas
  wdqs_streaming_updater_15_test
  wdqs_streaming_updater_test
  wdqs_streaming_updater_test_T289836
  wdqs_streaming_updater_test_cross_dc
  wdqs_streaming_updater_test_zbyszko
  webrequest
  webrequest-test-text
  webrequest_canary
  webrequest_canary_test
  webrequest_live_sampled
  webrequest_misc
  webrequest_misc_test
  webrequest_sampled
  webrequest_sampled_live
  webrequest_test
  webrequest_test_text
  with
  wmf_netflow

Broker change summary:
  Broker 1006 marked for removal
  Broker 1005 marked for removal
  Broker 1004 marked for removal
  Broker 1001 marked for removal
  Broker 1003 marked for removal
  Broker 1002 marked for removal
  New broker 1011
  New broker 1012
  New broker 1014
  New broker 1010
  New broker 1015
  New broker 1013
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Optimizing leader/follower ratios

Partition map changes:
  __consumer_offsets p0: [1002 1004 1006] -> [1015 1007 1008] replaced broker
  __consumer_offsets p1: [1007 1008 1002] -> [1008 1010 1007] replaced broker
  __consumer_offsets p2: [1001 1003 1004] -> [1009 1007 1012] replaced broker
  __consumer_offsets p3: [1009 1005 1001] -> [1014 1009 1012] replaced broker
  __consumer_offsets p4: [1005 1001 1008] -> [1010 1008 1015] replaced broker
  __consumer_offsets p5: [1003 1009 1005] -> [1009 1011 1015] replaced broker
  __consumer_offsets p6: [1004 1006 1003] -> [1013 1011 1009] replaced broker
  __consumer_offsets p7: [1006 1003 1007] -> [1014 1007 1012] replaced broker
  __consumer_offsets p8: [1008 1002 1005] -> [1008 1010 1014] replaced broker
  __consumer_offsets p9: [1002 1008 1004] -> [1011 1008 1015] replaced broker
  __consumer_offsets p10: [1007 1009 1002] -> [1009 1007 1013] replaced broker
  __consumer_offsets p11: [1001 1007 1009] -> [1009 1012 1007] replaced broker
  __consumer_offsets p12: [1009 1004 1001] -> [1010 1014 1009] replaced broker
  __consumer_offsets p13: [1005 1001 1008] -> [1008 1007 1010] replaced broker
  __consumer_offsets p14: [1003 1006 1004] -> [1009 1011 1014] replaced broker
  __consumer_offsets p15: [1004 1003 1009] -> [1009 1013 1011] replaced broker
  __consumer_offsets p16: [1006 1005 1003] -> [1007 1012 1014] replaced broker
  __consumer_offsets p17: [1008 1007 1001] -> [1007 1008 1011] replaced broker
  __consumer_offsets p18: [1002 1004 1006] -> [1012 1009 1015] replaced broker
  __consumer_offsets p19: [1007 1006 1003] -> [1008 1007 1015] replaced broker
  __consumer_offsets p20: [1001 1005 1008] -> [1008 1013 1012] replaced broker
  __consumer_offsets p21: [1009 1001 1007] -> [1007 1009 1010] replaced broker
  __consumer_offsets p22: [1005 1002 1006] -> [1008 1015 1010] replaced broker
  __consumer_offsets p23: [1003 1009 1005] -> [1009 1013 1011] replaced broker
  __consumer_offsets p24: [1004 1008 1002] -> [1008 1013 1010] replaced broker
  __consumer_offsets p25: [1006 1001 1004] -> [1007 1011 1014] replaced broker
  __consumer_offsets p26: [1008 1004 1002] -> [1008 1012 1015] replaced broker
  __consumer_offsets p27: [1002 1003 1009] -> [1009 1011 1015] replaced broker
  __consumer_offsets p28: [1007 1009 1003] -> [1009 1007 1012] replaced broker
  __consumer_offsets p29: [1001 1006 1007] -> [1007 1009 1012] replaced broker
  __consumer_offsets p30: [1009 1002 1005] -> [1009 1014 1010] replaced broker
  __consumer_offsets p31: [1005 1008 1001] -> [1008 1013 1010] replaced broker
  __consumer_offsets p32: [1003 1007 1008] -> [1008 1007 1011] replaced broker
  __consumer_offsets p33: [1004 1003 1009] -> [1013 1009 1011] replaced broker
  __consumer_offsets p34: [1006 1005 1003] -> [1007 1008 1014] replaced broker
  __consumer_offsets p35: [1008 1002 1004] -> [1007 1008 1013] replaced broker
  __consumer_offsets p36: [1002 1009 1007] -> [1007 1009 1015] replaced broker
  __consumer_offsets p37: [1007 1006 1002] -> [1007 1009 1014] replaced broker
  __consumer_offsets p38: [1001 1004 1006] -> [1008 1015 1012] replaced broker
  __consumer_offsets p39: [1009 1007 1001] -> [1009 1007 1014] replaced broker
  __consumer_offsets p40: [1005 1001 1009] -> [1009 1015 1010] replaced broker
  __consumer_offsets p41: [1003 1008 1005] -> [1008 1011 1013] replaced broker
  __consumer_offsets p42: [1004 1009 1001] -> [1009 1013 1010] replaced broker
  __consumer_offsets p43: [1006 1005 1002] -> [1009 1014 1010] replaced broker
  __consumer_offsets p44: [1008 1007 1003] -> [1007 1008 1015] replaced broker
  __consumer_offsets p45: [1002 1008 1004] -> [1008 1011 1015] replaced broker
  __consumer_offsets p46: [1007 1003 1006] -> [1007 1009 1013] replaced broker
  __consumer_offsets p47: [1001 1006 1007] -> [1007 1014 1012] replaced broker
  __consumer_offsets p48: [1009 1005 1001] -> [1009 1012 1015] replaced broker
  __consumer_offsets p49: [1005 1002 1008] -> [1007 1008 1010] replaced broker
  __transaction_state p0: [1003 1001 1004] -> [1008 1011 1014] replaced broker
  __transaction_state p1: [1004 1003 1006] -> [1009 1013 1012] replaced broker
  __transaction_state p2: [1006 1004 1002] -> [1007 1011 1014] replaced broker
  __transaction_state p3: [1002 1006 1005] -> [1009 1012 1015] replaced broker
  __transaction_state p4: [1005 1002 1003] -> [1008 1012 1015] replaced broker
  __transaction_state p5: [1001 1005 1003] -> [1008 1013 1010] replaced broker
  __transaction_state p6: [1003 1005 1001] -> [1011 1007 1013] replaced broker
  __transaction_state p7: [1004 1001 1003] -> [1007 1013 1010] replaced broker
  __transaction_state p8: [1006 1003 1004] -> [1008 1014 1010] replaced broker
  __transaction_state p9: [1002 1004 1006] -> [1009 1012 1015] replaced broker
  __transaction_state p10: [1005 1006 1002] -> [1008 1012 1014] replaced broker
  __transaction_state p11: [1001 1005 1003] -> [1014 1010 1009] replaced broker
  __transaction_state p12: [1003 1002 1005] -> [1007 1011 1013] replaced broker
  __transaction_state p13: [1004 1001 1003] -> [1010 1013 1008] replaced broker
  __transaction_state p14: [1006 1001 1003] -> [1011 1014 1009] replaced broker
  __transaction_state p15: [1002 1003 1004] -> [1015 1012 1007] replaced broker
  __transaction_state p16: [1005 1006 1002] -> [1012 1015 1008] replaced broker
  __transaction_state p17: [1001 1006 1005] -> [1014 1010 1007] replaced broker
  __transaction_state p18: [1003 1006 1002] -> [1011 1014 1009] replaced broker
  __transaction_state p19: [1004 1002 1003] -> [1012 1007 1013] replaced broker
  __transaction_state p20: [1006 1005 1001] -> [1014 1007 1011] replaced broker
  __transaction_state p21: [1002 1003 1004] -> [1015 1008 1012] replaced broker
  __transaction_state p22: [1005 1003 1006] -> [1013 1012 1009] replaced broker
  __transaction_state p23: [1001 1004 1006] -> [1010 1009 1015] replaced broker
  __transaction_state p24: [1003 1004 1006] -> [1011 1013 1008] replaced broker
  __transaction_state p25: [1004 1006 1002] -> [1013 1011 1009] replaced broker
  __transaction_state p26: [1006 1002 1005] -> [1010 1014 1008] replaced broker
  __transaction_state p27: [1002 1005 1003] -> [1012 1015 1007] replaced broker
  __transaction_state p28: [1005 1001 1003] -> [1015 1012 1007] replaced broker
  __transaction_state p29: [1001 1003 1004] -> [1015 1008 1010] replaced broker
  __transaction_state p30: [1003 1001 1004] -> [1013 1011 1009] replaced broker
  __transaction_state p31: [1004 1003 1006] -> [1011 1013 1009] replaced broker
  __transaction_state p32: [1006 1004 1002] -> [1014 1010 1008] replaced broker
  __transaction_state p33: [1002 1006 1005] -> [1015 1007 1011] replaced broker
  __transaction_state p34: [1005 1002 1003] -> [1012 1014 1008] replaced broker
  __transaction_state p35: [1001 1005 1003] -> [1010 1013 1007] replaced broker
  __transaction_state p36: [1003 1005 1001] -> [1013 1011 1009] replaced broker
  __transaction_state p37: [1004 1001 1003] -> [1010 1013 1009] replaced broker
  __transaction_state p38: [1006 1003 1004] -> [1014 1008 1012] replaced broker
  __transaction_state p39: [1002 1004 1006] -> [1011 1007 1015] replaced broker
  __transaction_state p40: [1005 1006 1002] -> [1012 1007 1014] replaced broker
  __transaction_state p41: [1001 1005 1003] -> [1015 1010 1009] replaced broker
  __transaction_state p42: [1003 1002 1005] -> [1014 1011 1008] replaced broker
  __transaction_state p43: [1004 1001 1003] -> [1013 1012 1007] replaced broker
  __transaction_state p44: [1006 1001 1003] -> [1010 1014 1008] replaced broker
  __transaction_state p45: [1002 1003 1004] -> [1010 1015 1008] replaced broker
  __transaction_state p46: [1005 1006 1002] -> [1012 1014 1009] replaced broker
  __transaction_state p47: [1001 1006 1005] -> [1015 1010 1007] replaced broker
  __transaction_state p48: [1003 1006 1002] -> [1015 1007 1011] replaced broker
  __transaction_state p49: [1004 1002 1003] -> [1013 1010 1008] replaced broker
  _schemas p0: [1007] -> [1007] no-op
  wdqs_streaming_updater_15_test p0: [1007 1006 1002] -> [1015 1007 1011] replaced broker
  wdqs_streaming_updater_test p0: [1005 1008 1001] -> [1011 1014 1008] replaced broker
  wdqs_streaming_updater_test_T289836 p0: [1001 1003 1004] -> [1012 1009 1015] replaced broker
  wdqs_streaming_updater_test_cross_dc p0: [1005 1003 1006] -> [1014 1012 1009] replaced broker
  wdqs_streaming_updater_test_zbyszko p0: [1006 1002 1005] -> [1010 1013 1008] replaced broker
  webrequest p0: [1002 1004 1006] -> [1011 1013 1008] replaced broker
  webrequest-test-text p0: [1001 1003 1004] -> [1013 1010 1007] replaced broker
  webrequest_canary p0: [1006 1002 1005] -> [1014 1011 1009] replaced broker
  webrequest_canary_test p0: [1003 1006 1002] -> [1012 1009 1015] replaced broker
  webrequest_canary_test p1: [1004 1002 1003] -> [1014 1012 1008] replaced broker
  webrequest_canary_test p2: [1006 1005 1001] -> [1010 1007 1015] replaced broker
  webrequest_live_sampled p0: [1003 1008 1007] -> [1011 1007 1008] replaced broker
  webrequest_misc p0: [1001 1005 1003] -> [1012 1013 1007] replaced broker
  webrequest_misc p1: [1003 1005 1001] -> [1014 1009 1010] replaced broker
  webrequest_misc p2: [1004 1001 1003] -> [1011 1007 1015] replaced broker
  webrequest_misc p3: [1006 1003 1004] -> [1013 1012 1009] replaced broker
  webrequest_misc p4: [1002 1004 1006] -> [1010 1013 1009] replaced broker
  webrequest_misc p5: [1005 1006 1002] -> [1011 1008 1014] replaced broker
  webrequest_misc p6: [1001 1006 1005] -> [1007 1013 1011] replaced broker
  webrequest_misc p7: [1003 1002 1005] -> [1010 1014 1008] replaced broker
  webrequest_misc p8: [1004 1001 1003] -> [1015 1009 1011] replaced broker
  webrequest_misc p9: [1006 1001 1003] -> [1015 1008 1010] replaced broker
  webrequest_misc p10: [1002 1003 1004] -> [1012 1013 1009] replaced broker
  webrequest_misc p11: [1005 1006 1002] -> [1014 1010 1008] replaced broker
  webrequest_misc_test p0: [1001 1006 1005] -> [1007 1011 1014] replaced broker
  webrequest_misc_test p1: [1003 1002 1005] -> [1013 1012 1009] replaced broker
  webrequest_misc_test p2: [1004 1001 1003] -> [1012 1014 1009] replaced broker
  webrequest_sampled p0: [1003 1004 1006] -> [1012 1009 1015] replaced broker
  webrequest_sampled p1: [1004 1006 1002] -> [1010 1015 1007] replaced broker
  webrequest_sampled p2: [1006 1002 1005] -> [1013 1008 1012] replaced broker
  webrequest_sampled_live p0: [1002 1005 1008] -> [1015 1010 1008] replaced broker
  webrequest_test p0: [1005 1008 1001] -> [1015 1011 1008] replaced broker
  webrequest_test_text p0: [1001 1005 1003] -> [1010 1013 1007] replaced broker
  with p0: [1005 1008 1001] -> [1014 1012 1008] replaced broker
  wmf_netflow p0: [1002 1003 1004] -> [1011 1013 1007] replaced broker

Broker distribution:
  degree [min/max/avg]: 6/8/6.44 -> 6/8/6.44
  -
  Broker 1007 - leader: 19, follower: 34, total: 53
  Broker 1008 - leader: 18, follower: 34, total: 52
  Broker 1009 - leader: 18, follower: 34, total: 52
  Broker 1010 - leader: 15, follower: 27, total: 42
  Broker 1011 - leader: 13, follower: 28, total: 41
  Broker 1012 - leader: 13, follower: 29, total: 42
  Broker 1013 - leader: 12, follower: 29, total: 41
  Broker 1014 - leader: 14, follower: 27, total: 41
  Broker 1015 - leader: 14, follower: 28, total: 42

WARN:
  [none]

New partition maps:
  anything-but-large-webrequests.json [combined map]
  webrequest-test-text.json
  webrequest_live_sampled.json
  webrequest_misc.json
  wmf_netflow.json
  wdqs_streaming_updater_15_test.json
  webrequest.json
  webrequest_canary.json
  webrequest_canary_test.json
  webrequest_sampled_live.json
  webrequest_test_text.json
  __consumer_offsets.json
  __transaction_state.json
  wdqs_streaming_updater_test_T289836.json
  wdqs_streaming_updater_test_cross_dc.json
  webrequest_misc_test.json
  with.json
  wdqs_streaming_updater_test.json
  wdqs_streaming_updater_test_zbyszko.json
  webrequest_sampled.json
  webrequest_test.json

We have now reassigned every single topic except the 2 largest ones:

webrequest_text
webrequest_upload

As it stands, the brokers 1009-> 1015 are unbalanced in storage, assigned partitions and leadership. This causes a bit of discrepancy in traffic and producer/consumer delay.

Before reassigning the last 2 topics, we will first attempt to perform an in-place storage rebalance.

We first fetch the broker and topic metrics using prometheus-metricsfetcher (installed since https://phabricator.wikimedia.org/T348315), and will then attempt to use topicmappr to re-binpack the partitions in a more efficient way.

brouberol@kafka-jumbo1010:~/topicmappr$ prometheus-metricsfetcher \
  --prometheus-url http://prometheus.svc.eqiad.wmnet/ops/ \
  --zk-addr conf1007.eqiad.wmnet/kafka/jumbo-eqiad \
  --partition-size-query 'max(kafka_log_Size{cluster="kafka_jumbo"}) by (topic,partition)' \
  --broker-id-map "kafka-jumbo1001:9100=1001,kafka-jumbo1002:9100=1002,kafka-jumbo1003:9100=1003,kafka-jumbo1004:9100=1004,kafka-jumbo1005:9100=1005,kafka-jumbo1006:9100=1006,kafka-jumbo1007:9100=1007,kafka-jumbo1008:9100=1008,kafka-jumbo1009:9100=1009,kafka-jumbo1010:9100=1010,kafka-jumbo1011:9100=1011,kafka-jumbo1012:9100=1012,kafka-jumbo1013:9100=1013,kafka-jumbo1014:9100=1014,kafka-jumbo1015:9100=1015" \
  --broker-id-label instance \
  --broker-storage-query 'node_filesystem_avail_bytes{cluster="kafka_jumbo", mountpoint="/srv"}'
time="2023-10-06T12:28:53.787Z" level=info msg="Getting broker storage stats from Prometheus"
time="2023-10-06T12:28:53.787Z" level=info msg="Connected to [2620:0:861:101:10:64:0:207]:2181" logger=zk
time="2023-10-06T12:28:53.789Z" level=info msg="authenticated: id=5982453769865003572, timeout=20000" logger=zk
time="2023-10-06T12:28:53.789Z" level=info msg="re-submitting `0` credentials after reconnect" logger=zk
time="2023-10-06T12:28:53.794Z" level=info msg="Broker ID Map: map[kafka-jumbo1001:9100:1001 kafka-jumbo1002:9100:1002 kafka-jumbo1003:9100:1003 kafka-jumbo1004:9100:1004 kafka-jumbo1005:9100:1005 kafka-jumbo1006:9100:1006 kafka-jumbo1007:9100:1007 kafka-jumbo1008:9100:1008 kafka-jumbo1009:9100:1009 kafka-jumbo1010:9100:1010 kafka-jumbo1011:9100:101 kafka-jumbo1012:9100:1012 kafka-jumbo1013:9100:1013 kafka-jumbo1014:9100:1014 kafka-jumbo1015:9100:1015]"
time="2023-10-06T12:28:53.794Z" level=info msg="Getting partition sizes from Prometheus"
time="2023-10-06T12:28:53.887Z" level=info msg="writing data to /topicmappr/partitionmeta"
time="2023-10-06T12:28:53.894Z" level=info msg="writing data to /topicmappr/brokermetrics"
time="2023-10-06T12:28:53.896Z" level=info msg="recv loop terminated: err=EOF" logger=zk
time="2023-10-06T12:28:53.896Z" level=info msg="send loop terminated: err=<nil>" logger=zk

$ topicmappr --zk-metrics-prefix kafka/jumbo-eqiad/topicmappr rebalance --topics-exclude '^webrequest_(text|upload)$' --topics '.*' --brokers '1007,1008,1009,1010,1011,1012,1013,1014,1015' --optimize-leadership --out-file rebalancing
Topics:
...
Topics excluded:
  webrequest_text
  webrequest_upload

Validating broker list:

Brokers targeted for partition offloading (>= 20.00% threshold below hmean):

Reassignment parameters:
  Ignoring partitions smaller than 512MB
  Free storage mean, harmonic mean: 13672.10GB, 13105.45GB
  Broker free storage limits (with a 46.00% tolerance from mean):
    Sources limited to <= 19961.26GB
    Destinations limited to >= 7382.93GB

Broker 1009 relocations planned:
  [92.91GB] codfw.mediawiki.job.refreshLinks p0 -> 1012
  [71.84GB] codfw.mediawiki.api-request p2 -> 1010
  [64.36GB] eqiad.mediawiki.api-request p3 -> 1015
  [64.31GB] eqiad.mediawiki.api-request p10 -> 1014
  [52.72GB] codfw.mediawiki.job.htmlCacheUpdate p0 -> 1012
  [40.83GB] codfw.resource-purge p0 -> 1015
  [36.02GB] eventlogging_DesktopWebUIActionsTracking p0 -> 1011
  [15.20GB] eqiad.mediawiki.job.parsoidCachePrewarm p0 -> 1010
  [14.65GB] eventlogging_VirtualPageView p9 -> 1015

Broker 1007 relocations planned:
  [87.83GB] eventlogging_SearchSatisfaction p0 -> 1012
  [71.84GB] codfw.mediawiki.api-request p0 -> 1015
  [64.35GB] eqiad.mediawiki.api-request p0 -> 1010
  [54.61GB] eqiad.mediawiki.cirrussearch-request p7 -> 1011
  [51.13GB] netflow p1 -> 1015
  [49.09GB] codfw.mediawiki.cirrussearch-request p8 -> 1014
  [34.71GB] codfw.mediawiki.job.parsoidCachePrewarm p0 -> 1011
  [14.69GB] eventlogging_VirtualPageView p1 -> 1012
  [14.56GB] eventlogging_VirtualPageView p2 -> 1010

Broker 1008 relocations planned:
  [72.02GB] codfw.mediawiki.api-request p3 -> 1015
  [71.64GB] codfw.mediawiki.api-request p11 -> 1011
  [64.34GB] eqiad.mediawiki.api-request p8 -> 1012
  [54.60GB] eqiad.mediawiki.cirrussearch-request p9 -> 1010
  [50.00GB] codfw.mediawiki.cirrussearch-request p6 -> 1014
  [14.71GB] eventlogging_VirtualPageView p7 -> 1015
  -
  Total relocation volume: 1222.96GB
Partition map changes:
  -L p0: [1008 1007 1015] -> [1007 1015 1008] preferred leader
  -l p0: [1009 1014 1012] -> [1012 1014 1009] preferred leader
  04 p0: [1015 1010 1007] -> [1015 1007 1010] preferred leader
  12 p0: [1008 1011 1013] -> [1013 1011 1008] preferred leader
  ContentTranslationAbuseFilter p0: [1010 1009 1013] -> [1010 1009 1013] no-op
  DataHubUpgradeHistory_v1 p0: [1009 1015 1007] -> [1015 1007 1009] preferred leader
  DataHubUsageEvent_v1 p0: [1014 1012 1008] -> [1014 1012 1008] no-op
  DataHubUsageEvent_v1 p1: [1015 1008 1010] -> [1008 1010 1015] preferred leader
  DataHubUsageEvent_v1 p2: [1008 1007 1012] -> [1012 1007 1008] preferred leader
  DataHubUsageEvent_v1 p3: [1007 1009 1011] -> [1007 1011 1009] preferred leader
  DataHubUsageEvent_v1 p4: [1009 1011 1014] -> [1011 1014 1009] preferred leader
  EditorJourney p0: [1009 1012 1014] -> [1012 1014 1009] preferred leader
  FailedMetadataChangeEvent_v4 p0: [1011 1013 1009] -> [1011 1013 1009] no-op
  FailedMetadataChangeEvent_v4 p1: [1007 1010 1014] -> [1007 1014 1010] preferred leader
  FailedMetadataChangeEvent_v4 p2: [1007 1015 1011] -> [1007 1015 1011] no-op
  FailedMetadataChangeEvent_v4 p3: [1007 1010 1013] -> [1013 1007 1010] preferred leader
  FailedMetadataChangeEvent_v4 p4: [1009 1014 1011] -> [1011 1014 1009] preferred leader
  FailedMetadataChangeProposal_v1 p0: [1007 1015 1012] -> [1012 1015 1007] preferred leader
  FailedMetadataChangeProposal_v1 p1: [1008 1013 1012] -> [1013 1012 1008] preferred leader
  FailedMetadataChangeProposal_v1 p2: [1015 1010 1007] -> [1015 1007 1010] preferred leader
  FailedMetadataChangeProposal_v1 p3: [1011 1013 1009] -> [1013 1011 1009] preferred leader
  FailedMetadataChangeProposal_v1 p4: [1012 1013 1008] -> [1012 1013 1008] no-op
  KaiOSAppFeedback p0: [1010 1008 1007] -> [1010 1007 1008] preferred leader
  MetadataAuditEvent_v4 p0: [1011 1015 1009] -> [1011 1009 1015] preferred leader
  MetadataAuditEvent_v4 p1: [1008 1012 1014] -> [1012 1014 1008] preferred leader
  MetadataAuditEvent_v4 p2: [1014 1010 1007] -> [1014 1007 1010] preferred leader
  MetadataAuditEvent_v4 p3: [1015 1010 1009] -> [1009 1010 1015] preferred leader
  MetadataAuditEvent_v4 p4: [1008 1013 1010] -> [1013 1008 1010] preferred leader
  MetadataChangeEvent_v4 p0: [1009 1011 1014] -> [1011 1009 1014] preferred leader
  MetadataChangeEvent_v4 p1: [1012 1015 1008] -> [1008 1012 1015] preferred leader
  MetadataChangeEvent_v4 p2: [1007 1012 1015] -> [1012 1007 1015] preferred leader
  MetadataChangeEvent_v4 p3: [1010 1008 1013] -> [1013 1008 1010] preferred leader
  MetadataChangeEvent_v4 p4: [1013 1008 1011] -> [1011 1008 1013] preferred leader
  MetadataChangeLog_Timeseries_v1 p0: [1013 1007 1011] -> [1013 1007 1011] no-op
  MetadataChangeLog_Timeseries_v1 p1: [1012 1008 1014] -> [1012 1008 1014] no-op
  MetadataChangeLog_Timeseries_v1 p2: [1014 1007 1009] -> [1007 1009 1014] preferred leader
  MetadataChangeLog_Timeseries_v1 p3: [1013 1009 1010] -> [1013 1009 1010] no-op
  MetadataChangeLog_Timeseries_v1 p4: [1008 1014 1012] -> [1008 1014 1012] no-op
...
Broker distribution:
  degree [min/max/avg]: 6/8/6.44 -> 6/8/6.44
  -
  Broker 1007 - leader: 175, follower: 340, total: 515
  Broker 1008 - leader: 176, follower: 341, total: 517
  Broker 1009 - leader: 175, follower: 336, total: 511
  Broker 1010 - leader: 149, follower: 290, total: 439
  Broker 1011 - leader: 147, follower: 285, total: 432
  Broker 1012 - leader: 146, follower: 284, total: 430
  Broker 1013 - leader: 145, follower: 285, total: 430
  Broker 1014 - leader: 147, follower: 286, total: 433
  Broker 1015 - leader: 147, follower: 289, total: 436

Storage free change estimations:
  range: 5730.84GB -> 4948.48GB
  range spread: 57.76% -> 47.70%
  std. deviation: 2590.89GB -> 2300.68GB
  min-max: 9922.40GB, 15653.24GB -> 10375.23GB, 15323.71GB
  -
  Broker 1007: 10028.98 -> 10471.79 (+442.81GB, 4.42%) 
  Broker 1008: 10081.62 -> 10408.92 (+327.31GB, 3.25%) 
  Broker 1009: 9922.40 -> 10375.23 (+452.83GB, 4.56%) 
  Broker 1010: 15512.70 -> 15292.15 (-220.55GB, -1.42%) 
  Broker 1011: 15484.97 -> 15287.99 (-196.98GB, -1.27%) 
  Broker 1012: 15611.59 -> 15299.10 (-312.49GB, -2.00%) 
  Broker 1013: 15281.33 -> 15281.33 (+0.00GB, 0.00%) 
  Broker 1014: 15472.05 -> 15308.65 (-163.40GB, -1.06%) 
  Broker 1015: 15653.24 -> 15323.71 (-329.53GB, -2.11%) 

WARN:
  [none]

I'm actually going to omit the --optimize-leadership flag here, as the same command run without it leads to the same number of leaderships / broker:

Broker distribution:
  degree [min/max/avg]: 6/8/6.44 -> 6/8/6.44
  -
  Broker 1007 - leader: 176, follower: 339, total: 515
  Broker 1008 - leader: 176, follower: 341, total: 517
  Broker 1009 - leader: 182, follower: 329, total: 511
  Broker 1010 - leader: 151, follower: 288, total: 439
  Broker 1011 - leader: 143, follower: 289, total: 432
  Broker 1012 - leader: 140, follower: 290, total: 430
  Broker 1013 - leader: 139, follower: 291, total: 430
  Broker 1014 - leader: 148, follower: 285, total: 433
  Broker 1015 - leader: 152, follower: 284, total: 436

As the partitions widely differ in size, the storage bin-packing should be enough: as all topics have the same retention (7 days): their size is proportional to the input traffic the same way for all topics. De-hotspotting storage should de-hotspot traffic.

The next step is to evacuate webrequest_upload. We'll generate a reassignment plan in multiple steps:

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild --topics webrequest_upload --brokers 1015,1014,1013,1012,1011,1010,1009,1008,1007 --chunk-step-size 1 --force-rebuild | grep -v no-op

Topics:
  webrequest_upload

Broker change summary:
  Broker 1003 marked for removal
  Broker 1004 marked for removal
  Broker 1006 marked for removal
  Broker 1001 marked for removal
  Broker 1002 marked for removal
  Broker 1005 marked for removal
  New broker 1012
  New broker 1015
  New broker 1011
  New broker 1013
  New broker 1010
  New broker 1014
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Force rebuilding map

Partition map changes:
  webrequest_upload p0: [1003 1004 1008] -> [1008 1010 1014] replaced broker
  webrequest_upload p1: [1004 1003 1008] -> [1013 1012 1007] replaced broker
  webrequest_upload p2: [1006 1007 1002] -> [1007 1014 1008] replaced broker
  webrequest_upload p3: [1008 1007 1001] -> [1015 1010 1009] replaced broker
  webrequest_upload p4: [1002 1008 1004] -> [1011 1015 1008] replaced broker
  webrequest_upload p5: [1007 1006 1003] -> [1009 1007 1013] replaced broker
  webrequest_upload p6: [1001 1005 1006] -> [1010 1013 1007] replaced broker
  webrequest_upload p7: [1009 1001 1007] -> [1012 1009 1014] replaced broker
  webrequest_upload p8: [1005 1009 1003] -> [1014 1012 1009] replaced broker
  webrequest_upload p9: [1003 1002 1009] -> [1008 1014 1011] replaced broker
  webrequest_upload p10: [1004 1001 1009] -> [1013 1011 1007] replaced broker
  webrequest_upload p11: [1006 1004 1001] -> [1007 1008 1011] replaced broker
  webrequest_upload p12: [1008 1007 1002] -> [1015 1010 1009] replaced broker
  webrequest_upload p13: [1002 1005 1006] -> [1011 1008 1013] replaced broker
  webrequest_upload p14: [1007 1008 1002] -> [1009 1012 1015] replaced broker
  webrequest_upload p15: [1001 1006 1005] -> [1010 1007 1015] replaced broker
  webrequest_upload p16: [1009 1004 1001] -> [1012 1015 1008] replaced broker
  webrequest_upload p17: [1005 1003 1008] -> [1014 1011 1007] replaced broker
  webrequest_upload p18: [1003 1002 1007] -> [1008 1014 1010] replaced broker
  webrequest_upload p19: [1004 1009 1003] -> [1013 1009 1012] replaced broker
  webrequest_upload p20: [1006 1001 1004] -> [1007 1013 1011] replaced broker
  webrequest_upload p21: [1008 1003 1005] -> [1015 1010 1008] replaced broker
  webrequest_upload p22: [1002 1007 1006] -> [1011 1015 1009] replaced broker
  webrequest_upload p23: [1007 1002 1009] -> [1009 1012 1013] replaced broker

Broker distribution:
  degree [min/max/avg]: 6/7/6.22 -> 6/8/6.44
  -
  Broker 1007 - leader: 3, follower: 6, total: 9
  Broker 1008 - leader: 3, follower: 6, total: 9
  Broker 1009 - leader: 3, follower: 6, total: 9
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 3, follower: 5, total: 8
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 3, follower: 5, total: 8
  Broker 1014 - leader: 2, follower: 5, total: 7
  Broker 1015 - leader: 3, follower: 5, total: 8


Generating reassignments in chunks 1 brokers at a time: 



Skipping noop map output for brokers 1015

Skipping noop map output for brokers 1014

Skipping noop map output for brokers 1013

Skipping noop map output for brokers 1012

Skipping noop map output for brokers 1011

Skipping noop map output for brokers 1010

Changes for partition map chunk 1 for brokers 1009
Partition map changes:
  webrequest_upload p7: [1009 1001 1007] -> [1012 1001 1007] replaced broker
  webrequest_upload p8: [1005 1009 1003] -> [1005 1012 1003] replaced broker
  webrequest_upload p9: [1003 1002 1009] -> [1003 1002 1011] replaced broker
  webrequest_upload p10: [1004 1001 1009] -> [1004 1001 1007] replaced broker
  webrequest_upload p16: [1009 1004 1001] -> [1012 1004 1001] replaced broker
  webrequest_upload p23: [1007 1002 1009] -> [1007 1002 1013] replaced broker


Changes for partition map chunk 2 for brokers 1008
Partition map changes:
  webrequest_upload p0: [1003 1004 1008] -> [1003 1004 1014] replaced broker
  webrequest_upload p1: [1004 1003 1008] -> [1004 1003 1007] replaced broker
  webrequest_upload p3: [1008 1007 1001] -> [1015 1007 1001] replaced broker
  webrequest_upload p4: [1002 1008 1004] -> [1002 1015 1004] replaced broker
  webrequest_upload p12: [1008 1007 1002] -> [1015 1007 1002] replaced broker
  webrequest_upload p14: [1007 1008 1002] -> [1007 1012 1002] replaced broker
  webrequest_upload p17: [1005 1003 1008] -> [1005 1003 1007] replaced broker
  webrequest_upload p21: [1008 1003 1005] -> [1015 1003 1005] replaced broker


Changes for partition map chunk 3 for brokers 1007
Partition map changes:
  webrequest_upload p2: [1006 1007 1002] -> [1006 1014 1002] replaced broker
  webrequest_upload p3: [1015 1007 1001] -> [1015 1010 1001] replaced broker
  webrequest_upload p5: [1007 1006 1003] -> [1009 1006 1003] replaced broker
  webrequest_upload p7: [1012 1001 1007] -> [1012 1001 1014] replaced broker
  webrequest_upload p12: [1015 1007 1002] -> [1015 1010 1002] replaced broker
  webrequest_upload p14: [1007 1012 1002] -> [1009 1012 1002] replaced broker
  webrequest_upload p18: [1003 1002 1007] -> [1003 1002 1010] replaced broker
  webrequest_upload p22: [1002 1007 1006] -> [1002 1015 1006] replaced broker
  webrequest_upload p23: [1007 1002 1013] -> [1009 1002 1013] replaced broker


Changes for partition map chunk 4 for brokers 1006
Partition map changes:
  webrequest_upload p2: [1006 1014 1002] -> [1007 1014 1002] replaced broker
  webrequest_upload p5: [1009 1006 1003] -> [1009 1007 1003] replaced broker
  webrequest_upload p6: [1001 1005 1006] -> [1001 1005 1007] replaced broker
  webrequest_upload p11: [1006 1004 1001] -> [1007 1004 1001] replaced broker
  webrequest_upload p13: [1002 1005 1006] -> [1002 1005 1013] replaced broker
  webrequest_upload p15: [1001 1006 1005] -> [1001 1007 1005] replaced broker
  webrequest_upload p20: [1006 1001 1004] -> [1007 1001 1004] replaced broker
  webrequest_upload p22: [1002 1015 1006] -> [1002 1015 1009] replaced broker


Changes for partition map chunk 5 for brokers 1005
Partition map changes:
  webrequest_upload p6: [1001 1005 1007] -> [1001 1013 1007] replaced broker
  webrequest_upload p8: [1005 1012 1003] -> [1014 1012 1003] replaced broker
  webrequest_upload p13: [1002 1005 1013] -> [1002 1008 1013] replaced broker
  webrequest_upload p15: [1001 1007 1005] -> [1001 1007 1015] replaced broker
  webrequest_upload p17: [1005 1003 1007] -> [1014 1003 1007] replaced broker
  webrequest_upload p21: [1015 1003 1005] -> [1015 1003 1008] replaced broker


Changes for partition map chunk 6 for brokers 1004
Partition map changes:
  webrequest_upload p0: [1003 1004 1014] -> [1003 1010 1014] replaced broker
  webrequest_upload p1: [1004 1003 1007] -> [1013 1003 1007] replaced broker
  webrequest_upload p4: [1002 1015 1004] -> [1002 1015 1008] replaced broker
  webrequest_upload p10: [1004 1001 1007] -> [1013 1001 1007] replaced broker
  webrequest_upload p11: [1007 1004 1001] -> [1007 1008 1001] replaced broker
  webrequest_upload p16: [1012 1004 1001] -> [1012 1015 1001] replaced broker
  webrequest_upload p19: [1004 1009 1003] -> [1013 1009 1003] replaced broker
  webrequest_upload p20: [1007 1001 1004] -> [1007 1001 1011] replaced broker


Changes for partition map chunk 7 for brokers 1003
Partition map changes:
  webrequest_upload p0: [1003 1010 1014] -> [1008 1010 1014] replaced broker
  webrequest_upload p1: [1013 1003 1007] -> [1013 1012 1007] replaced broker
  webrequest_upload p5: [1009 1007 1003] -> [1009 1007 1013] replaced broker
  webrequest_upload p8: [1014 1012 1003] -> [1014 1012 1009] replaced broker
  webrequest_upload p9: [1003 1002 1011] -> [1008 1002 1011] replaced broker
  webrequest_upload p17: [1014 1003 1007] -> [1014 1011 1007] replaced broker
  webrequest_upload p18: [1003 1002 1010] -> [1008 1002 1010] replaced broker
  webrequest_upload p19: [1013 1009 1003] -> [1013 1009 1012] replaced broker
  webrequest_upload p21: [1015 1003 1008] -> [1015 1010 1008] replaced broker


Changes for partition map chunk 8 for brokers 1002
Partition map changes:
  webrequest_upload p2: [1007 1014 1002] -> [1007 1014 1008] replaced broker
  webrequest_upload p4: [1002 1015 1008] -> [1011 1015 1008] replaced broker
  webrequest_upload p9: [1008 1002 1011] -> [1008 1014 1011] replaced broker
  webrequest_upload p12: [1015 1010 1002] -> [1015 1010 1009] replaced broker
  webrequest_upload p13: [1002 1008 1013] -> [1011 1008 1013] replaced broker
  webrequest_upload p14: [1009 1012 1002] -> [1009 1012 1015] replaced broker
  webrequest_upload p18: [1008 1002 1010] -> [1008 1014 1010] replaced broker
  webrequest_upload p22: [1002 1015 1009] -> [1011 1015 1009] replaced broker
  webrequest_upload p23: [1009 1002 1013] -> [1009 1012 1013] replaced broker


Changes for partition map chunk 9 for brokers 1001
Partition map changes:
  webrequest_upload p3: [1015 1010 1001] -> [1015 1010 1009] replaced broker
  webrequest_upload p6: [1001 1013 1007] -> [1010 1013 1007] replaced broker
  webrequest_upload p7: [1012 1001 1014] -> [1012 1009 1014] replaced broker
  webrequest_upload p10: [1013 1001 1007] -> [1013 1011 1007] replaced broker
  webrequest_upload p11: [1007 1008 1001] -> [1007 1008 1011] replaced broker
  webrequest_upload p15: [1001 1007 1015] -> [1010 1007 1015] replaced broker
  webrequest_upload p16: [1012 1015 1001] -> [1012 1015 1008] replaced broker
  webrequest_upload p20: [1007 1001 1011] -> [1007 1013 1011] replaced broker

WARN:
  [none]

New partition maps:
  webrequest_upload-phase0.json
  webrequest_upload-phase1.json
  webrequest_upload-phase2.json
  webrequest_upload-phase4.json
  webrequest_upload-phase7.json
  webrequest_upload-phase8.json
  webrequest_upload-phase3.json
  webrequest_upload-phase5.json
  webrequest_upload-phase6.json

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase0.json --execute --throttle 30000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase0.json --execute --throttle 30000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1009,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1009,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1006,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1007,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1005,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1005,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1008,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1007,1008,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1006,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1006,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1007,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1008,1007,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1007,1006,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1008,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1006,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1008,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1008],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 30000000 B/s
Successfully started reassignment of partitions.

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase1.json --execute --throttle 30000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase1.json --execute --throttle 30000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1006,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1007,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1005,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1005,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1008,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1007,1008,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1006,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1006,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1007,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1008,1007,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1007,1006,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1008,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1006,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1008,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1008],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 30000000 B/s
Successfully started reassignment of partitions.

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase2.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase2.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1006,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1007,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1005,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1005,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1007,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1007,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1006,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1006,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1007,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1007,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1007,1006,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1006,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

rouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase3.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase3.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1006,1014,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1005,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1005,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1006,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1006,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1015,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1006,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1006,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase4.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase4.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1007,1014,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1005,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1005,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1007,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1005,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1007,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1005,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1015,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1007,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1007,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

MoritzMuehlenhoff subscribed.Oct 10 2023, 6:35 AM

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase5.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase5.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1007,1014,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1004,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1014,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1014,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1007,1004,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1008,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1007,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1004,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1015,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1004,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1007,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1007,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1004,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

brouberol added a subtask: T348495: Upgrade kafka-jumbo100[7-9] to Debian Bullseye.Oct 10 2023, 7:27 AM

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase6.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase6.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1015,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1007,1014,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1013,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1014,1003,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1014,1012,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1003,1002,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1008,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1003,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1007,1001,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1013,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1015,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1013,1009,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1007,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1007,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1003,1010,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_upload-phase7.json --execute --throttle 80000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase7.json --execute --throttle 80000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1015,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1007,1014,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1002,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1013,1012,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1014,1011,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1014,1012,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1008,1002,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1002,1008,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1008,1002,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1007,1001,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1013,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1002,1015,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1013,1009,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1007,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1010,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1007,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1002,1015,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1008,1010,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 80000000 B/s
Successfully started reassignment of partitions.

kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_upload-phase8.json --execute --throttle 80000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_upload","partition":16,"replicas":[1012,1015,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":7,"replicas":[1012,1001,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":2,"replicas":[1007,1014,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":23,"replicas":[1009,1012,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":1,"replicas":[1013,1012,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":17,"replicas":[1014,1011,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":8,"replicas":[1014,1012,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":12,"replicas":[1015,1010,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":18,"replicas":[1008,1014,1010],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":14,"replicas":[1009,1012,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":11,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":13,"replicas":[1011,1008,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":9,"replicas":[1008,1014,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":20,"replicas":[1007,1001,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":10,"replicas":[1013,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":6,"replicas":[1001,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":22,"replicas":[1011,1015,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":3,"replicas":[1015,1010,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":19,"replicas":[1013,1009,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":5,"replicas":[1009,1007,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":21,"replicas":[1015,1010,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":15,"replicas":[1001,1007,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":4,"replicas":[1011,1015,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_upload","partition":0,"replicas":[1008,1010,1014],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 80000000 B/s
Successfully started reassignment of partitions.

We have migrated all topics but webrequest_text which amounts to 20.5TB of data (replication factor included), so about 35% of the cluster size. This will take a bit less than 2 weeks to migrate (in 9 steps, as for webrequest_upload), after which we'll be done for data movement!

brouberol@kafka-jumbo1010:~/topicmappr$ topicmappr rebuild --topics webrequest_text --brokers 1015,1014,1013,1012,1011,1010,1009,1008,1007 --chunk-step-size 1 --force-rebuild | grep -v no-op

Topics:
  webrequest_text

Broker change summary:
  Broker 1004 marked for removal
  Broker 1005 marked for removal
  Broker 1001 marked for removal
  Broker 1002 marked for removal
  Broker 1006 marked for removal
  Broker 1003 marked for removal
  New broker 1013
  New broker 1014
  New broker 1012
  New broker 1010
  New broker 1015
  New broker 1011
  -
  Replacing 6, added 6, missing 0, total count changed by 0

Action:
  Rebuild topic with 6 broker(s) marked for replacement
  Force rebuilding map

Partition map changes:
  webrequest_text p0: [1008 1001 1004] -> [1008 1010 1014] replaced broker
  webrequest_text p1: [1002 1005 1008] -> [1013 1012 1007] replaced broker
  webrequest_text p2: [1007 1009 1002] -> [1007 1014 1008] replaced broker
  webrequest_text p3: [1001 1006 1005] -> [1015 1010 1009] replaced broker
  webrequest_text p4: [1009 1002 1007] -> [1011 1015 1008] replaced broker
  webrequest_text p5: [1005 1008 1001] -> [1009 1007 1013] replaced broker
  webrequest_text p6: [1003 1007 1009] -> [1010 1013 1007] replaced broker
  webrequest_text p7: [1004 1003 1006] -> [1012 1009 1014] replaced broker
  webrequest_text p8: [1006 1001 1007] -> [1014 1012 1009] replaced broker
  webrequest_text p9: [1008 1004 1003] -> [1008 1014 1011] replaced broker
  webrequest_text p10: [1002 1008 1004] -> [1013 1011 1007] replaced broker
  webrequest_text p11: [1007 1002 1009] -> [1007 1008 1011] replaced broker
  webrequest_text p12: [1001 1007 1008] -> [1015 1010 1009] replaced broker
  webrequest_text p13: [1009 1003 1005] -> [1011 1008 1013] replaced broker
  webrequest_text p14: [1005 1009 1002] -> [1009 1012 1015] replaced broker
  webrequest_text p15: [1003 1005 1006] -> [1010 1007 1015] replaced broker
  webrequest_text p16: [1004 1006 1003] -> [1012 1015 1008] replaced broker
  webrequest_text p17: [1006 1002 1007] -> [1014 1011 1007] replaced broker
  webrequest_text p18: [1008 1003 1004] -> [1008 1014 1010] replaced broker
  webrequest_text p19: [1002 1005 1009] -> [1013 1009 1012] replaced broker
  webrequest_text p20: [1007 1008 1001] -> [1007 1013 1011] replaced broker
  webrequest_text p21: [1001 1009 1005] -> [1015 1010 1008] replaced broker
  webrequest_text p22: [1009 1001 1005] -> [1011 1015 1009] replaced broker
  webrequest_text p23: [1005 1006 1002] -> [1009 1012 1013] replaced broker

Broker distribution:
  degree [min/max/avg]: 5/6/5.78 -> 6/8/6.44
  -
  Broker 1007 - leader: 3, follower: 6, total: 9
  Broker 1008 - leader: 3, follower: 6, total: 9
  Broker 1009 - leader: 3, follower: 6, total: 9
  Broker 1010 - leader: 2, follower: 5, total: 7
  Broker 1011 - leader: 3, follower: 5, total: 8
  Broker 1012 - leader: 2, follower: 5, total: 7
  Broker 1013 - leader: 3, follower: 5, total: 8
  Broker 1014 - leader: 2, follower: 5, total: 7
  Broker 1015 - leader: 3, follower: 5, total: 8


Generating reassignments in chunks 1 brokers at a time: 



Skipping noop map output for brokers 1015

Skipping noop map output for brokers 1014

Skipping noop map output for brokers 1013

Skipping noop map output for brokers 1012

Skipping noop map output for brokers 1011

Skipping noop map output for brokers 1010

Changes for partition map chunk 1 for brokers 1009
Partition map changes:
  webrequest_text p2: [1007 1009 1002] -> [1007 1014 1002] replaced broker
  webrequest_text p4: [1009 1002 1007] -> [1011 1002 1007] replaced broker
  webrequest_text p6: [1003 1007 1009] -> [1003 1007 1007] replaced broker
  webrequest_text p11: [1007 1002 1009] -> [1007 1002 1011] replaced broker
  webrequest_text p13: [1009 1003 1005] -> [1011 1003 1005] replaced broker
  webrequest_text p14: [1005 1009 1002] -> [1005 1012 1002] replaced broker
  webrequest_text p19: [1002 1005 1009] -> [1002 1005 1012] replaced broker
  webrequest_text p21: [1001 1009 1005] -> [1001 1010 1005] replaced broker
  webrequest_text p22: [1009 1001 1005] -> [1011 1001 1005] replaced broker


Changes for partition map chunk 2 for brokers 1008
Partition map changes:
  webrequest_text p1: [1002 1005 1008] -> [1002 1005 1007] replaced broker
  webrequest_text p5: [1005 1008 1001] -> [1005 1007 1001] replaced broker
  webrequest_text p10: [1002 1008 1004] -> [1002 1011 1004] replaced broker
  webrequest_text p12: [1001 1007 1008] -> [1001 1007 1009] replaced broker
  webrequest_text p20: [1007 1008 1001] -> [1007 1013 1001] replaced broker


Changes for partition map chunk 3 for brokers 1007
Partition map changes:
  webrequest_text p4: [1011 1002 1007] -> [1011 1002 1008] replaced broker
  webrequest_text p6: [1003 1007 1007] -> [1003 1013 1007] replaced broker
  webrequest_text p8: [1006 1001 1007] -> [1006 1001 1009] replaced broker
  webrequest_text p12: [1001 1007 1009] -> [1001 1010 1009] replaced broker


Changes for partition map chunk 4 for brokers 1006
Partition map changes:
  webrequest_text p3: [1001 1006 1005] -> [1001 1010 1005] replaced broker
  webrequest_text p7: [1004 1003 1006] -> [1004 1003 1014] replaced broker
  webrequest_text p8: [1006 1001 1009] -> [1014 1001 1009] replaced broker
  webrequest_text p15: [1003 1005 1006] -> [1003 1005 1015] replaced broker
  webrequest_text p16: [1004 1006 1003] -> [1004 1015 1003] replaced broker
  webrequest_text p17: [1006 1002 1007] -> [1014 1002 1007] replaced broker
  webrequest_text p23: [1005 1006 1002] -> [1005 1012 1002] replaced broker


Changes for partition map chunk 5 for brokers 1005
Partition map changes:
  webrequest_text p1: [1002 1005 1007] -> [1002 1012 1007] replaced broker
  webrequest_text p3: [1001 1010 1005] -> [1001 1010 1009] replaced broker
  webrequest_text p5: [1005 1007 1001] -> [1009 1007 1001] replaced broker
  webrequest_text p13: [1011 1003 1005] -> [1011 1003 1013] replaced broker
  webrequest_text p14: [1005 1012 1002] -> [1009 1012 1002] replaced broker
  webrequest_text p15: [1003 1005 1015] -> [1003 1007 1015] replaced broker
  webrequest_text p19: [1002 1005 1012] -> [1002 1009 1012] replaced broker
  webrequest_text p21: [1001 1010 1005] -> [1001 1010 1008] replaced broker
  webrequest_text p22: [1011 1001 1005] -> [1011 1001 1009] replaced broker
  webrequest_text p23: [1005 1012 1002] -> [1009 1012 1002] replaced broker


Changes for partition map chunk 6 for brokers 1004
Partition map changes:
  webrequest_text p0: [1008 1001 1004] -> [1008 1001 1014] replaced broker
  webrequest_text p7: [1004 1003 1014] -> [1012 1003 1014] replaced broker
  webrequest_text p9: [1008 1004 1003] -> [1008 1014 1003] replaced broker
  webrequest_text p10: [1002 1011 1004] -> [1002 1011 1007] replaced broker
  webrequest_text p16: [1004 1015 1003] -> [1012 1015 1003] replaced broker
  webrequest_text p18: [1008 1003 1004] -> [1008 1003 1010] replaced broker


Changes for partition map chunk 7 for brokers 1003
Partition map changes:
  webrequest_text p6: [1003 1013 1007] -> [1010 1013 1007] replaced broker
  webrequest_text p7: [1012 1003 1014] -> [1012 1009 1014] replaced broker
  webrequest_text p9: [1008 1014 1003] -> [1008 1014 1011] replaced broker
  webrequest_text p13: [1011 1003 1013] -> [1011 1008 1013] replaced broker
  webrequest_text p15: [1003 1007 1015] -> [1010 1007 1015] replaced broker
  webrequest_text p16: [1012 1015 1003] -> [1012 1015 1008] replaced broker
  webrequest_text p18: [1008 1003 1010] -> [1008 1014 1010] replaced broker


Changes for partition map chunk 8 for brokers 1002
Partition map changes:
  webrequest_text p1: [1002 1012 1007] -> [1013 1012 1007] replaced broker
  webrequest_text p2: [1007 1014 1002] -> [1007 1014 1008] replaced broker
  webrequest_text p4: [1011 1002 1008] -> [1011 1015 1008] replaced broker
  webrequest_text p10: [1002 1011 1007] -> [1013 1011 1007] replaced broker
  webrequest_text p11: [1007 1002 1011] -> [1007 1008 1011] replaced broker
  webrequest_text p14: [1009 1012 1002] -> [1009 1012 1015] replaced broker
  webrequest_text p17: [1014 1002 1007] -> [1014 1011 1007] replaced broker
  webrequest_text p19: [1002 1009 1012] -> [1013 1009 1012] replaced broker
  webrequest_text p23: [1009 1012 1002] -> [1009 1012 1013] replaced broker


Changes for partition map chunk 9 for brokers 1001
Partition map changes:
  webrequest_text p0: [1008 1001 1014] -> [1008 1010 1014] replaced broker
  webrequest_text p3: [1001 1010 1009] -> [1015 1010 1009] replaced broker
  webrequest_text p5: [1009 1007 1001] -> [1009 1007 1013] replaced broker
  webrequest_text p8: [1014 1001 1009] -> [1014 1012 1009] replaced broker
  webrequest_text p12: [1001 1010 1009] -> [1015 1010 1009] replaced broker
  webrequest_text p20: [1007 1013 1001] -> [1007 1013 1011] replaced broker
  webrequest_text p21: [1001 1010 1008] -> [1015 1010 1008] replaced broker
  webrequest_text p22: [1011 1001 1009] -> [1011 1015 1009] replaced broker

WARN:
  [none]

New partition maps:
  webrequest_text-phase4.json
  webrequest_text-phase7.json
  webrequest_text-phase3.json
  webrequest_text-phase5.json
  webrequest_text-phase6.json
  webrequest_text-phase8.json
  webrequest_text-phase0.json
  webrequest_text-phase1.json
  webrequest_text-phase2.json

I'm using --force-rebuild there, as I'd rather we get a homogeneous amount of partitions / broker, as this will widely decrease skew in storage usage / broker, and thus skew in network traffic / broker.

I found a bug in topicmappr as the chunked force rebuild cause a broker to be listed multiple times among the replica list:

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_text-phase0.json --execute --throttle 80000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_text-phase0.json --execute --throttle 80000000
Partitions reassignment failed due to Partition replica lists may not contain duplicate entries: webrequest_text-6 contains multiple entries for 1007
kafka.common.AdminCommandFailedException: Partition replica lists may not contain duplicate entries: webrequest_text-6 contains multiple entries for 1007
	at kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:290)
	at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:203)
	at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:199)
	at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:62)
	at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

I'm then dropping the --force-rebuild flag, even if it will lead to a less optimal placement at the end. We can always correct with a rebalancing, should we need it.

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_text-phase0.json --execute --throttle 80000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_text-phase0.json --execute --throttle 80000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_text","partition":12,"replicas":[1001,1007,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":23,"replicas":[1005,1006,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":15,"replicas":[1003,1005,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":2,"replicas":[1007,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":20,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":6,"replicas":[1003,1007,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":5,"replicas":[1005,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":22,"replicas":[1009,1001,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":14,"replicas":[1005,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":19,"replicas":[1002,1005,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":11,"replicas":[1007,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":7,"replicas":[1004,1003,1006],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":1,"replicas":[1002,1005,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":4,"replicas":[1009,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":16,"replicas":[1004,1006,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":8,"replicas":[1006,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":17,"replicas":[1006,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":9,"replicas":[1008,1004,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":10,"replicas":[1002,1008,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":13,"replicas":[1009,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":0,"replicas":[1008,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":18,"replicas":[1008,1003,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":21,"replicas":[1001,1009,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":3,"replicas":[1001,1006,1005],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 80000000 B/s
Successfully started reassignment of partitions.

In the meantime, I've reported the issue upstream: https://github.com/DataDog/kafka-kit/issues/432

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_text-phase1.json --execute --throttle 80000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_text-phase1.json --execute --throttle 80000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_text","partition":12,"replicas":[1001,1007,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":23,"replicas":[1005,1015,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":15,"replicas":[1003,1005,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":2,"replicas":[1007,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":20,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":6,"replicas":[1003,1007,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":5,"replicas":[1005,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":22,"replicas":[1009,1001,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":14,"replicas":[1005,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":19,"replicas":[1002,1005,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":11,"replicas":[1007,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":7,"replicas":[1004,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":1,"replicas":[1002,1005,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":4,"replicas":[1009,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":16,"replicas":[1004,1010,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":8,"replicas":[1014,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":17,"replicas":[1014,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":9,"replicas":[1008,1004,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":10,"replicas":[1002,1008,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":13,"replicas":[1009,1003,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":0,"replicas":[1008,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":18,"replicas":[1008,1003,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":21,"replicas":[1001,1009,1005],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":3,"replicas":[1001,1014,1005],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 80000000 B/s
Successfully started reassignment of partitions.

brouberol mentioned this in T346425: Reassign partitions away from kafka-jumbo100[1-6] to kafka-jumbo10[07-15] brokers.Oct 11 2023, 1:15 PM

Change 965159 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Change the kafka-jumbo bootstrap host for karapace

https://gerrit.wikimedia.org/r/965159

Change 965160 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Change the kafka-jumbo bootstrap host for the analytics cluster hosts

https://gerrit.wikimedia.org/r/965160

Change 965161 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Change the underlying host for the kafka-jumbo-canary cumin alias

https://gerrit.wikimedia.org/r/965161

Change 965162 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Remove kafka-jumbo100[1-6] from puppet

https://gerrit.wikimedia.org/r/965162

Change 965164 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Remove kafka-jumbo100[1-6] brokers from bootstrap hosts

https://gerrit.wikimedia.org/r/965164

Change 965166 had a related patch set uploaded (by Brouberol; author: Brouberol):

[analytics/refinery@master] Remove kafka-jumbo100[1-6] brokers from bootstrap hosts

https://gerrit.wikimedia.org/r/965166

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_text-phase2.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_text-phase2.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_text","partition":12,"replicas":[1001,1007,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":23,"replicas":[1010,1015,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":15,"replicas":[1003,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":2,"replicas":[1007,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":20,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":6,"replicas":[1003,1007,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":5,"replicas":[1010,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":22,"replicas":[1009,1001,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":14,"replicas":[1010,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":19,"replicas":[1002,1012,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":11,"replicas":[1007,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":7,"replicas":[1004,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":1,"replicas":[1002,1012,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":4,"replicas":[1009,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":16,"replicas":[1004,1010,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":8,"replicas":[1014,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":17,"replicas":[1014,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":9,"replicas":[1008,1004,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":10,"replicas":[1002,1008,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":13,"replicas":[1009,1003,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":0,"replicas":[1008,1001,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":18,"replicas":[1008,1003,1004],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":21,"replicas":[1001,1009,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":3,"replicas":[1001,1014,1007],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Warning: You must run Verify periodically, until the reassignment completes, to ensure the throttle is removed. You can also alter the throttle by rerunning the Execute command passing a new value.
The inter-broker throttle limit was set to 60000000 B/s
Successfully started reassignment of partitions.

Change 965159 merged by Brouberol:

[operations/puppet@production] Change the kafka-jumbo bootstrap host for karapace

https://gerrit.wikimedia.org/r/965159

Change 965160 merged by Brouberol:

[operations/puppet@production] Change the kafka-jumbo bootstrap host for the analytics cluster hosts

https://gerrit.wikimedia.org/r/965160

Change 965161 merged by Brouberol:

[operations/puppet@production] Change the underlying host for the kafka-jumbo-canary cumin alias

https://gerrit.wikimedia.org/r/965161

brouberol@kafka-jumbo1010:~/topicmappr$ kafka reassign-partitions --reassignment-json-file ./webrequest_text-phase3.json --execute --throttle 60000000
kafka-reassign-partitions --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/jumbo-eqiad --reassignment-json-file ./webrequest_text-phase3.json --execute --throttle 60000000
Current partition replica assignment

{"version":1,"partitions":[{"topic":"webrequest_text","partition":12,"replicas":[1001,1007,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":23,"replicas":[1010,1015,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":15,"replicas":[1003,1013,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":2,"replicas":[1007,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":20,"replicas":[1007,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":6,"replicas":[1003,1007,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":5,"replicas":[1010,1008,1001],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":22,"replicas":[1009,1001,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":14,"replicas":[1010,1009,1002],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":19,"replicas":[1002,1012,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":11,"replicas":[1007,1002,1009],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":7,"replicas":[1013,1003,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":1,"replicas":[1002,1012,1008],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":4,"replicas":[1009,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":16,"replicas":[1013,1010,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":8,"replicas":[1014,1001,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":17,"replicas":[1014,1002,1007],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":9,"replicas":[1008,1015,1003],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":10,"replicas":[1002,1008,1011],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":13,"replicas":[1009,1003,1012],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":0,"replicas":[1008,1001,1013],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":18,"replicas":[1008,1003,1015],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":21,"replicas":[1001,1009,1014],"log_dirs":["any","any","any"]},{"topic":"webrequest_text","partition":3,"replicas":[1001,1014,1007],"log_dirs":["any","any","any"]}]}

Change 965166 merged by Brouberol:

[analytics/refinery@master] Remove kafka-jumbo100[1-6] brokers from bootstrap hosts

https://gerrit.wikimedia.org/r/965166

Change 965164 merged by Brouberol:

[operations/deployment-charts@master] Remove kafka-jumbo100[1-6] brokers from bootstrap hosts

https://gerrit.wikimedia.org/r/965164

Mentioned in SAL (#wikimedia-analytics) [2023-10-16T08:34:25Z] <brouberol> redeploying eventstreams-internal with the new kafka broker list T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-16T08:38:19Z] <brouberol> redeploying eventgate-analytics with the new kafka broker list T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-16T08:42:03Z] <brouberol> redeploying eventgate-analytics-external with the new kafka broker list T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-16T08:49:12Z] <brouberol> redeploying datahub with the new kafka broker list T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-16T13:05:51Z] <brouberol> deploying mw-page-content-change-enrich with the new kafka broker list T336044

brouberol changed the status of subtask T348495: Upgrade kafka-jumbo100[7-9] to Debian Bullseye from Open to In Progress.Oct 16 2023, 1:59 PM

Change 965162 merged by Brouberol:

[operations/puppet@production] Remove kafka-jumbo100[1-6] from puppet

https://gerrit.wikimedia.org/r/965162

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2023, 8:10 AM

Change 966497 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Remove kafka-jumbo100[1-6] brokers fron the inventory

https://gerrit.wikimedia.org/r/966497

gerritbot added a project: Patch-For-Review.Oct 17 2023, 9:37 AM

Jgreen mentioned this in T349249: Improve fundraising kafkatee monitoring.Oct 18 2023, 9:09 PM

Jgreen mentioned this in T349253: Backfill logs for 2023-10-11 to 2023-11-18 into fundraising banner stats..Oct 18 2023, 9:42 PM

Change 966497 merged by Brouberol:

[operations/puppet@production] Remove kafka-jumbo100[1-6] brokers from the inventory

https://gerrit.wikimedia.org/r/966497

Maintenance_bot removed a project: Patch-For-Review.Oct 19 2023, 12:30 PM

The kafka services on kafka-jumbo100[1-6] are now unreachable from the fleet.

Mentioned in SAL (#wikimedia-analytics) [2023-10-19T15:02:01Z] <brouberol> disabling puppet on kafka-jumbo100[1-6] to make sure kafka isn't resarted - T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-19T15:04:13Z] <brouberol> sudo cumin --batch-size 1 --batch-sleep 60 'kafka-jumbo100[1-6].eqiad.wmnet' 'sudo systemctl stop kafka.service' - T336044

Mentioned in SAL (#wikimedia-analytics) [2023-10-19T15:22:13Z] <brouberol> The kafka service has been stopped on kafka-jumbo100[1-6] - T336044

Change 967240 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Drop kafka-jumbo100[1-6].eqiad.wmnet from the puppet site

https://gerrit.wikimedia.org/r/967240

gerritbot added a project: Patch-For-Review.Oct 19 2023, 3:52 PM

Change 967400 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Replace the IPs of kafka-jumbo100[1-6] with kafka-jumbo101[0-5]

https://gerrit.wikimedia.org/r/967400

Change 967400 merged by jenkins-bot:

[operations/deployment-charts@master] Replace the IPs of kafka-jumbo100[1-6] with kafka-jumbo101[0-5]

https://gerrit.wikimedia.org/r/967400

Change 967470 had a related patch set uploaded (by Brouberol; author: Brouberol):

[wikidata/query/rdf@master] Reflect the fact that kafka-jumbo1001.eqiad.wmnet has been decommissioned

https://gerrit.wikimedia.org/r/967470

Change 967470 merged by jenkins-bot:

[wikidata/query/rdf@master] Reflect the fact that kafka-jumbo1001.eqiad.wmnet has been decommissioned

https://gerrit.wikimedia.org/r/967470

sudo cookbook sre.hosts.decommission -t T336044 kafka-jumbo1006.eqiad.wmnet

cookbooks.sre.hosts.decommission executed by brouberol@cumin1001 for hosts: kafka-jumbo1006.eqiad.wmnet

kafka-jumbo1006.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB

Mentioned in SAL (#wikimedia-analytics) [2023-10-23T08:28:51Z] <brouberol> sudo cookbook sre.hosts.decommission -t T336044 kafka-jumbo1005.eqiad.wmnet - T336044

cookbooks.sre.hosts.decommission executed by brouberol@cumin1001 for hosts: kafka-jumbo1005.eqiad.wmnet

kafka-jumbo1005.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB

Mentioned in SAL (#wikimedia-analytics) [2023-10-23T08:53:19Z] <brouberol> sudo cookbook sre.hosts.decommission -t T336044 kafka-jumbo1004.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-10-23T09:35:05Z] <brouberol@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-jumbo1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin1001 - brouberol@cumin1001 - T336044"

Mentioned in SAL (#wikimedia-operations) [2023-10-23T09:36:08Z] <brouberol@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-jumbo1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin1001 - brouberol@cumin1001 - T336044"

Cookbook cookbooks.sre.debmonitor.remove-hosts run by brouberol: for 1 hosts: kafka-jumbo1004.eqiad.wmnet

cookbooks.sre.hosts.decommission executed by brouberol@cumin1001 for hosts: kafka-jumbo1003.eqiad.wmnet

kafka-jumbo1003.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB

Mentioned in SAL (#wikimedia-analytics) [2023-10-23T09:58:05Z] <brouberol> sudo cookbook sre.hosts.decommission -t T336044 kafka-jumbo1002.eqiad.wmnet

cookbooks.sre.hosts.decommission executed by brouberol@cumin1001 for hosts: kafka-jumbo1002.eqiad.wmnet

kafka-jumbo1002.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB

Mentioned in SAL (#wikimedia-analytics) [2023-10-23T10:14:09Z] <brouberol> sudo cookbook sre.hosts.decommission -t T336044 kafka-jumbo1001.eqiad.wmnet

cookbooks.sre.hosts.decommission executed by brouberol@cumin1001 for hosts: kafka-jumbo1001.eqiad.wmnet

kafka-jumbo1001.eqiad.wmnet (PASS)
- Downtimed host on Icinga/Alertmanager
- Found physical host
- Downtimed management interface on Alertmanager
- Wiped all swraid, partition-table and filesystem signatures
- Powered off
- [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
- Configured the linked switch interface(s)
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB

brouberol moved this task from In Progress to Needs Review on the Data-Platform-SRE board.Oct 23 2023, 11:26 AM

Change 967240 merged by Brouberol:

[operations/puppet@production] Drop kafka-jumbo100[1-6].eqiad.wmnet from the puppet site

https://gerrit.wikimedia.org/r/967240

brouberol closed this task as Resolved.Oct 23 2023, 12:42 PM

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2023, 1:11 PM

brouberol closed subtask T348495: Upgrade kafka-jumbo100[7-9] to Debian Bullseye as Resolved.Oct 26 2023, 9:26 AM

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE board.Nov 6 2023, 1:36 PM

BTullis merged a task: T352759: Decommission kafka-jumbo100[1-6].Dec 5 2023, 12:35 PM

	F37829664: reassignment-a-to-d-phase0.json
	Sep 28 2023, 1:50 PM

	F37998836: image.png
	Oct 6 2023, 12:33 PM

	F37998807: image.png
	Oct 6 2023, 12:33 PM

	F37998774: image.png
	Oct 6 2023, 12:33 PM

	F37998747: image.png
	Oct 6 2023, 12:33 PM

Decommission kafka-jumbo100[1-6]Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Decommission kafka-jumbo100[1-6]
Closed, ResolvedPublic
Actions

Related Objects
Search...