Page MenuHomePhabricator
Create Task
Maniphest T316711

Reduce shard count on all wikis in beta cluster to 1
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

Beta cluster is using the production shard counts but has trivial amounts of data. Currently there are 434 shards and 184 indices in beta-cluster, we could bring that down to 184 shards and save a bit of memory pressure on elasticsearch.

Details

SubjectRepoBranchLines +/-
beta: Set shard count for commonswiki_file to 1operations/mediawiki-configmaster+1 -1
wmgCirrusSearchShardCount: Override prod settings for beta clusteroperations/mediawiki-configmaster+1 -1
cirrus: Limit shard count to 1 in deployment-prepoperations/mediawiki-configmaster+6 -0
Customize query in gerrit

Event Timeline

EBernhardson created this task.Aug 30 2022, 7:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2022, 7:46 PM
Gehel triaged this task as High priority.Sep 5 2022, 3:24 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel set the point value for this task to 2.Sep 5 2022, 3:41 PM
Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
gerritbot added a comment.Sep 20 2022, 10:30 PM

Change 833463 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Limit shard count to 1 in deployment-prep

https://gerrit.wikimedia.org/r/833463

gerritbot added a project: Patch-For-Review.Sep 20 2022, 10:30 PM
EBernhardson claimed this task.Sep 20 2022, 10:34 PM
EBernhardson moved this task from Ready for Dev -- SWE to To Be Deployed on the Discovery-Search (Current work) board.
gerritbot added a comment.Sep 21 2022, 8:20 PM

Change 833463 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Limit shard count to 1 in deployment-prep

https://gerrit.wikimedia.org/r/833463

Stashbot added a comment.Sep 21 2022, 8:20 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:20:49Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:833463|cirrus: Limit shard count to 1 in deployment-prep (T316711)]]

Stashbot added a comment.Sep 21 2022, 8:21 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:21:12Z] <samtar@deploy1002> samtar and ebernhardson: Backport for [[gerrit:833463|cirrus: Limit shard count to 1 in deployment-prep (T316711)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Stashbot added a comment.Sep 21 2022, 8:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:25:08Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:833463|cirrus: Limit shard count to 1 in deployment-prep (T316711)]] (duration: 04m 19s)

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 8:30 PM
EBernhardson added a comment.Sep 27 2022, 10:14 PM

started reindex from deployment-mwmaint02

EBernhardson added a comment.Sep 28 2022, 7:08 PM

reindexing finished, but a variety of indices still have more shards than expected. Will need to review why the config change didn't do as I expected.

gerritbot added a comment.Sep 28 2022, 9:58 PM

Change 836301 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] wmgCirrusSearchShardCount: Override prod settings for beta cluster

https://gerrit.wikimedia.org/r/836301

gerritbot added a project: Patch-For-Review.Sep 28 2022, 9:58 PM
gerritbot added a comment.Sep 29 2022, 8:49 PM

Change 836301 merged by jenkins-bot:

[operations/mediawiki-config@master] wmgCirrusSearchShardCount: Override prod settings for beta cluster

https://gerrit.wikimedia.org/r/836301

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2022, 9:30 PM
EBernhardson added a comment.Oct 3 2022, 5:22 PM

started up reindexing on deployment-prep again, after verifying that $wgCirrusSearchShardCount is appropriately set via shell.php

gerritbot added a comment.Oct 4 2022, 9:29 PM

Change 838272 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] beta: Set shard count for commonswiki_file to 1

https://gerrit.wikimedia.org/r/838272

gerritbot added a project: Patch-For-Review.Oct 4 2022, 9:29 PM
EBernhardson added a comment.Oct 4 2022, 9:29 PM

This is 99% of the way there, only remaining index is commonswiki_file. When applying the new shard settings I failed to set an explicit value for the file index suffix, which only exists on commonswiki, which results in cirrus refusing to create a new index there. Once the next patch is shipped we should be able to close this out.

gerritbot added a comment.Oct 13 2022, 8:14 PM

Change 838272 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Set shard count for commonswiki_file to 1

https://gerrit.wikimedia.org/r/838272

Maintenance_bot removed a project: Patch-For-Review.Oct 13 2022, 8:31 PM
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Oct 17 2022, 5:58 PM

commonswiki_file is now complete as well. Test reports no indices with multiple shards:

curl -s https://deployment-elastic09.deployment-prep.eqiad1.wikimedia.cloud:9243/_cat/indices | awk '$5 > 1 { print $0 }'
Gehel closed this task as Resolved.Nov 7 2022, 3:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits