Page MenuHomePhabricator

commonswiki shard size grew more than 50G in eqiad and codfw
Closed, ResolvedPublic

Description

There are two alerts in icinga right now:

CRIT - commonswiki_content_1582399079(65gb) - eqiad 9243
WARN - commonswiki_content_1582512959(50.666666666666664gb) - codfw 9243

The commonswiki index might need more shards?

Acked alerts in icinga with this task as reference.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 576659 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] use 2 shards for commonswiki_content

https://gerrit.wikimedia.org/r/576659

dcausse triaged this task as Medium priority.Mar 4 2020, 11:00 AM
dcausse moved this task from needs triage to Current work on the Discovery-Search board.
dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.

Change 576659 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] use 2 shards for commonswiki_content

https://gerrit.wikimedia.org/r/576659

There is already a global re-index ongoing, which is still going to take some time (days at least). Once this is done, we can reindex commonswiki.

Acked this morning warnings in icinga for commonswiki_content shard size for cloudelastic.

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:42:20Z] <dcausse> elastic (search cluster) reindex commmonswiki_content on cloudelastic (T246882)

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:43:10Z] <dcausse> errata: elastic (search cluster) reindexing commonswiki_content on cloudelastic (T246882)

Mentioned in SAL (#wikimedia-operations) [2020-04-16T14:14:48Z] <dcausse> elastic (search cluster) reindexing commonswiki_content in codfw and ediad (T246882)

TJones claimed this task.