Page MenuHomePhabricator

commonswiki shard size grew more than 50G in eqiad and codfw
Closed, ResolvedPublic

Description

There are two alerts in icinga right now:

CRIT - commonswiki_content_1582399079(65gb) - eqiad 9243
WARN - commonswiki_content_1582512959(50.666666666666664gb) - codfw 9243

The commonswiki index might need more shards?

Acked alerts in icinga with this task as reference.

Event Timeline

elukey created this task.Mar 4 2020, 10:49 AM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 4 2020, 10:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 576659 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [cirrus] use 2 shards for commonswiki_content

https://gerrit.wikimedia.org/r/576659

dcausse triaged this task as Medium priority.Mar 4 2020, 11:00 AM
dcausse moved this task from needs triage to Current work on the Discovery-Search board.
dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.

Change 576659 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] use 2 shards for commonswiki_content

https://gerrit.wikimedia.org/r/576659

Gehel added a comment.Mar 5 2020, 1:35 PM

There is already a global re-index ongoing, which is still going to take some time (days at least). Once this is done, we can reindex commonswiki.

elukey added a comment.Apr 6 2020, 6:37 AM

Acked this morning warnings in icinga for commonswiki_content shard size for cloudelastic.

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:42:20Z] <dcausse> elastic (search cluster) reindex commmonswiki_content on cloudelastic (T246882)

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:43:10Z] <dcausse> errata: elastic (search cluster) reindexing commonswiki_content on cloudelastic (T246882)

Mentioned in SAL (#wikimedia-operations) [2020-04-16T14:14:48Z] <dcausse> elastic (search cluster) reindexing commonswiki_content in codfw and ediad (T246882)

TJones closed this task as Resolved.Apr 29 2020, 2:37 PM
TJones claimed this task.