Page MenuHomePhabricator

Unallocated shards on elasticsearch codfw cluster after switch failure
Closed, ResolvedPublic5 Estimated Story Points

Description

Due to the failure of a switch (T267865) 3 elasticsearch nodes are currently unavailable (elastic2048,elastic2049,elastic2059). While most shards have moved to other nodes, some enwiki_content shards are still unallocated (see below). This has some impact on the throughput and the resilience of that index.

Failed allocation might be transient and we might be able to recover with the related cookbook. Or depending on how long we expect those nodes to be down, we might need to review the sharding of enwiki.

gehel@elastic2040:~$ curl -s localhost:9200/_cat/shards | grep -v STARTED
enwiki_content_1605226908              7  r UNASSIGNED                                 
enwiki_content_1605226908              13 r UNASSIGNED                                 
enwiki_content_1605226908              6  r UNASSIGNED                                 
enwiki_content_1605226908              12 r UNASSIGNED                                 
enwiki_content_1603009218              8  r UNASSIGNED                                 
enwiki_content_1603009218              6  r UNASSIGNED                                 
enwiki_content_1603009218              2  r UNASSIGNED                                 
enwiki_content_1603009218              1  r UNASSIGNED                                 
enwiki_content_1594735844              11 r UNASSIGNED                                 
enwiki_content_1594735844              9  r UNASSIGNED                                 
enwiki_content_1594735844              15 r UNASSIGNED                                 
enwiki_content_1594735844              2  r UNASSIGNED

Event Timeline

Gehel created this task.Nov 16 2020, 9:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2020, 9:41 AM
Gehel added a comment.Nov 16 2020, 6:36 PM

Note that we have 3 different indices for enwiki_content, this looks like 2 stale indices. This might require some cleanup and investigation.

CBogen set the point value for this task to 5.Nov 16 2020, 6:37 PM

Per https://wikitech.wikimedia.org/wiki/Search#Removing_Duplicate_Indices, stale indices were removed like so:

curl -XDELETE https://search.svc.eqiad.wmnet:9243/ruwikinews_general_1605705596
curl -XDELETE https://search.svc.eqiad.wmnet:9243/metawiki_content_1605567957
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikibooks_content_1605260916
curl -XDELETE https://search.svc.eqiad.wmnet:9243/incubatorwiki_content_1603175850
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikivoyage_content_1605271713
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwiktionary_content_1605498388
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwikiquote_content_1605496662
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwiki_content_1605494987
curl -XDELETE https://search.svc.eqiad.wmnet:9243/commonswiki_content_1602933054
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwiki_content_1605260235
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwikiquote_content_1603180759
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwiktionary_content_1605272077
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikinews_general_1602981266
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikisource_content_1605269574
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwiktionary_content_1603181652
curl -XDELETE https://search.svc.eqiad.wmnet:9243/wikidatawiki_content_1603716445
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwiki_content_1602979688
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwikisource_content_1605496785
curl -XDELETE https://search.svc.eqiad.wmnet:9243/simplewiki_content_1603465265
curl -XDELETE https://search.svc.eqiad.wmnet:9243/shwiki_content_1605722294
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikiversity_content_1605271680
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwiki_content_1603178591
curl -XDELETE https://search.svc.eqiad.wmnet:9243/sourceswiki_content_1603476089
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikivoyage_content_1602989710
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikinews_general_1605261729
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikiquote_content_1605268250
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikibooks_content_1602980485
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikisource_content_1602988013
curl -XDELETE https://search.svc.eqiad.wmnet:9243/mediawikiwiki_content_1603243887
curl -XDELETE https://search.svc.eqiad.wmnet:9243/mediawikiwiki_content_1605567894
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwiktionary_content_1602990045
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikiquote_content_1602987377
curl -XDELETE https://search.svc.eqiad.wmnet:9243/enwikiversity_content_1602989675
curl -XDELETE https://search.svc.eqiad.wmnet:9243/commonswiki_content_1605207170
curl -XDELETE https://search.svc.eqiad.wmnet:9243/itwikisource_content_1603180854
curl -XDELETE https://search.svc.eqiad.wmnet:9243/incubatorwiki_content_1605492086
curl -XDELETE https://search.svc.eqiad.wmnet:9243/specieswiki_content_1603476358
curl -XDELETE https://search.svc.eqiad.wmnet:9243/ruwiktionary_content_1605712311
curl -XDELETE https://search.svc.eqiad.wmnet:9243/metawiki_content_1603243906
curl -XDELETE https://search.svc.eqiad.wmnet:9443/collabwiki_general_1605207161
curl -XDELETE https://search.svc.eqiad.wmnet:9443/knwiktionary_content_1583664130
curl -XDELETE https://search.svc.eqiad.wmnet:9443/id_internalwikimedia_general_1583493108
curl -XDELETE https://search.svc.eqiad.wmnet:9443/ombudsmenwiki_general_1603299316
curl -XDELETE https://search.svc.eqiad.wmnet:9443/wikimania2007wiki_content_1603716799
curl -XDELETE https://search.svc.eqiad.wmnet:9443/itwikinews_general_1605496658
curl -XDELETE https://search.svc.eqiad.wmnet:9443/testwikidatawiki_content_1603608580
curl -XDELETE https://search.svc.eqiad.wmnet:9443/collabwiki_general_1602933047
curl -XDELETE https://search.svc.eqiad.wmnet:9443/betawikiversity_content_1602850839
curl -XDELETE https://search.svc.eqiad.wmnet:9443/ombudsmenwiki_general_1605625669
curl -XDELETE https://search.svc.eqiad.wmnet:9443/legalteamwiki_content_1603233776
curl -XDELETE https://search.svc.eqiad.wmnet:9443/itwikinews_general_1603180754
curl -XDELETE https://search.svc.eqiad.wmnet:9443/testwiki_content_1603608569
curl -XDELETE https://search.svc.eqiad.wmnet:9443/betawikiversity_content_1605109807
curl -XDELETE https://search.svc.eqiad.wmnet:9443/qualitywiki_general_1605673767
curl -XDELETE https://search.svc.eqiad.wmnet:9443/test2wiki_content_1603608367
curl -XDELETE https://search.svc.eqiad.wmnet:9443/qualitywiki_general_1603340991
curl -XDELETE https://search.svc.eqiad.wmnet:9443/legalteamwiki_content_1605556563
curl -XDELETE https://search.svc.eqiad.wmnet:9643/itwikiversity_content_1603181491
curl -XDELETE https://search.svc.eqiad.wmnet:9643/officewiki_content_1603299114
curl -XDELETE https://search.svc.eqiad.wmnet:9643/outreachwiki_content_1605627437
curl -XDELETE https://search.svc.eqiad.wmnet:9643/labswiki_general_1603229687
curl -XDELETE https://search.svc.eqiad.wmnet:9643/stewardwiki_content_1603580805
curl -XDELETE https://search.svc.eqiad.wmnet:9643/itwikivoyage_content_1605498295
curl -XDELETE https://search.svc.eqiad.wmnet:9643/simplewikibooks_general_1603465337
curl -XDELETE https://search.svc.eqiad.wmnet:9643/checkuserwiki_content_1602931356
curl -XDELETE https://search.svc.eqiad.wmnet:9643/officewiki_content_1605625439
curl -XDELETE https://search.svc.eqiad.wmnet:9643/mrwikisource_content_1583851179
curl -XDELETE https://search.svc.eqiad.wmnet:9643/checkuserwiki_content_1605205361
curl -XDELETE https://search.svc.eqiad.wmnet:9643/itwikivoyage_content_1603181527
curl -XDELETE https://search.svc.eqiad.wmnet:9643/usabilitywiki_general_1603660900
curl -XDELETE https://search.svc.eqiad.wmnet:9643/labswiki_general_1605551826
curl -XDELETE https://search.svc.eqiad.wmnet:9643/outreachwiki_content_1603300949
curl -XDELETE https://search.svc.eqiad.wmnet:9643/itwikiversity_content_1605498141
curl -XDELETE https://search.svc.codfw.wmnet:9243/commonswiki_content_1602956414
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikiversity_content_1605235928
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikiquote_content_1603017339
curl -XDELETE https://search.svc.codfw.wmnet:9243/specieswiki_content_1605690403
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikibooks_content_1603009771
curl -XDELETE https://search.svc.codfw.wmnet:9243/specieswiki_content_1603561827
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikisource_content_1603018453
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwiktionary_content_1605236177
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwiki_content_1605226908
curl -XDELETE https://search.svc.codfw.wmnet:9243/commonswiki_content_1605182361
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwiktionary_content_1603248911
curl -XDELETE https://search.svc.codfw.wmnet:9243/incubatorwiki_content_1605414266
curl -XDELETE https://search.svc.codfw.wmnet:9243/jvwiki_content_1605432348
curl -XDELETE https://search.svc.codfw.wmnet:9243/sourceswiki_content_1603561535
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikisource_content_1605234689
curl -XDELETE https://search.svc.codfw.wmnet:9243/mediawikiwiki_content_1605474585
curl -XDELETE https://search.svc.codfw.wmnet:9243/ruwikinews_general_1603449062
curl -XDELETE https://search.svc.codfw.wmnet:9243/metawiki_content_1603317263
curl -XDELETE https://search.svc.codfw.wmnet:9243/mediawikiwiki_content_1603317242
curl -XDELETE https://search.svc.codfw.wmnet:9243/viwiki_general_1603777216
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwiki_content_1603244548
curl -XDELETE https://search.svc.codfw.wmnet:9243/incubatorwiki_content_1603241848
curl -XDELETE https://search.svc.codfw.wmnet:9243/simplewiki_content_1603548687
curl -XDELETE https://search.svc.codfw.wmnet:9243/mrwiki_content_1605490554
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikinews_general_1605228094
curl -XDELETE https://search.svc.codfw.wmnet:9243/viwikisource_content_1603807147
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikivoyage_content_1603020745
curl -XDELETE https://search.svc.codfw.wmnet:9243/metawiki_content_1605474603
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikibooks_content_1605227315
curl -XDELETE https://search.svc.codfw.wmnet:9243/nnwiki_content_1605513552
curl -XDELETE https://search.svc.codfw.wmnet:9243/lvwiki_content_1605473436
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikinews_general_1603010548
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwiki_content_1603009218
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwiki_content_1605416766
curl -XDELETE https://search.svc.codfw.wmnet:9243/srwiki_general_1605741158
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikiversity_content_1603020711
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikivoyage_content_1605235962
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwiktionary_content_1605418379
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwikiquote_content_1603247363
curl -XDELETE https://search.svc.codfw.wmnet:9243/simplewiki_content_1605678929
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwikiquote_content_1605233753
curl -XDELETE https://search.svc.codfw.wmnet:9243/enwiktionary_content_1603021229
curl -XDELETE https://search.svc.codfw.wmnet:9243/itwikisource_content_1603247486
curl -XDELETE https://search.svc.codfw.wmnet:9243/sourceswiki_content_1605690183
curl -XDELETE https://search.svc.codfw.wmnet:9443/test2wiki_content_1603710880
curl -XDELETE https://search.svc.codfw.wmnet:9443/betawikiversity_content_1602857046
curl -XDELETE https://search.svc.codfw.wmnet:9443/betawikiversity_content_1605102354
curl -XDELETE https://search.svc.codfw.wmnet:9443/legalteamwiki_content_1603306353
curl -XDELETE https://search.svc.codfw.wmnet:9443/qualitywiki_general_1605561546
curl -XDELETE https://search.svc.codfw.wmnet:9443/itwikinews_general_1603247359
curl -XDELETE https://search.svc.codfw.wmnet:9443/ombudsmenwiki_general_1603374645
curl -XDELETE https://search.svc.codfw.wmnet:9443/qualitywiki_general_1603422522
curl -XDELETE https://search.svc.codfw.wmnet:9443/itwikinews_general_1605418040
curl -XDELETE https://search.svc.codfw.wmnet:9443/id_internalwikimedia_general_1583493349
curl -XDELETE https://search.svc.codfw.wmnet:9443/collabwiki_general_1605182342
curl -XDELETE https://search.svc.codfw.wmnet:9443/collabwiki_general_1602956396
curl -XDELETE https://search.svc.codfw.wmnet:9443/ombudsmenwiki_general_1605522402
curl -XDELETE https://search.svc.codfw.wmnet:9443/testwikidatawiki_content_1603711117
curl -XDELETE https://search.svc.codfw.wmnet:9443/testwiki_content_1603711111
curl -XDELETE https://search.svc.codfw.wmnet:9443/legalteamwiki_content_1605466742
curl -XDELETE https://search.svc.codfw.wmnet:9643/outreachwiki_content_1605523870
curl -XDELETE https://search.svc.codfw.wmnet:9643/outreachwiki_content_1603376335
curl -XDELETE https://search.svc.codfw.wmnet:9643/checkuserwiki_content_1605180755
curl -XDELETE https://search.svc.codfw.wmnet:9643/checkuserwiki_content_1602954658
curl -XDELETE https://search.svc.codfw.wmnet:9643/itwikivoyage_content_1603248877
curl -XDELETE https://search.svc.codfw.wmnet:9643/simplewikibooks_general_1603548730
curl -XDELETE https://search.svc.codfw.wmnet:9643/stewardwiki_content_1603682503
curl -XDELETE https://search.svc.codfw.wmnet:9643/officewiki_content_1603374444
curl -XDELETE https://search.svc.codfw.wmnet:9643/officewiki_content_1605522211
curl -XDELETE https://search.svc.codfw.wmnet:9643/labswiki_general_1605462963
curl -XDELETE https://search.svc.codfw.wmnet:9643/viwikiquote_archive_1603807143
curl -XDELETE https://search.svc.codfw.wmnet:9643/usabilitywiki_general_1603770531
curl -XDELETE https://search.svc.codfw.wmnet:9643/itwikivoyage_content_1605418284
curl -XDELETE https://search.svc.codfw.wmnet:9643/itwikiversity_content_1605418157
curl -XDELETE https://search.svc.codfw.wmnet:9643/simplewikibooks_general_1605679180
curl -XDELETE https://search.svc.codfw.wmnet:9643/labswiki_general_1603301801
curl -XDELETE https://search.svc.codfw.wmnet:9643/itwikiversity_content_1603248754
Gehel closed this task as Resolved.Nov 23 2020, 1:15 PM
Gehel claimed this task.