opensearch psi and omega clusters red in eqiad
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	May 29 2025, 9:44 AM

Description

The two psi & omega clusters appear to be red:

Affected indices in omega:

cswikiversity_titlesuggest_1746619447
pnbwiktionary_general_1728064460
avkwiki_general_1727968296
plwikibooks_archive_1717287746
niawiki_archive_1717295247
mlwikiquote_titlesuggest_1727330941
ladwiki_general_1728049756
ptwikiquote_general_1727941540
cswikiversity_general_1727974191
nowikimedia_archive_1717382890
olowiki_titlesuggest_1709273490
collabwiki_content_1717525891
azwikibooks_general_1728062185
iewikibooks_content_1728084157
sswiki_archive_1717212472
wikimania2018wiki_general_1727952909
rowikibooks_titlesuggest_1709188929
cywikiquote_content_1728065706
rswikimedia_titlesuggest_1709189069
liwiktionary_general_1727877513
fawikibooks_content_1727945786
svwikiquote_content_1727882314
cswikinews_archive_1717344083
sahwikisource_archive_1717275213
biwiktionary_content_1727936082
iewikibooks_archive_1717322287
lldwiki_archive_1717374833
oswiki_titlesuggest_1709273521
quwikibooks_archive_1717203018
afwiktionary_content_1728115734
nnwiktionary_general_1728035948
kywiktionary_content_1728034920
trwikisource_content_1728067599
mnwiktionary_content_1728056885
trwikisource_titlesuggest_1709192891
vowikibooks_general_1728075279
crwiktionary_archive_1717292001
ndswiktionary_titlesuggest_1709187191
vecwikisource_archive_1717301182
bat_smgwiki_archive_1717316906
wikimania2007wiki_content_1728041155
bmwikiquote_general_1728032926
bmwiktionary_archive_1717353117
sgwiki_content_1728061832
ilowiki_titlesuggest_1709269972
kowikibooks_archive_1717348583
gorwiktionary_archive_1717364543
pawiktionary_titlesuggest_1727332285
nowikibooks_general_1727963507
hewiktionary_archive_1717375905
zhwikiversity_general_1728038293
tawikiquote_titlesuggest_1726125599
hawiktionary_titlesuggest_1709182608
akwikibooks_general_1727943281
kgwiki_archive_1717351856

affected indices in psi:

bowikibooks_content_1728053708
wikimania2014wiki_archive_1717384554
xhwikibooks_general_1728091270
udmwiki_titlesuggest_1709278491

AC:

unblock the update pipeline in eqiad
re-create empty broken indices
populate broken archive indices
copy broken content & general indices from codfw
refresh broken completion suggester indices
file an incident report: https://wikitech.wikimedia.org/wiki/Incidents/2025-05-29_OpenSearch_clusters_unavailable
understand the root cause of why many indices went red
- see https://wikitech.wikimedia.org/wiki/Incidents/2025-05-29_OpenSearch_clusters_unavailable
~~[] better alerting on this, the update-pipeline started to misbehave on may 28 21:00~~ should be done as incident followup
~~[] adapt the tooling/cookbooks/runbooks to make sure it does not happen again~~ should be done as incident followup

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		BTullis	T395546 opensearch psi and omega clusters red in eqiad
		Open		None	T396393 Verify/fix elasticsearch ban cookbook/playbook

Event Timeline

dcausse created this task.May 29 2025, 9:44 AM

Restricted Application added subscribers: Nemoralis, Huji, Strainu and 3 others. · View Herald TranscriptMay 29 2025, 9:44 AM

I just deleted all these red indices because I'm not sure I could have recovered them, this should hopefully unblock the update pipeline.
Next step is to recover these indices, my current plan is as follow:

build the indices using UpdateSearchIndexConfig
run a copy from codfw -> eqiad using a script

Wikis with a broken archive, content, general index:

afwiktionary
akwikibooks
avkwiki
azwikibooks
bat
biwiktionary
bmwikiquote
bmwiktionary
bowikibooks
collabwiki
crwiktionary
cswikinews
cswikiversity
cywikiquote
fawikibooks
gorwiktionary
hewiktionary
iewikibooks
kgwiki
kowikibooks
kywiktionary
ladwiki
liwiktionary
lldwiki
mnwiktionary
niawiki
nnwiktionary
nowikibooks
nowikimedia
plwikibooks
pnbwiktionary
ptwikiquote
quwikibooks
sahwikisource
sgwiki
sswiki
svwikiquote
trwikisource
vecwikisource
vowikibooks
wikimania2007wiki
wikimania2014wiki
wikimania2018wiki
xhwikibooks
zhwikiversity

Wikis with a broken title suggest index:

cswikiversity
hawiktionary
ilowiki
mlwikiquote
ndswiktionary
olowiki
oswiki
pawiktionary
rowikibooks
rswikimedia
tawikiquote
trwikisource
udmwiki

Mentioned in SAL (#wikimedia-operations) [2025-05-29T10:10:13Z] <dcausse> T395546: creating empty content indices in eqiad for afwiktionary biwiktionary bowikibooks collabwiki cywikiquote fawikibooks iewikibooks kywiktionary mnwiktionary sgwiki svwikiquote trwikisource wikimania2007wiki

Mentioned in SAL (#wikimedia-operations) [2025-05-29T10:15:17Z] <dcausse> T395546: creating empty general indices in eqiad for akwikibooks avkwiki azwikibooks bmwikiquote cswikiversity ladwiki liwiktionary nnwiktionary nowikibooks pnbwiktionary ptwikiquote vowikibooks wikimania2018wiki xhwikibooks zhwikiversity

Mentioned in SAL (#wikimedia-operations) [2025-05-29T11:52:05Z] <dcausse> T395546: creating empty archive indices in eqiad for bat_smgwiki bmwiktionary crwiktionary cswikinews gorwiktionary hewiktionary iewikibooks kgwiki kowikibooks lldwiki niawiki nowikimedia plwikibooks quwikibooks sahwikisource sswiki vecwikisource wikimania2014wiki

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:09:28Z] <dcausse> T395546: populating archive indices in eqiad for bat_smgwiki bmwiktionary crwiktionary cswikinews gorwiktionary hewiktionary iewikibooks kgwiki kowikibooks lldwiki niawiki nowikimedia plwikibooks quwikibooks sahwikisource sswiki vecwikisource wikimania2014wiki

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:33:27Z] <dcausse> T395546 restoring bowikibooks_content in psi@eqiad from psi@codfw

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:34:38Z] <dcausse> T395546 restoring xhwikibooks_general in psi@eqiad from psi@codfw

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:36:32Z] <dcausse> T395546 restoring collabwiki iewikibooks cywikiquote fawikibooks svwikiquote biwiktionary afwiktionary kywiktionary trwikisource mnwiktionary wikimania2007wiki sgwiki content indices in psi@eqiad from psi@codfw

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:38:39Z] <dcausse> T395546 (errata from previous log: s/psi/omega/) restoring collabwiki iewikibooks cywikiquote fawikibooks svwikiquote biwiktionary afwiktionary kywiktionary trwikisource mnwiktionary wikimania2007wiki sgwiki content indices in omega@eqiad from omega@codfw

Mentioned in SAL (#wikimedia-operations) [2025-05-29T12:45:38Z] <dcausse> T395546: restoring pnbwiktionary avkwiki ladwiki ptwikiquote cswikiversity azwikibooks wikimania2018wiki liwiktionary nnwiktionary vowikibooks bmwikiquote nowikibooks zhwikiversity akwikibooks general indices in omega@eqiad from omega@codfw

dcausse triaged this task as High priority.May 29 2025, 12:53 PM

dcausse updated the task description. (Show Details)

Restored indices in omega:

green open pnbwiktionary_general_first                  6xo89dkTRAKJ4ac6sXMoSg 1 2    504     0    4.9mb    1.6mb
green open avkwiki_general_first                        BnyNtR69QjSwxV4O3h_9Zg 1 2   5873     0   53.7mb   17.6mb
green open ladwiki_general_first                        TSWg8ToqS0SoBxUhOSnCbQ 1 2   6908     0     92mb     30mb
green open ptwikiquote_general_first                    YP_8JxZIQUmYxH4ZAU1SPA 1 2  19919     0  249.6mb   82.6mb
green open cswikiversity_general_first                  nyonNChhRVOemUiSBtZ9OA 1 2   7229    33  225.4mb   74.9mb
green open azwikibooks_general_first                    speqCfzwRd-Quut-4CqM8g 1 2   5500     0   72.1mb   24.1mb
green open wikimania2018wiki_general_first              quhWLKCeRv6Yfri3M3dSDQ 1 2   6097     0   35.6mb   11.8mb
green open liwiktionary_general_first                   4OJuhxtCRh2NXDVJ9Rna4A 1 2  16618     0  176.6mb   58.5mb
green open nnwiktionary_general_first                   H47nSodOT4S2Qtsm9B6FVw 1 2   1640     0   46.6mb   15.5mb
green open vowikibooks_general_first                    eBvVR8CuT8yr4vtnG6ZB4g 1 2    262     0    2.5mb    856kb
green open bmwikiquote_general_first                    EppydKhuS6KixBWfQ-eNRQ 1 2     52     0  599.3kb  199.7kb
green open nowikibooks_general_first                    USM0x_KQQ0qCNFnjQESsJA 1 2   2552     0   40.6mb   13.5mb
green open zhwikiversity_general_first                  G9KoP0fRRrakZOtWdhLHGQ 1 2   7891     0  400.5mb  134.2mb
green open akwikibooks_general_first                    iTLc_9W5SX6Ka7baEPvCtw 1 2     84     0    1.5mb  528.2kb
green open collabwiki_content_first                     eG9JgGIPRWayT8YSNKPVhQ 1 2   3169   471  546.8mb  183.3mb
green open iewikibooks_content_first                    KQkwajbfS32wavqqhy1Ewg 1 2     65     0    2.6mb  906.5kb
green open cywikiquote_content_first                    GI4h_fiMT1isSQep4OdqOw 1 2    457     0    6.9mb    2.3mb
green open fawikibooks_content_first                    UWCtUCLiQS63bFITlr-Wxg 1 2   4271     0  397.6mb  132.6mb
green open svwikiquote_content_first                    1rb46QjtSauQ9rVr9C9uYg 1 2   1906     0   60.6mb   20.9mb
green open biwiktionary_content_first                   6k9oFADRST2zvTnbnEJgtQ 1 2      0     0     624b     208b
green open afwiktionary_content_first                   SlU_uOrcT_en1isCPWO2_Q 1 2  23575     0  227.9mb   75.7mb
green open kywiktionary_content_first                   3Q1hENhDQjuAxmHHMYuIcA 1 2  21290     0  120.9mb   40.1mb
green open trwikisource_content_first                   5PhxU9u6QGuKzf8OS6nRjQ 1 2  11848     0  881.8mb  294.3mb
green open mnwiktionary_content_first                   HoeC_efhRuqaso7w65EMnw 1 2   8932     0  106.4mb   35.4mb
green open wikimania2007wiki_content_first              xHjtKAeDQ-qiahIpN5wV_w 1 2    708     0   34.3mb     11mb
green open sgwiki_content_first                         yXk6MI_USNyxa8Biv4z6Lg 1 2    587     0      7mb    2.3mb
green open bat_smgwiki_archive_first                    rstpepYLR4eyiG0m7jFxdQ 1 2   1937     0    438kb  150.1kb
green open bmwiktionary_archive_first                   ln6VdSm3Q2KIVcOpwdUcPQ 1 2      4     0   14.1kb    4.7kb
green open crwiktionary_archive_first                   kgKWX502RSO0uFugkXQmdQ 1 2      0     0     624b     208b
green open cswikinews_archive_first                     M4uk4lGQTXW15yxjiMOs2g 1 2    552     0  212.9kb   70.9kb
green open gorwiktionary_archive_first                  O96K3pDHRvCOHoNMYkSP0g 1 2    185     0   45.4kb   15.1kb
green open hewiktionary_archive_first                   3p8ZGniaRa2z2ZXMUn2xOA 1 2   4703     0      1mb  372.9kb
green open iewikibooks_archive_first                    CTE27X02S66NgcCSUGShkQ 1 2    197     0   49.8kb   16.6kb
green open kgwiki_archive_first                         7wt9BfeVSGODn9R0-WbHpg 1 2    800     0    154kb   54.7kb
green open kowikibooks_archive_first                    UKrmgC0FQTSyq5_wKXHrEw 1 2   1938     0  411.9kb  137.3kb
green open lldwiki_archive_first                        GwL6ROKtT2eTWTDoXG4V2w 1 2   1319     0  194.9kb   64.9kb
green open niawiki_archive_first                        0rGaElfRT32vOSajEhgHzQ 1 2    255     0     74kb   24.6kb
green open nowikimedia_archive_first                    I0XjPY2fQm2h3Rg_iE-Exg 1 2    683     0  138.2kb   49.5kb
green open plwikibooks_archive_first                    ypf49ohZQDCvFcOZJ5GOWw 1 2   7178     4    1.5mb  537.4kb
green open quwikibooks_archive_first                    MLmimEdPRoqXVISq5HNRLA 1 2      6     0   14.7kb    4.9kb
green open sahwikisource_archive_first                  Bk9EUZVFQNuMsDsqnp2YdA 1 2    156     0   44.9kb   14.9kb
green open sswiki_archive_first                         1116brBAQAy-BkUIXn5NXQ 1 2   1015     0  201.8kb   67.2kb
green open vecwikisource_archive_first                  vMazl--5Q-2x18YHbbuJlQ 1 2    356     0   97.7kb   32.5kb

And their equivalent in omega@codfw:

green open pnbwiktionary_general_1728075097             aGvqCsNRTz6f6SlVr1c6vQ 1 2    504     25    8.2mb    2.7mb
green open avkwiki_general_1728116214                   XthRXUlWSfihWIZwB9RyBw 1 2   5873    112   55.6mb   18.5mb                                                                                                                                            
green open ladwiki_general_1728001086                   bAkkxq0MTTGL4ueo_YDipQ 1 2   6908    365  119.4mb   41.1mb
green open ptwikiquote_general_1728076292               qsFGnexyRcK94vlbLsfiEg 1 2  19919   2573  277.4mb   92.2mb
green open cswikiversity_general_1728087138             DO1VBV9ZRLOcVEZ51pxXNQ 1 2   7229    731  276.1mb     92mb
green open azwikibooks_general_1728115807               cPzirkq7S5SFfdpvsktAVA 1 2   5500   1527   82.3mb   27.4mb
green open wikimania2018wiki_general_1728117876         fPZHsxhrTWW-LczempHtbw 1 2   6097     22   34.3mb   11.4mb
green open liwiktionary_general_1728100445              dnAWbqINScSDWgiWrqUMug 1 2  16618   4984  262.1mb   87.5mb
green open nnwiktionary_general_1727945030              T4pq1GKaQV2WaKFiJE3-Nw 1 2   1640     71   51.3mb   17.1mb
green open vowikibooks_general_1728092725               n_w9LbSEQ7SyAh791Am2kQ 1 2    262     14    2.7mb      1mb
green open bmwikiquote_general_1728109910               mz7TjizATSu8RonZsuuf0A 1 2     52      0  642.7kb  214.2kb
green open nowikibooks_general_1728095110               -8mpzBNyRkSpuUuPFFhOFQ 1 2   2552    123   48.7mb   16.2mb
green open zhwikiversity_general_1727926346             RH1n5avvS-m2sFQeH-XVQw 1 2   7891    773  561.4mb    187mb
green open akwikibooks_general_1728098330               PRbTYhLVQQqsWl_cYZB6rw 1 2     84      1    1.5mb  543.7kb
green open collabwiki_content_1717525775                Sa2TnepgTjyOaYI_y7r_vQ 1 2   3169    383  455.1mb  154.2mb                                                                                                                                            
green open iewikibooks_content_1728072119               c80oClFKQmm8ocZWI7nIVQ 1 2     65      0    2.5mb  885.7kb                                                                                                                                            
green open cywikiquote_content_1728108647               BOq-wT5MQT-m-3KKS6vEDA 1 2    457      0      7mb    2.3mb                                                                                                                                            
green open fawikibooks_content_1728063127               zoftV85wTEOSNKlddU-0FA 1 2   4271     11    302mb  103.3mb                                                                                                                                            
green open svwikiquote_content_1728063385               4T5pEPl2TAuxktFxW0wmRA 1 2   1906      6   60.4mb   20.1mb                                                                                                                                            
green open biwiktionary_content_1728081153              dYQW-fiURZm0juR0chlTAg 1 2      0      0     786b     262b                                                                                                                                            
green open afwiktionary_content_1728063928              DCP911oaTd6KE-lt0kMESw 1 2  23575      8  235.9mb   78.2mb                                                                                                                                            
green open kywiktionary_content_1727977186              V_buKCrCQtuNlT_QPgx2Rw 1 2  76857  10198  308.1mb   99.4mb                                                                                                                                            
green open trwikisource_content_1727976421              4SYsXtWLSG6ti8LymOckDg 1 2  19762     50    1.2gb  379.4mb                                                                                                                                            
green open mnwiktionary_content_1728089407              U_-4jHDjSe6HP4LaI_lJYg 1 2  44950      1  423.8mb  139.9mb             
green open wikimania2007wiki_content_1728091884         MH-m_GMsRdiH1X7XbRQrYg 1 2    708     19     34mb   11.3mb             
green open sgwiki_content_1728069557                    kkZUvBDRTkCrcQjQthtbPg 1 2    587      4    7.6mb    2.5mb                                                                                                                                            
green open bat_smgwiki_archive_1717076412               LR48Puq9S3ye4LMZlGEBHA 1 2   1984      0  478.6kb  159.5kb                                                                                                                                            
green open bmwiktionary_archive_1717147635              YT1FUlrgQeq6PDamCrQy5w 1 2      4      0   14.3kb    4.7kb                                                                                                                                            
green open crwiktionary_archive_1717190326              7KhbFPENQre6NGUZFt8vRw 1 2      5      0   14.6kb    4.8kb                                                                                                                                            
green open cswikinews_archive_1717209262                Ikx4-lSNQOq9udspJvsKkw 1 2    562      0  265.5kb   88.5kb             
green open gorwiktionary_archive_1717177742             zzJRNsbxTNuBa27EobiwlA 1 2    181      0   82.5kb   27.5kb             
green open hewiktionary_archive_1717139660              qcUz2vszQ820EO53tXgD0w 1 2   4797      0    1.1mb  392.7kb                                                                                                                                            
green open iewikibooks_archive_1717139793               XgU_wvQ6SjGtt_8gaF6Ecw 1 2    204      0     87kb     29kb                                                                                                                                            
green open kgwiki_archive_1717175466                    -ZE8yDpsRf6ha6wMEHbY9A 1 2    807      0  189.4kb   63.1kb                                                                                                                                            
green open kowikibooks_archive_1717155643               ngZCpSS0Q7Cznd7mCoYVkA 1 2   1947      0  474.1kb  157.9kb                                                                                                                                            
green open lldwiki_archive_1717132126                   XespNKcVRVmqPdAm5yWnbw 1 2   1344      0  303.3kb  101.1kb             
green open niawiki_archive_1717209393                   L3Ssl2XySjiKezhH3Fw3lw 1 2    280      0   93.8kb   31.2kb             
green open nowikimedia_archive_1717134066               ItCSWOHEQKKDrptePrFhsQ 1 2    725      0  257.2kb   85.7kb             
green open plwikibooks_archive_1717140493               RB75s2jaS1aMSOPOfiotbQ 1 2   7356      1    1.6mb  575.8kb             
green open quwikibooks_archive_1717114275               5B6G8Gk5TwiB0ZOYdegNIg 1 2      6      0   26.7kb    8.9kb             
green open sahwikisource_archive_1717198334             cVmU2TEpT5Coek0VR8NoqQ 1 2    166      0   82.9kb   27.6kb             
green open sswiki_archive_1717139947                    z6rgpy8FStOsgGkYmV4C3w 1 2   1160      0  329.2kb  109.7kb             
green open vecwikisource_archive_1717139758             pzgVqMj7TSioioD-PFJ-Ew 1 2    364      0  195.5kb   65.1kb

Restored indices in psi:

green open bowikibooks_content_first                         zQVg0w7kSfm7Pu4AszzEIg 1 2      1     0   61.4kb   20.4kb
green open xhwikibooks_general_first                         B9NG7ZH4S26RSAdMZB2VRg 1 2    119     0    1.3mb  474.7kb
green open wikimania2014wiki_archive_first                   wi7DZXTvTfuzntdyL6wa7Q 1 2    213     0   72.9kb   24.3kb

And their equivalent in psi@codfw:

green open bowikibooks_content_1728074052                    kAzHruTMQUyoX1hFN7p_KQ 1 2      1     0   60.4kb   20.1kb
green open xhwikibooks_general_1728065982                    OFgVQXfDSdy3tyMXaO2N_Q 1 2    119     4    1.5mb    537kb
green open wikimania2014wiki_archive_1717155822              E_YDq7xaTuOHOF1tFZHN_Q 1 2    223     0  100.6kb   33.5kb

Mentioned in SAL (#wikimedia-operations) [2025-05-29T14:31:14Z] <dcausse> T395546: rebuildind eqiad completion indices for cswikiversity pnbwiktionary avkwiki mlwikiquote ladwiki ptwikiquote cswikiversity olowiki azwikibooks wikimania2018wiki rowikibooks rswikimedia liwiktionary oswiki nnwiktionary trwikisource vowikibooks ndswiktionary bmwikiquote ilowiki pawiktionary nowikibooks zhwikiversity tawikiquote hawiktionary akwikibooks udmwiki xhwikibooks

bking mentioned this in T395356: Test snapshot repos in production Cirrussearch.May 29 2025, 2:37 PM

bking mentioned this in T395571: Verify/fix Logstash pipeline/log rotate for Search Platform-owned OpenSearch clusters.May 29 2025, 2:40 PM

dcausse lowered the priority of this task from High to Medium.May 29 2025, 2:53 PM

dcausse updated the task description. (Show Details)

Nemoralis unsubscribed.May 29 2025, 3:26 PM

bking mentioned this in T393786: Document review/refresh for DPE-owned OpenSearch clusters.May 29 2025, 10:10 PM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.May 30 2025, 7:45 AM

Gehel edited projects, added Discovery-Search (2025.05.24 - 2025.06.13); removed Discovery-Search.May 30 2025, 9:38 AM

Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.05.24 - 2025.06.13) board.

Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE.

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.

bking mentioned this in T396393: Verify/fix elasticsearch ban cookbook/playbook.Jun 10 2025, 2:11 PM