Page MenuHomePhabricator

Emergency database primary master failover on s3 primary master
Closed, ResolvedPublic

Description

We need to failover the primary database master for s3 T213858: s3 master emergency failover (db1075) due to some emergency going on with the rack PDUs T213748: swap a2-eqiad PDU with on-site spare.

We would need a 30 minutes read-only window for all those wikis:

1root@cumin1001:/home/marostegui/git/mediawiki-config/dblists# cat s3.dblist
2aawiki
3aawikibooks
4aawiktionary
5abwiki
6abwiktionary
7acewiki
8advisorswiki
9advisorywiki
10adywiki
11afwiki
12afwikibooks
13afwikiquote
14afwiktionary
15akwiki
16akwikibooks
17akwiktionary
18alswiki
19amwiki
20amwikimedia
21amwikiquote
22amwiktionary
23angwiki
24angwikibooks
25angwikiquote
26angwikisource
27angwiktionary
28anwiki
29anwiktionary
30arbcom_cswiki
31arbcom_dewiki
32arbcom_enwiki
33arbcom_fiwiki
34arbcom_nlwiki
35arcwiki
36arwikibooks
37arwikimedia
38arwikinews
39arwikiquote
40arwikisource
41arwikiversity
42arwiktionary
43arzwiki
44astwiki
45astwikibooks
46astwikiquote
47astwiktionary
48aswiki
49aswikibooks
50aswikisource
51aswiktionary
52atjwiki
53auditcomwiki
54avwiki
55avwiktionary
56aywiki
57aywikibooks
58aywiktionary
59azbwiki
60azwiki
61azwikibooks
62azwikiquote
63azwikisource
64azwiktionary
65barwiki
66bat_smgwiki
67bawiki
68bawikibooks
69bclwiki
70bdwikimedia
71be_x_oldwiki
72betawikiversity
73bewiki
74bewikibooks
75bewikimedia
76bewikiquote
77bewikisource
78bewiktionary
79bgwikibooks
80bgwikinews
81bgwikiquote
82bgwikisource
83bhwiki
84bhwiktionary
85biwiki
86biwikibooks
87biwiktionary
88bjnwiki
89bmwiki
90bmwikibooks
91bmwikiquote
92bmwiktionary
93bnwiki
94bnwikibooks
95bnwikisource
96bnwiktionary
97bnwikivoyage
98boardgovcomwiki
99boardwiki
100bowiki
101bowikibooks
102bowiktionary
103bpywiki
104brwiki
105brwikimedia
106brwikiquote
107brwikisource
108brwiktionary
109bswiki
110bswikibooks
111bswikinews
112bswikiquote
113bswikisource
114bswiktionary
115bugwiki
116bxrwiki
117cawikibooks
118cawikimedia
119cawikinews
120cawikiquote
121cawikisource
122cawiktionary
123cbk_zamwiki
124cdowiki
125cewiki
126chairwiki
127chapcomwiki
128checkuserwiki
129chowiki
130chrwiki
131chrwiktionary
132chwiki
133chwikibooks
134chwiktionary
135chywiki
136ckbwiki
137cnwikimedia
138collabwiki
139cowiki
140cowikibooks
141cowikimedia
142cowikiquote
143cowiktionary
144crhwiki
145crwiki
146crwikiquote
147crwiktionary
148csbwiki
149csbwiktionary
150cswikibooks
151cswikinews
152cswikiquote
153cswikisource
154cswikiversity
155cswiktionary
156cuwiki
157cvwiki
158cvwikibooks
159cywiki
160cywikibooks
161cywikiquote
162cywikisource
163cywiktionary
164dawiki
165dawikibooks
166dawikiquote
167dawikisource
168dawiktionary
169dewikibooks
170dewikinews
171dewikiquote
172dewikisource
173dewikiversity
174dewikivoyage
175dewiktionary
176dinwiki
177diqwiki
178dkwikimedia
179donatewiki
180dsbwiki
181dtywiki
182dvwiki
183dvwiktionary
184dzwiki
185dzwiktionary
186ecwikimedia
187eewiki
188electcomwiki
189elwiki
190elwikibooks
191elwikinews
192elwikiquote
193elwikisource
194elwikiversity
195elwikivoyage
196elwiktionary
197emlwiki
198enwikibooks
199enwikinews
200enwikisource
201enwikiversity
202eowikibooks
203eowikinews
204eowikiquote
205eowikisource
206eowiktionary
207eswikibooks
208eswikinews
209eswikiquote
210eswikisource
211eswikiversity
212eswikivoyage
213eswiktionary
214etwiki
215etwikibooks
216etwikimedia
217etwikiquote
218etwikisource
219etwiktionary
220euwiki
221euwikibooks
222euwikiquote
223euwikisource
224euwiktionary
225execwiki
226extwiki
227fawikibooks
228fawikinews
229fawikiquote
230fawikisource
231fawikivoyage
232fawiktionary
233fdcwiki
234ffwiki
235fiu_vrowiki
236fiwikibooks
237fiwikimedia
238fiwikinews
239fiwikiquote
240fiwikisource
241fiwikiversity
242fiwikivoyage
243fiwiktionary
244fixcopyrightwiki
245fjwiki
246fjwiktionary
247foundationwiki
248fowiki
249fowikisource
250fowiktionary
251frpwiki
252frrwiki
253frwikibooks
254frwikinews
255frwikiquote
256frwikisource
257frwikiversity
258frwikivoyage
259furwiki
260fywiki
261fywikibooks
262fywiktionary
263gagwiki
264ganwiki
265gawiki
266gawikibooks
267gawikiquote
268gawiktionary
269gdwiki
270gdwiktionary
271glkwiki
272glwiki
273glwikibooks
274glwikiquote
275glwikisource
276glwiktionary
277gnwiki
278gnwikibooks
279gnwiktionary
280gomwiki
281gorwiki
282gotwiki
283gotwikibooks
284grantswiki
285guwiki
286guwikibooks
287guwikiquote
288guwikisource
289guwiktionary
290gvwiki
291gvwiktionary
292hakwiki
293hawiki
294hawiktionary
295hawwiki
296hewikibooks
297hewikinews
298hewikiquote
299hewikisource
300hewikivoyage
301hewiktionary
302hifwiki
303hifwiktionary
304hiwiki
305hiwikibooks
306hiwikimedia
307hiwikiquote
308hiwikiversity
309hiwikivoyage
310hiwiktionary
311howiki
312hrwiki
313hrwikibooks
314hrwikiquote
315hrwikisource
316hrwiktionary
317hsbwiki
318hsbwiktionary
319htwiki
320htwikisource
321huwikibooks
322huwikinews
323huwikiquote
324huwikisource
325huwiktionary
326hywiki
327hywikibooks
328hywikiquote
329hywikisource
330hywiktionary
331hzwiki
332iawiki
333iawikibooks
334iawiktionary
335idwikibooks
336idwikimedia
337id_internalwikimedia
338idwikiquote
339idwikisource
340idwiktionary
341iegcomwiki
342iewiki
343iewikibooks
344iewiktionary
345igwiki
346iiwiki
347ikwiki
348ikwiktionary
349ilowiki
350ilwikimedia
351incubatorwiki
352inhwiki
353internalwiki
354iowiki
355iowiktionary
356iswiki
357iswikibooks
358iswikiquote
359iswikisource
360iswiktionary
361itwikibooks
362itwikinews
363itwikiquote
364itwikisource
365itwikiversity
366itwikivoyage
367itwiktionary
368iuwiki
369iuwiktionary
370jamwiki
371jawikibooks
372jawikinews
373jawikiquote
374jawikisource
375jawikiversity
376jawiktionary
377jbowiki
378jbowiktionary
379jvwiki
380jvwiktionary
381kaawiki
382kabwiki
383kawiki
384kawikibooks
385kawikiquote
386kawiktionary
387kbdwiki
388kbpwiki
389kgwiki
390kiwiki
391kjwiki
392kkwiki
393kkwikibooks
394kkwikiquote
395kkwiktionary
396klwiki
397klwiktionary
398kmwiki
399kmwikibooks
400kmwiktionary
401knwiki
402knwikibooks
403knwikiquote
404knwikisource
405knwiktionary
406koiwiki
407kowikibooks
408kowikinews
409kowikiquote
410kowikisource
411kowikiversity
412kowiktionary
413krcwiki
414krwiki
415krwikiquote
416kshwiki
417kswiki
418kswikibooks
419kswikiquote
420kswiktionary
421kuwiki
422kuwikibooks
423kuwikiquote
424kuwiktionary
425kvwiki
426kwwiki
427kwwikiquote
428kwwiktionary
429kywiki
430kywikibooks
431kywikiquote
432kywiktionary
433ladwiki
434lawiki
435lawikibooks
436lawikiquote
437lawikisource
438lawiktionary
439lbewiki
440lbwiki
441lbwikibooks
442lbwikiquote
443lbwiktionary
444legalteamwiki
445lezwiki
446lfnwiki
447lgwiki
448lijwiki
449liwiki
450liwikibooks
451liwikinews
452liwikiquote
453liwikisource
454liwiktionary
455lmowiki
456lnwiki
457lnwikibooks
458lnwiktionary
459loginwiki
460lowiki
461lowiktionary
462lrcwiki
463ltgwiki
464ltwiki
465ltwikibooks
466ltwikiquote
467ltwikisource
468ltwiktionary
469lvwiki
470lvwikibooks
471lvwiktionary
472maiwiki
473maiwikimedia
474map_bmswiki
475mdfwiki
476mediawikiwiki
477mgwiki
478mgwikibooks
479mhrwiki
480mhwiki
481mhwiktionary
482minwiki
483miwiki
484miwikibooks
485miwiktionary
486mkwiki
487mkwikibooks
488mkwikimedia
489mkwikisource
490mkwiktionary
491mlwiki
492mlwikibooks
493mlwikiquote
494mlwikisource
495mlwiktionary
496mnwiki
497mnwikibooks
498mnwiktionary
499movementroleswiki
500mrjwiki
501mrwiki
502mrwikibooks
503mrwikiquote
504mrwikisource
505mrwiktionary
506mswiki
507mswikibooks
508mswiktionary
509mtwiki
510mtwiktionary
511muswiki
512mwlwiki
513mxwikimedia
514myvwiki
515mywiki
516mywikibooks
517mywiktionary
518mznwiki
519nahwiki
520nahwikibooks
521nahwiktionary
522napwiki
523nawiki
524nawikibooks
525nawikiquote
526nawiktionary
527nds_nlwiki
528ndswiki
529ndswikibooks
530ndswikiquote
531ndswiktionary
532newiki
533newikibooks
534newiktionary
535newwiki
536ngwiki
537nlwikibooks
538nlwikimedia
539nlwikinews
540nlwikiquote
541nlwikisource
542nlwikivoyage
543nlwiktionary
544nnwiki
545nnwikiquote
546nnwiktionary
547noboard_chapterswikimedia
548nostalgiawiki
549novwiki
550nowikibooks
551nowikimedia
552nowikinews
553nowikiquote
554nowikisource
555nowiktionary
556nrmwiki
557nsowiki
558nvwiki
559nycwikimedia
560nywiki
561nzwikimedia
562ocwiki
563ocwikibooks
564ocwiktionary
565officewiki
566olowiki
567ombudsmenwiki
568omwiki
569omwiktionary
570orwiki
571orwikisource
572orwiktionary
573oswiki
574otrs_wikiwiki
575outreachwiki
576pa_uswikimedia
577pagwiki
578pamwiki
579papwiki
580pawiki
581pawikibooks
582pawikisource
583pawiktionary
584pcdwiki
585pdcwiki
586pflwiki
587pihwiki
588piwiki
589piwiktionary
590plwikibooks
591plwikimedia
592plwikinews
593plwikiquote
594plwikisource
595plwikivoyage
596plwiktionary
597pmswiki
598pmswikisource
599pnbwiki
600pnbwiktionary
601pntwiki
602projectcomwiki
603pswiki
604pswikibooks
605pswikivoyage
606pswiktionary
607ptwikibooks
608ptwikimedia
609ptwikinews
610ptwikiquote
611ptwikisource
612ptwikiversity
613ptwikivoyage
614ptwiktionary
615punjabiwikimedia
616qualitywiki
617quwiki
618quwikibooks
619quwikiquote
620quwiktionary
621rmwiki
622rmwikibooks
623rmwiktionary
624rmywiki
625rnwiki
626rnwiktionary
627roa_rupwiki
628roa_rupwiktionary
629roa_tarawiki
630romdwikimedia
631rowikibooks
632rowikinews
633rowikiquote
634rowikisource
635rowikivoyage
636rowiktionary
637rswikimedia
638ruewiki
639ruwikibooks
640ruwikimedia
641ruwikinews
642ruwikiquote
643ruwikisource
644ruwikiversity
645ruwikivoyage
646ruwiktionary
647rwwiki
648rwwiktionary
649sahwiki
650sahwikiquote
651sahwikisource
652satwiki
653sawiki
654sawikibooks
655sawikiquote
656sawikisource
657sawiktionary
658scnwiki
659scnwiktionary
660scowiki
661scwiki
662scwiktionary
663sdwiki
664sdwikinews
665sdwiktionary
666searchcomwiki
667sewiki
668sewikibooks
669sewikimedia
670sgwiki
671sgwiktionary
672shwiktionary
673shnwiki
674simplewiki
675simplewikibooks
676simplewikiquote
677simplewiktionary
678siwiki
679siwikibooks
680siwiktionary
681skwiki
682skwikibooks
683skwikiquote
684skwikisource
685skwiktionary
686slwiki
687slwikibooks
688slwikiquote
689slwikisource
690slwikiversity
691slwiktionary
692smwiki
693smwiktionary
694snwiki
695snwiktionary
696sourceswiki
697sowiki
698sowiktionary
699spcomwiki
700specieswiki
701sqwiki
702sqwikibooks
703sqwikinews
704sqwikiquote
705sqwiktionary
706srnwiki
707srwikibooks
708srwikinews
709srwikiquote
710srwikisource
711srwiktionary
712sswiki
713sswiktionary
714stewardwiki
715stqwiki
716strategywiki
717stwiki
718stwiktionary
719suwiki
720suwikibooks
721suwikiquote
722suwiktionary
723svwikibooks
724svwikinews
725svwikiquote
726svwikisource
727svwikiversity
728svwikivoyage
729svwiktionary
730swwiki
731swwikibooks
732swwiktionary
733szlwiki
734tawiki
735tawikibooks
736tawikinews
737tawikiquote
738tawikisource
739tawiktionary
740tcywiki
741techconductwiki
742tenwiki
743test2wiki
744testwiki
745testwikidatawiki
746tetwiki
747tewiki
748tewikibooks
749tewikiquote
750tewikisource
751tewiktionary
752tgwiki
753tgwikibooks
754tgwiktionary
755thwikibooks
756thwikinews
757thwikiquote
758thwikisource
759thwiktionary
760tiwiki
761tiwiktionary
762tkwiki
763tkwikibooks
764tkwikiquote
765tkwiktionary
766tlwiki
767tlwikibooks
768tlwiktionary
769tnwiki
770tnwiktionary
771towiki
772towiktionary
773tpiwiki
774tpiwiktionary
775transitionteamwiki
776trwikibooks
777trwikimedia
778trwikinews
779trwikiquote
780trwikisource
781trwiktionary
782tswiki
783tswiktionary
784ttwiki
785ttwikibooks
786ttwikiquote
787ttwiktionary
788tumwiki
789twwiki
790twwiktionary
791tyvwiki
792tywiki
793uawikimedia
794udmwiki
795ugwiki
796ugwikibooks
797ugwikiquote
798ugwiktionary
799ukwikibooks
800ukwikinews
801ukwikiquote
802ukwikisource
803ukwikivoyage
804ukwiktionary
805urwiki
806urwikibooks
807urwikiquote
808urwiktionary
809usabilitywiki
810uzwiki
811uzwikibooks
812uzwikiquote
813uzwiktionary
814vecwiki
815vecwikisource
816vecwiktionary
817vepwiki
818vewiki
819viwikibooks
820viwikiquote
821viwikisource
822viwikivoyage
823viwiktionary
824vlswiki
825votewiki
826vowiki
827vowikibooks
828vowikiquote
829vowiktionary
830warwiki
831wawiki
832wawikibooks
833wawiktionary
834wbwikimedia
835wg_enwiki
836wikimania2005wiki
837wikimania2006wiki
838wikimania2007wiki
839wikimania2008wiki
840wikimania2009wiki
841wikimania2010wiki
842wikimania2011wiki
843wikimania2012wiki
844wikimania2013wiki
845wikimania2014wiki
846wikimania2015wiki
847wikimania2016wiki
848wikimania2017wiki
849wikimania2018wiki
850wikimaniawiki
851wikimaniateamwiki
852wowiki
853wowikiquote
854wowiktionary
855wuuwiki
856xalwiki
857xhwiki
858xhwikibooks
859xhwiktionary
860xmfwiki
861yiwiki
862yiwikisource
863yiwiktionary
864yowiki
865yowikibooks
866yowiktionary
867yuewiktionary
868zawiki
869zawikibooks
870zawikiquote
871zawiktionary
872zeawiki
873zerowiki
874zh_classicalwiki
875zh_min_nanwiki
876zh_min_nanwikibooks
877zh_min_nanwikiquote
878zh_min_nanwikisource
879zh_min_nanwiktionary
880zh_yuewiki
881zhwikibooks
882zhwikinews
883zhwikiquote
884zhwikisource
885zhwikiversity
886zhwikivoyage
887zhwiktionary
888zuwiki
889zuwikibooks
890zuwiktionary

We still need to confirm when, but it will be in the next few days. Sorry for the short notice, but this is an emergency.

Date: Thursday 17th January
Time: 07:00 AM UTC - 07:30 AM UTC (we expect not to use the full 30 minutes window)

Impact: All those wikis will go read-only. No edits will be allowed. Reads will not be impacted.

Event Timeline

Marostegui triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2019, 8:28 PM
Marostegui renamed this task from Emergency master failover on s3 primary master to Emergency database primary master failover on s3 primary master.Jan 15 2019, 8:42 PM
Elitre added a subscriber: Elitre.

Thanks for filing,

  • What is the problem? - Possibly already explained in the description
  • How can we help you?
  • What does success look like?
  • What is your deadline? (I know this isn't firm, but let us know when it is!)
Thanks for filing, 
- What is the problem? - Possibly already explained in the description

As explained on the description: the current database primary master is on a rack that needs emergency maintenance due to failures on the PDUs that supply power to the servers (T213748)

  • How can we help you?

Once we know when this failover will be done, this needs to be announced to the users.

  • What does success look like?

After the read-only period, wikis will be writable again.

  • What is your deadline? (I know this isn't firm, but let us know when it is!)

Will do as soon as I know.

Johan claimed this task.Jan 16 2019, 7:53 AM
Johan added a subscriber: Johan.

(Assuming this is the task we want to use for communicating the issue.)

The next issue of Tech News is going out Monday evening, UTC. The issue is frozen on Friday afternoon, UTC. If we don't know when in two days, but think it'll be at some point next week, I'll include it and give a link (to Phab or elsewhere) to check for updates.

Restricted Application added a project: User-Johan. · View Herald TranscriptJan 16 2019, 7:53 AM

(Assuming this is the task we want to use for communicating the issue.)

The next issue of Tech News is going out Monday evening, UTC. The issue is frozen on Friday afternoon, UTC. If we don't know when in two days, but think it'll be at some point next week, I'll include it and give a link (to Phab or elsewhere) to check for updates.

I think today we will know when, most likely.

Johan added a comment.Jan 16 2019, 8:09 AM

Excellent.

So, this list is our wikis except:

  • Wikipedias
  • Commons
  • Wikidata
  • Meta
  • Wikispecies
  • A few Wiktionaries (English, Malagasy, French)
  • English Wikivoyage
  • English Wikiquote

(And potentially a few non-content wikis.)

Am I missing something?

Not sure what you mean, the full list of wikis that will be affected is at the task description.
Let me know if I need to clarify something further

Johan moved this task from Backlog to Do soon on the User-Johan board.Jan 16 2019, 9:15 AM
jcrespo added a comment.EditedJan 16 2019, 9:28 AM

Wikipedias

Some wikipedias will be affected, if you want the shorter list of wikis that WON'T be affected it is at https://noc.wikimedia.org/db.php :

enwiki
bgwiki
bgwiktionary
cswiki
enwikiquote
enwiktionary
eowiki
fiwiki
idwiki
itwiki
nlwiki
nowiki
plwiki
ptwiki
svwiki
thwiki
trwiki
zhwiki
commonswiki
testcommonswiki
cebwiki
dewiki
enwikivoyage
mgwiktionary
shwiki
srwiki
frwiki
jawiki
ruwiki
eswiki
huwiki
hewiki
ukwiki
frwiktionary
metawiki
arwiki
centralauth
cawiki
viwiki
fawiki
rowiki
kowiki
wikidatawiki
labswiki
labtestwiki

While the affected wikis are the the least 5-10% traffic wikis, there are a lot of low-load projects (wikipedias in the "top 20", commons, meta, wikidata not affected).

Just to confirm when this will take place:
Thusday 17th January at 07:00 AM UTC

Marostegui updated the task description. (Show Details)Jan 16 2019, 12:53 PM
Johan added a comment.Jan 16 2019, 1:34 PM

Thanks!

What's the best place to link to for updates?

For the past failovers, I think there has been a page on meta informing about the failover.
This is what was done when we did a failover on the db master that contained enwiki: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Reminder:_No_editing_06:00%E2%80%9306:30_AM_(UTC),_18_July
As s3 contains more than 800 wikis, probably meta is the right place?

Johan added a comment.Jan 16 2019, 2:02 PM

It doesn't really matter, I just want to link somewhere people can check for more information and/or in case something happens. If a Phab task if what you'll keep most up to date, a Phab task is fine. Your choice.

(I'll send this out ASAP.)

We tend to avoid using phabricator links as we are not sure our phabricator servers might be able to cope with all the traffic if everything gets linked to it

Johan added a comment.Jan 16 2019, 2:08 PM

OK, I'll create a page myself.

Thank you very much - really appreciated it.

Johan added a comment.Jan 16 2019, 2:50 PM

Message delivered to Village Pumps of most affected wikis. Will continue to follow up with some more messages.

Thank you very much!

Johan added a comment.Jan 16 2019, 4:31 PM

For some for me unknown reason, not all wikis on the target list have been getting the MassMessage. Some have. I don't know what's causing the problem.

Johan added a subscriber: Jseddon.Jan 16 2019, 4:58 PM

(I've also emailed @Jseddon to make him aware of the problem and see if he can put up something based on what we've said previously in time. If he can, great. If not, that's what happens in emergencies.)

Johan added a subscriber: Tgr.Jan 16 2019, 6:05 PM

@Tgr gave me a script to find out which wikis hadn't got the delivery. I'll try to send it out again.

Johan added a comment.Jan 16 2019, 7:04 PM

Failed MassMessages have been resent.

Johan added a comment.Jan 16 2019, 7:05 PM

In small batches, because MassMessage seems to be choking on large taget lists – the first attempt ran through the first 20%, then started claiming readonly for the remaining pages.

Marostegui closed this task as Resolved.Jan 17 2019, 7:15 AM

Thank you for all your help
This was done:
Read only ON at: 07:01:00
Read only OFF at: 07:04:20

Total time read only time: 03:20 minutes

If you see something strange, please let us know.
Thanks

Amazing work, y'all.

Johan moved this task from Do soon to Archive on the User-Johan board.Jan 22 2019, 6:03 PM