Page MenuHomePhabricator
Create Task
Maniphest T329611

Toolforge grid: start webservices after outage
Closed, ResolvedPublic
Actions

Description

After outage in parent task T329535: Cloud Ceph outage 2023-02-13 and in a bad timing (because T329467: remove webservicemonitor (down due to DNS errors) ) we need to manually start webservices.

We rescued this information from redis which contains hints about what to re-start:

P44617 toolforge grid webservices redis keys
1127.0.0.1:6379> KEYS *
2 1) "prefix:render-tests"
3 2) "redirect:bub"
4 3) "prefix:edcounter"
5 4) "redirect:nn1l2bot"
6 5) "redirect:video-cut-tool-back-end"
7 6) "prefix:locktool"
8 7) "redirect:watchr"
9 8) "redirect:orator-matcher"
10 9) "prefix:ipinfo"
11 10) "redirect:wd-query-builder"
12 11) "redirect:croptool-test"
13 12) "prefix:ash-dev"
14 13) "prefix:laaknortools"
15 14) "redirect:liangent-toolserver"
16 15) "prefix:cyberbot"
17 16) "prefix:zygimantus-dev"
18 17) "prefix:video-cut-tool-back-end"
19 18) "redirect:ruwikisource"
20 19) "prefix:wd-shex-infer"
21 20) "prefix:simple"
22 21) "prefix:metmuseum"
23 22) "redirect:wmtran"
24 23) "redirect:canarybot"
25 24) "redirect:projektneuheiten-feed"
26 25) "prefix:ptbot"
27 26) "prefix:blockcalc"
28 27) "redirect:xtools-mab-dev"
29 28) "redirect:articleplaceholderwiki"
30 29) "redirect:catscore"
31 30) "prefix:wiki-irc"
32 31) "prefix:languageproofing-ui"
33 32) "prefix:etwikt"
34 33) "prefix:steinsplitter"
35 34) "prefix:gtirloni-sandbox"
36 35) "redirect:atiro"
37 36) "redirect:joinedventure"
38 37) "prefix:pg2ws"
39 38) "redirect:family"
40 39) "prefix:centralnotice-bannergenerator"
41 40) "prefix:panoviewer"
42 41) "prefix:wikintu"
43 42) "redirect:sge-status"
44 43) "redirect:veblenbot"
45 44) "prefix:quality-analyzer"
46 45) "prefix:listpages"
47 46) "prefix:checkpersondata"
48 47) "redirect:unpatrollededitstats"
49 48) "redirect:lahitools"
50 49) "prefix:chocobot"
51 50) "prefix:wiki-needs-pictures"
52 51) "prefix:ext-lnk-discover"
53 52) "redirect:wiki-as-git"
54 53) "prefix:morph"
55 54) "prefix:atiro"
56 55) "redirect:ash-dev"
57 56) "redirect:plaintexteditcounter"
58 57) "redirect:jawi"
59 58) "prefix:wmtran"
60 59) "k8s_services"
61 60) "prefix:query2map"
62 61) "prefix:t187305-demo"
63 62) "redirect:os"
64 63) "prefix:alex-wiki"
65 64) "prefix:itemfinder"
66 65) "redirect:iplookup"
67 66) "prefix:wikitrends"
68 67) "prefix:joinedventure"
69 68) "prefix:paws-dev"
70 69) "redirect:drewbot"
71 70) "prefix:mostlinkedmissing"
72 71) "prefix:missing-from-wikipedia"
73 72) "prefix:croptool-test"
74 73) "prefix:dispenser"
75 74) "prefix:multichill"
76 75) "prefix:mjbmrbot"
77 76) "prefix:lahitools"
78 77) "redirect:analytics"
79 78) "prefix:rangecontrib"
80 79) "prefix:userimpact"
81 80) "redirect:aleph"
82 81) "prefix:wd-query-builder"
83 82) "redirect:rotpunkt-bot"
84 83) "prefix:parliament-diagram-generator"
85 84) "prefix:wahldiagramm"
86 85) "prefix:covid-obit"
87 86) "redirect:locktool"
88 87) "redirect:pg2ws"
89 88) "redirect:tulsibot"
90 89) "redirect:submitter"
91 90) "redirect:linkchecker"
92 91) "prefix:mavrikant"
93 92) "prefix:wikisource-penguin-classics"
94 93) "prefix:afdstats"
95 94) "redirect:denisa"
96 95) "prefix:wnegar"
97 96) "prefix:articleplaceholderwiki"
98 97) "prefix:furutani"
99 98) "redirect:postcardcatfinder"
100 99) "prefix:videoconvert"
101100) "redirect:zumraband"
102101) "redirect:wd-art"
103102) "prefix:wm-metrics"
104103) "prefix:hrwiki"
105104) "prefix:rotpunkt-bot"
106105) "redirect:checkpersondata"
107106) "redirect:citation-template-filling"
108107) "prefix:zhdeletionpedia"
109108) "prefix:mrmetadata"
110109) "prefix:articles-by-lat-lon-without-images"
111110) "prefix:xmlfeed"
112111) "redirect:render"
113112) "prefix:mdwiki"
114113) "redirect:panoviewer"
115114) "prefix:gerakitools"
116115) "redirect:friskobot"
117116) "prefix:fountain"
118117) "redirect:urdubot"
119118) "redirect:orpheus"
120119) "prefix:tessdata"
121120) "prefix:draftifyhistory"
122121) "prefix:twl"
123122) "redirect:osmviews"
124123) "redirect:twl"
125124) "prefix:footygen"
126125) "redirect:slumpartikel"
127126) "redirect:blame"
128127) "prefix:relgen"
129128) "prefix:suggestbot"
130129) "redirect:sbot"
131130) "prefix:wptestblog2"
132131) "prefix:shuaib-bot"
133132) "prefix:wmds-archive"
134133) "redirect:gerakibot"
135134) "prefix:denisa"
136135) "prefix:kaleem-bot"
137136) "redirect:huggle"
138137) "redirect:draftifyhistory"
139138) "prefix:shumariyat"
140139) "prefix:wikidata-timeline"
141140) "prefix:wikipedia-contributor-locations"
142141) "prefix:freddy2001"
143142) "prefix:csbot"
144143) "redirect:wikisource-penguin-classics"
145144) "redirect:qrank"
146145) "prefix:wd-art"
147146) "redirect:wikintu"
148147) "redirect:render-tests"
149148) "redirect:energybot"
150149) "prefix:jembot"
151150) "redirect:toolschecker-ge-ws"
152151) "redirect:zhwiki"
153152) "redirect:honeypot95"
154153) "prefix:urlinktranslator"
155154) "redirect:wiki-irc"
156155) "prefix:validator"
157156) "prefix:slumpartikel"
158157) "redirect:ipp"
159158) "prefix:huggle"
160159) "prefix:ryu"
161160) "prefix:postcardcatfinder"
162161) "prefix:afdstats2"
163162) "prefix:revertstat"
164163) "prefix:zimmerbot"
165164) "prefix:vectorizer"
166165) "redirect:zhdeletionpedia"
167166) "prefix:nakon"
168167) "prefix:friskobot"
169168) "prefix:nn1l2bot"
170169) "redirect:zygimantus-dev"
171170) "prefix:zhwiki"
172171) "redirect:maria"
173172) "prefix:nccroptool"
174173) "redirect:expose-data"
175174) "redirect:contributionsurveyor"
176175) "prefix:bene"
177176) "redirect:nyandata"
178177) "redirect:rezabot"
179178) "prefix:wikidata-trends"
180179) "redirect:archivesearch"
181180) "redirect:bibleversefinder"
182181) "prefix:stockholm-mania"
183182) "prefix:bracketbot"
184183) "redirect:analytalks"
185184) "redirect:mardetanha-dev"
186185) "redirect:videoconvert"
187186) "prefix:globalusagecount"
188187) "redirect:yemen"
189188) "prefix:outreachy-recent-edits-tool"
190189) "redirect:osmlint"
191190) "prefix:wdmap"
192191) "prefix:qrank"
193192) "redirect:wiki-needs-pictures"
194193) "prefix:jawi"
195194) "redirect:abcgames"
196195) "redirect:khanomalumat"
197196) "prefix:plstools"
198197) "prefix:htools"
199198) "prefix:wikipedia-library"
200199) "redirect:bays"
201200) "prefix:alkamidbot"
202201) "prefix:khanomalumat"
203202) "redirect:krdbot"
204203) "redirect:fountain"
205204) "redirect:codeqc"
206205) "redirect:afdstats"
207206) "prefix:fscbot"
208207) "redirect:igloo"
209208) "redirect:primary-sources-v2"
210209) "prefix:germancontributioncounts"
211210) "prefix:wmf-sitematrix"
212211) "prefix:citation-template-filling"
213212) "redirect:arkivbot"
214213) "prefix:matsubot"
215214) "prefix:ancestors2"
216215) "prefix:simplewikt"
217216) "prefix:projektneuheiten-feed"
218217) "prefix:navlink-recommendation"
219218) "redirect:alex-wiki"
220219) "prefix:urbanecmbot"
221220) "redirect:rangecontrib"
222221) "prefix:fun"
223222) "prefix:bibleversefinder"
224223) "prefix:zayenbot"
225224) "prefix:mohib"
226225) "prefix:bub"
227226) "prefix:arkivbot"
228227) "redirect:geophotoreq"
229228) "redirect:archaeo"
230229) "prefix:title-search"
231230) "redirect:amdb"
232231) "prefix:enhourly"
233232) "prefix:contributionsurveyor"
234233) "prefix:dexibotnet"
235234) "prefix:man-pages"
236235) "redirect:wahldiagramm"
237236) "redirect:multichill"
238237) "prefix:vvoters"
239238) "redirect:dimensioner"
240239) "redirect:mjbmrbot"
241240) "prefix:zhnotofu"
242241) "redirect:shuaib"
243242) "redirect:edcounter"
244243) "redirect:botwikiawk"
245244) "prefix:archaeo"
246245) "redirect:freddy2001"
247246) "redirect:ruprecht"
248247) "redirect:mohib"
249248) "redirect:gerakitools"
250249) "prefix:yemen"
251250) "redirect:t187305-demo"
252251) "prefix:neechalbot"
253252) "redirect:stockholm-mania"
254253) "prefix:historyview"
255254) "redirect:tedbot"
256255) "prefix:mjbmr"
257256) "redirect:urduspellchecker"
258257) "prefix:blame"
259258) "redirect:navlink-recommendation"
260259) "redirect:npp"
261260) "prefix:commons-android-app"
262261) "prefix:betacommand-dev"
263262) "redirect:ideasbot"
264263) "prefix:steinsplitter2"
265264) "prefix:fr-wikiversity-ns"
266265) "redirect:bibleversefinder2"
267266) "prefix:zoomviewer"
268267) "redirect:urbanecmbot"
269268) "redirect:embeddedincount"
270269) "prefix:ft"
271270) "prefix:analytics"
272271) "prefix:checkdictation-fa"
273272) "redirect:wikidata-timeline"
274273) "prefix:dibot"
275274) "prefix:urdubot"
276275) "prefix:codeqc"
277276) "redirect:csbot"
278277) "prefix:germancon-mobile"
279278) "redirect:npp-lv"
280279) "redirect:antigng-bot"
281280) "redirect:mp"
282281) "redirect:wdmap"
283282) "prefix:labelimgohs"
284283) "redirect:cp"
285284) "redirect:redirecter"
286285) "redirect:vtwo"
287286) "prefix:dimensioner"
288287) "prefix:dykstats"
289288) "redirect:jitrixis-test"
290289) "redirect:languageproofing-ui"
291290) "prefix:aleph"
292291) "prefix:ytcleaner"
293292) "redirect:dschwenbot"
294293) "prefix:ruwikisource"
295294) "redirect:blockyquery"
296295) "prefix:redpanda"
297296) "prefix:map-search"
298297) "prefix:sbot"
299298) "prefix:mardetanha-dev"
300299) "prefix:chie-bot"
301300) "redirect:spellcheck"
302301) "redirect:centralnotice-bannergenerator"
303302) "prefix:canarybot"
304303) "prefix:dtcheck"
305304) "redirect:sigma"
306305) "prefix:ruprecht"
307306) "redirect:vocabulary-index"
308307) "redirect:historyview"
309308) "redirect:croptool"
310309) "redirect:wd-shex-infer"
311310) "redirect:cil2"
312311) "prefix:maria"
313312) "prefix:croptool"
314313) "prefix:drewbot"
315314) "prefix:embeddedincount"
316315) "redirect:itemfinder"
317316) "redirect:mrmetadata"
318317) "redirect:gendergapdashboard"
319318) "prefix:ramp2"
320319) "prefix:bibleversefinder2"
321320) "prefix:contrabandapp"
322321) "redirect:etwikt"
323322) "prefix:zygserv"
324323) "prefix:zumraband"
325324) "redirect:nakon"
326325) "redirect:video-cut-tool"
327326) "prefix:catscore"
328327) "prefix:wp-signpost"
329328) "prefix:tulsibot"
330329) "redirect:newbie-uploads"
331330) "redirect:khanamalumat"
332331) "prefix:osmlint"
333332) "redirect:cyberbot"
334333) "redirect:quality-assisted-editor"
335334) "prefix:david-tool"
336335) "redirect:ytcleaner"
337336) "prefix:blockyquery"
338337) "redirect:zygserv"
339338) "redirect:outreachy-userrank"
340339) "prefix:copywhat"
341340) "redirect:matsubot"
342341) "prefix:npp"
343342) "redirect:furutani"
344343) "redirect:crocodylia"
345344) "redirect:ocr4wikisource"
346345) "prefix:mp"
347346) "prefix:pinyin-wiki"
348347) "prefix:igloo"
349348) "prefix:abcgames"
350349) "redirect:query2map"
351350) "prefix:gerakibot"
352351) "redirect:wikitrends"
353352) "prefix:gns"
354353) "redirect:fscbot"
355354) "redirect:pinyin-wiki"
356355) "prefix:gorlingor"
357356) "redirect:vvoters"
358357) "redirect:studiesworld"
359358) "redirect:chocobot"
360359) "prefix:mono"
361360) "prefix:abbe98tools"
362361) "redirect:revertstat"
363362) "redirect:gutrs"
364363) "redirect:every-other-wiki-has"
365364) "redirect:coord"
366365) "redirect:simplewikt"
367366) "redirect:mono"
368367) "redirect:derivative"
369368) "redirect:declare"
370369) "redirect:mdwiki"
371370) "prefix:outreachy-userrank"
372371) "prefix:npp-lv"
373372) "prefix:my-first-pywikibot-tool"
374373) "prefix:quality-assisted-editor"
375374) "prefix:xtools-mab-dev"
376375) "prefix:rezabot"
377376) "redirect:steinsplitter"
378377) "prefix:every-other-wiki-has"
379378) "redirect:copywhat"
380379) "prefix:spellcheck"
381380) "prefix:ores-afc"
382381) "prefix:pb"
383382) "prefix:soweego"
384383) "hi"
385384) "redirect:zhnotofu"
386385) "redirect:xmlfeed"
387386) "redirect:anno"
388387) "prefix:cp"
389388) "redirect:validator"
390389) "redirect:ipinfo"
391390) "redirect:redpanda"
392391) "redirect:morph"
393392) "prefix:ideasbot"
394393) "redirect:blockcalc"
395394) "redirect:test-vvv"
396395) "redirect:autopromote-status"
397396) "redirect:zayenbot"
398397) "prefix:ocr4wikisource"
399398) "redirect:manypedia"
400399) "redirect:calling-card"
401400) "prefix:lonelylinks"
402401) "prefix:ipython"
403402) "redirect:my-first-pywikibot-tool"
404403) "redirect:hrwiki"
405404) "prefix:osmviews"
406405) "prefix:wikidata-nolabels"
407406) "prefix:declare"
408407) "prefix:cil2"
409408) "redirect:otrs-helper"
410409) "redirect:raymond"
411410) "prefix:amdb"
412411) "redirect:abbe98tools"
413412) "prefix:autopromote-status"
414413) "prefix:raymond"
415414) "redirect:david-tool"
416415) "redirect:shuaib-bot"
417416) "redirect:germancontributioncounts"
418417) "prefix:wikidata-compare"
419418) "redirect:tessdata"
420419) "redirect:ores-afc"
421420) "redirect:suggestbot"
422421) "prefix:jitrixis-test"
423422) "prefix:khanamalumat"
424423) "prefix:archivesearch"
425424) "prefix:urduspellchecker"
426425) "redirect:dibot"
427426) "redirect:ext-lnk-discover"
428427) "prefix:phetools"
429428) "prefix:imagery"
430429) "redirect:dykautobot"
431430) "redirect:kaleem-bot-i"
432431) "redirect:zimmerbot"
433432) "redirect:gtirloni-sandbox"
434433) "redirect:fr-wikiversity-ns"
435434) "prefix:dexbot"
436435) "prefix:crocodylia"
437436) "redirect:outreachy-recent-edits-tool"
438437) "redirect:articles-by-lat-lon-without-images"
439438) "prefix:mbh"
440439) "redirect:man-pages"
441440) "prefix:itemlister"
442441) "redirect:inactiveadmins"
443442) "redirect:orphan-groups"
444443) "prefix:vocabulary-index"
445444) "redirect:jembot"
446445) "redirect:fun"
447446) "prefix:kasparbot"
448447) "redirect:mavrikant"
449448) "redirect:soweego"
450449) "prefix:energybot"
451450) "redirect:imagery"
452451) "redirect:nccroptool"
453452) "prefix:plaintexteditcounter"
454453) "prefix:tedbot"
455454) "prefix:nyandata"
456455) "redirect:wikidata-compare"
457456) "redirect:mdaniels-refill-ng"
458457) "redirect:germancon-mobile"
459458) "redirect:covid-obit"
460459) "redirect:ahechtbot"
461460) "redirect:wm-metrics"
462461) "prefix:ooui-debug"
463462) "redirect:bene"
464463) "redirect:itemlister"
465464) "redirect:mostlinkedmissing"
466465) "redirect:blogconverter"
467466) "prefix:orpheus"
468467) "redirect:shumariyat"
469468) "redirect:checkdictation-fa"
470469) "prefix:studiesworld"
471470) "redirect:missingpages"
472471) "redirect:wp-signpost"
473472) "redirect:dispenser"
474473) "prefix:bays"
475474) "prefix:tasmania"
476475) "prefix:sge-status"
477476) "redirect:phetools"
478477) "prefix:orphan-groups"
479478) "prefix:anno"
480479) "redirect:wnegar"
481480) "redirect:governance-timeline"
482481) "redirect:dykstats"
483482) "prefix:video-cut-tool"
484483) "redirect:lonelylinks"
485484) "prefix:wikiedudashboard-test"
486485) "prefix:liangent-misc"
487486) "redirect:map-search"
488487) "redirect:commons-android-app"
489488) "prefix:os"
490489) "redirect:kaleem-bot"
491490) "redirect:tasmania"
492491) "prefix:derivative"
493492) "prefix:calling-card"
494493) "redirect:dexibotnet"
495494) "redirect:pb"
496495) "redirect:enhourly"
497496) "prefix:watchr"
498497) "prefix:convert"
499498) "prefix:expose-data"
500499) "prefix:patrolstats"
501500) "prefix:vtwo"
502501) "redirect:mjbmr"
503502) "redirect:wmf-sitematrix"
504503) "redirect:mbh"
505504) "prefix:otrs-helper"
506505) "prefix:gutrs"
507506) "redirect:quality-analyzer"
508507) "redirect:canary"
509508) "redirect:simple"
510509) "prefix:geophotoreq"
511510) "prefix:blogconverter"
512511) "prefix:botwikiawk"
513512) "prefix:honeypot95"
514513) "redirect:afdstats2"
515514) "prefix:owintes"
516515) "prefix:render"
517516) "prefix:kaleem-bot-i"
518517) "redirect:alkamidbot"
519518) "prefix:lolrrit-wm"
520519) "redirect:gns"
521520) "redirect:contrabandapp"
522521) "redirect:dtcheck"
523522) "prefix:canary"
524523) "prefix:manypedia"
525524) "redirect:laaknortools"
526525) "redirect:chie-bot"
527526) "redirect:footygen"
528527) "redirect:zoomviewer"
529528) "prefix:newbie-uploads"
530529) "prefix:test-vvv"
531530) "redirect:htools"
532531) "prefix:orator-matcher"
533532) "redirect:kasparbot"
534533) "redirect:ptbot"
535534) "prefix:inactiveadmins"
536535) "prefix:javatest"
537536) "redirect:parliament-diagram-generator"
538537) "redirect:wikipedia-contributor-locations"
539538) "redirect:relgen"
540539) "redirect:metmuseum"
541540) "redirect:urlinktranslator"
542541) "redirect:wptestblog2"
543542) "prefix:unpatrollededitstats"
544543) "prefix:iplookup"
545544) "redirect:ooui-debug"
546545) "prefix:liangent-toolserver"
547546) "redirect:betacommand-dev"
548547) "redirect:wmds-archive"
549548) "prefix:coord"
550549) "prefix:krdbot"
551550) "prefix:submitter"
552551) "prefix:missingpages"
553552) "prefix:linkchecker"
554553) "prefix:enwnbot"
555554) "prefix:valhallasw-test-redis"
556555) "prefix:veblenbot"
557556) "prefix:dschwenbot"
558557) "prefix:sigma"
559558) "prefix:ahechtbot"
560559) "prefix:family"
561560) "prefix:dykautobot"
562561) "redirect:ramp2"
563562) "prefix:antigng-bot"
564563) "prefix:mdaniels-refill-ng"
565564) "prefix:mywikitool"
566565) "prefix:shuaib"
567566) "redirect:globalusagecount"
568567) "prefix:governance-timeline"
569568) "redirect:userimpact"
570569) "prefix:redirecter"
571570) "prefix:artemisia"
572571) "redirect:wikiedudashboard-test"
573572) "redirect:gorlingor"
574573) "prefix:analytalks"
575574) "prefix:ipp"
576575) "redirect:convert"
577576) "redirect:patrolstats"
578577) "redirect:bracketbot"
579578) "redirect:labelimgohs"
580579) "prefix:wiki-talk"
581580) "redirect:wikidata-nolabels"
582581) "prefix:wiki-as-git"
583582) "redirect:dexbot"
584583) "redirect:artemisia"
585584) "prefix:db"
586585) "prefix:gendergapdashboard"
587586) "redirect:wikipedia-library"
588587) "redirect:vectorizer"
589588) "redirect:steinsplitter2"
590589) "prefix:toolschecker-ge-ws"
591590) "redirect:ancestors2"
592591) "redirect:title-search"
593592) "prefix:primary-sources-v2"
594593) "redirect:ryu"
595594) "redirect:owintes"
596595) "redirect:db"
597596) "redirect:listpages"

Details

SubjectRepoBranchLines +/-
tools-manifests: don't collect statsd metricsoperations/software/tools-manifestmaster+1 -26
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero created this task.Feb 14 2023, 9:51 AM
taavi edited projects, added Toolforge, Sustainability (Incident Followup); removed Wikimedia-Incident, Cloud-VPS.Feb 14 2023, 10:09 AM
aborrero added a comment.Feb 14 2023, 10:48 AM

I think the actual fix here may be to fix T329467: remove webservicemonitor (down due to DNS errors) and let it recover webservices instead of me doing manually

gerritbot added a comment.Feb 14 2023, 11:16 AM

Change 889085 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/software/tools-manifest@master] tools-manifests: don't collect statsd metrics

https://gerrit.wikimedia.org/r/889085

gerritbot added a project: Patch-For-Review.Feb 14 2023, 11:16 AM
gerritbot added a comment.Feb 14 2023, 11:54 AM

Change 889085 merged by jenkins-bot:

[operations/software/tools-manifest@master] tools-manifests: don't collect statsd metrics

https://gerrit.wikimedia.org/r/889085

Stashbot added a comment.Feb 14 2023, 12:02 PM

Mentioned in SAL (#wikimedia-cloud) [2023-02-14T12:02:53Z] <arturo> included tools-manifests 0.25 in toolsbeta-buster aptly repo (T329611, T329467, T244809)

Stashbot mentioned this in T329467: remove webservicemonitor (down due to DNS errors).Feb 14 2023, 12:02 PM
Stashbot mentioned this in T244809: Remove or fix stats collecting from tools-manifest (webservice-monitor).
Stashbot added a comment.Feb 14 2023, 12:09 PM

Mentioned in SAL (#wikimedia-cloud) [2023-02-14T12:09:57Z] <arturo> included tools-manifests 0.25 in tools-buster aptly repo, deploying it now! (T329611, T329467, T244809)

Maintenance_bot removed a project: Patch-For-Review.Feb 14 2023, 12:10 PM
Stashbot added a comment.Feb 14 2023, 12:12 PM

Mentioned in SAL (#wikimedia-cloud) [2023-02-14T12:12:10Z] <arturo> the fixed webservicemonitor is starting a bunch of grid webservices (T329611)

aborrero added a comment.Feb 14 2023, 12:13 PM

The webservicemonitor doing its thing:

aborrero@tools-sgecron-2:~$ sudo journalctl -u webservicemonitor.service -f
-- Logs begin at Tue 2023-02-14 04:45:01 UTC. --
Feb 14 12:10:59 tools-sgecron-2 sudo[14101]: pam_unix(sudo:session): session closed for user tools.mdaniels-refill-ng
Feb 14 12:10:59 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:10:59,236 Started webservice for mdaniels-refill-ng
Feb 14 12:10:59 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:10:59,329 Starting webservice for tool every-other-wiki-has
Feb 14 12:10:59 tools-sgecron-2 sudo[15201]:     root : TTY=unknown ; PWD=/ ; USER=tools.every-other-wiki-has ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:10:59 tools-sgecron-2 sudo[15201]: pam_unix(sudo:session): session opened for user tools.every-other-wiki-has by (uid=0)
Feb 14 12:11:13 tools-sgecron-2 sudo[15201]: pam_unix(sudo:session): session closed for user tools.every-other-wiki-has
Feb 14 12:11:13 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:13,771 Started webservice for every-other-wiki-has
Feb 14 12:11:14 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:14,215 Starting webservice for tool projektneuheiten-feed
Feb 14 12:11:14 tools-sgecron-2 sudo[17183]:     root : TTY=unknown ; PWD=/ ; USER=tools.projektneuheiten-feed ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:11:14 tools-sgecron-2 sudo[17183]: pam_unix(sudo:session): session opened for user tools.projektneuheiten-feed by (uid=0)
Feb 14 12:11:28 tools-sgecron-2 sudo[17183]: pam_unix(sudo:session): session closed for user tools.projektneuheiten-feed
Feb 14 12:11:28 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:28,553 Started webservice for projektneuheiten-feed
Feb 14 12:11:28 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:28,571 Starting webservice for tool betacommand-dev
Feb 14 12:11:28 tools-sgecron-2 sudo[17894]:     root : TTY=unknown ; PWD=/ ; USER=tools.betacommand-dev ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:11:28 tools-sgecron-2 sudo[17894]: pam_unix(sudo:session): session opened for user tools.betacommand-dev by (uid=0)
Feb 14 12:11:44 tools-sgecron-2 sudo[17894]: pam_unix(sudo:session): session closed for user tools.betacommand-dev
Feb 14 12:11:44 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:44,178 Started webservice for betacommand-dev
Feb 14 12:11:44 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:44,229 Starting webservice for tool catscore
Feb 14 12:11:44 tools-sgecron-2 sudo[18651]:     root : TTY=unknown ; PWD=/ ; USER=tools.catscore ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:11:44 tools-sgecron-2 sudo[18651]: pam_unix(sudo:session): session opened for user tools.catscore by (uid=0)
Feb 14 12:11:58 tools-sgecron-2 sudo[18651]: pam_unix(sudo:session): session closed for user tools.catscore
Feb 14 12:11:58 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:58,976 Started webservice for catscore
Feb 14 12:11:59 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:11:59,028 Starting webservice for tool contributionsurveyor
Feb 14 12:11:59 tools-sgecron-2 sudo[19363]:     root : TTY=unknown ; PWD=/ ; USER=tools.contributionsurveyor ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:11:59 tools-sgecron-2 sudo[19363]: pam_unix(sudo:session): session opened for user tools.contributionsurveyor by (uid=0)
Feb 14 12:12:13 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:12:13,370 Started webservice for contributionsurveyor
Feb 14 12:12:13 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:12:13,419 Starting webservice for tool vtwo
Feb 14 12:12:13 tools-sgecron-2 sudo[21404]:     root : TTY=unknown ; PWD=/ ; USER=tools.vtwo ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:12:13 tools-sgecron-2 sudo[21404]: pam_unix(sudo:session): session opened for user tools.vtwo by (uid=0)
Feb 14 12:12:29 tools-sgecron-2 sudo[21404]: pam_unix(sudo:session): session closed for user tools.vtwo
Feb 14 12:12:29 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:12:29,127 Started webservice for vtwo
Feb 14 12:12:29 tools-sgecron-2 collector-runner[4667]: 2023-02-14 12:12:29,147 Starting webservice for tool commons-android-app
Feb 14 12:12:29 tools-sgecron-2 sudo[22161]:     root : TTY=unknown ; PWD=/ ; USER=tools.commons-android-app ; COMMAND=/bin/bash -c /usr/bin/webservice --backend=gridengine --release=buster restart
Feb 14 12:12:29 tools-sgecron-2 sudo[22161]: pam_unix(sudo:session): session opened for user tools.commons-android-app by (uid=0)
aborrero triaged this task as High priority.Feb 14 2023, 12:16 PM
aborrero moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q3) board.
aborrero added a comment.Feb 14 2023, 1:11 PM

Some additional information.

I researched the meaning of the redis data. Entries with prefix are the ones indicating running webservices in the grid (see modules/dynamicproxy/files/urlproxy.lua)

Therefore the redis data references 303 running weservices in the grid.

We can use 2 ways to evaluate how many webservices are there:

aborrero@tools-sgegrid-master:~$ cat stats.sh 
#!/bin/bash
total=0
for i in $(seq 13 32) ; do
	n=$(qhost -j -h tools-sgeweblight-10-$i.tools.eqiad1.wikimedia.cloud 2>/dev/null | grep ^[[:space:]]*[[:digit:]] | wc -l || echo 0)
	total=$(($total+$n))
done
echo $total
aborrero@tools-sgegrid-master:~$ bash stats.sh 
240
aborrero closed this task as Resolved.Feb 14 2023, 1:26 PM
<taavi> Feb 14 13:21:28 tools-sgecron-2 collector-runner[4667]: 2023-02-14 13:21:28,798 Service monitor run completed, 283 webservices restarted
dcaro moved this task from In progress to Done on the cloud-services-team (FY2022/2023-Q3) board.Feb 14 2023, 1:32 PM
Jdforrester-WMF removed a parent task: T329535: Cloud Ceph outage 2023-02-13.Feb 14 2023, 5:44 PM
Jdforrester-WMF added a subtask: T329535: Cloud Ceph outage 2023-02-13.
dcaro closed subtask T329535: Cloud Ceph outage 2023-02-13 as Resolved.Oct 27 2023, 3:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL