Page MenuHomePhabricator
Create Task
Maniphest T268524

Upgrade Parsoid servers to buster
Closed, ResolvedPublic
Actions

Description

Since parse2001 is a canary server and has no traffic, I suggest we use it to test at least the buster installation and check if everything is in order. I understand that if the work are doing for the upgrading the appservers/api mediawiki servers, we do not expect many surprises here.

ParsoidJS is not in use anymore, so this is a chance to remove it from the servers as well

Q2

  • try to have finished reimaging parse2001 to buster
  • run some tests

Q3

  • reimage scandium, testing setup
  • remove parsoidJS from production
  • reimage codfw
  • reimage eqiad

Details

SubjectRepoBranchLines +/-
profile::parsoid: remove parsoid class from parsoid profileoperations/puppetproduction+0 -20
httpbb: add tests for parsoid serversoperations/puppetproduction+30 -0
DHCP: switch parse2001 to use buster installeroperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedjijikiT268524 Upgrade Parsoid servers to buster
DeclinedNoneT245888 Rename wtp* servers to parse* (Parsoid PHP servers)
 ResolvedDzahnT268248 upgrade scandium to buster

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 30 2020, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-30T20:39:32Z] <mutante> reimaging parse2001 (parsoid canary) with buster (T268524)

ops-monitoring-bot added a comment.Nov 30 2020, 9:22 PM

Completed auto-reimage of hosts:

['parse2001.codfw.wmnet']

and were ALL successful.

Dzahn added a comment.EditedNov 30 2020, 9:53 PM

@jijiki parse2001 is now on buster. I noticed no puppet errors or warnings. It went surprisingly smooth. Icinga also looking good. It is currently back in "pooled=no" (from inactive).

I also scap pulled.

Dzahn updated the task description. (Show Details)Nov 30 2020, 9:54 PM
jijiki added a subscriber: ssastry.Nov 30 2020, 9:55 PM

That sounds lovely! Are there any sanity tests we could possibly do? Maybe @ssastry can give us an idea?

Dzahn added projects: serviceops, Parsoid.Nov 30 2020, 9:55 PM
Dzahn added a comment.Nov 30 2020, 10:04 PM

So I used httpbb to test it and we still have this issue, which looks like firewalling..to my surprise.

P13485 testing parse2001
1[deploy1001:~] $ httpbb --hosts parse2001.codfw.wmnet /srv/deployment/httpbb-tests/appserver/baseurls.yaml
2Sending to parse2001.codfw.wmnet...
3https://de.wikipedia.org/wiki/Wikipedia:Hauptseite (/srv/deployment/httpbb-tests/appserver/baseurls.yaml:25)
4 ERROR: HTTPSConnectionPool(host='parse2001.codfw.wmnet', port=443): Read timed out. (read timeout=10)
5https://meta.wikimedia.org/wiki/List_of_Wikipedias (/srv/deployment/httpbb-tests/appserver/baseurls.yaml:98)
6 ERROR: HTTPSConnectionPool(host='parse2001.codfw.wmnet', port=443): Read timed out. (read timeout=10)
7https://fr.wikipedia.org/wiki/K.A.Z (/srv/deployment/httpbb-tests/appserver/baseurls.yaml:31)
8 ERROR: HTTPSConnectionPool(host='parse2001.codfw.wmnet', port=443): Read timed out. (read timeout=10)
9===
10ERRORS: 19 requests attempted to parse2001.codfw.wmnet. Errors connecting to 1 host.
11
12
13[deploy1001:~] $ httpbb --hosts parse2002.codfw.wmnet /srv/deployment/httpbb-tests/appserver/baseurls.yaml
14Sending to parse2002.codfw.wmnet...
15PASS: 19 requests sent to parse2002.codfw.wmnet. All assertions passed.

Stashbot added a comment.Nov 30 2020, 10:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-30T22:14:34Z] <mutante> parse2001 - systemctl restart ferm - had to restart ferm after reimaging (though there weren't any alerts about that) but it fixed running httpbb tests on it (T268524)

Dzahn added a subscriber: RLazarus.Nov 30 2020, 10:16 PM

@jijiki @RLazarus I had to manually restart ferm (meh! no alerts about that and should not happen) but that made the httpbb tests work now:

[deploy1001:~] $ httpbb --hosts parse2001.codfw.wmnet /srv/deployment/httpbb-tests/appserver/baseurls.yaml 
Sending to parse2001.codfw.wmnet...
PASS: 19 requests sent to parse2001.codfw.wmnet. All assertions passed.
...
[deploy1001:~] $ httpbb --hosts parse2001.codfw.wmnet /srv/deployment/httpbb-tests/appserver/test_main.yaml 
Sending to parse2001.codfw.wmnet...
PASS: 23 requests sent to parse2001.codfw.wmnet. All assertions passed.
jijiki added a comment.Dec 2 2020, 11:55 AM

@Dzahn is there something we need to fix on puppet ?

Dzahn added a comment.Dec 2 2020, 10:34 PM

@jijiki Not sure, I think in the past the "official" instructions included rebooting once and that would also fix it.

MoritzMuehlenhoff subscribed.Dec 3 2020, 8:07 AM

I checked the journalctl of ferm on parse2001 and there are no errors logged, so it's not a case of Ferm failing to start (via syntax errors, failing DNS lookup or whatever). But be should get to the bottom of this, let's maybe reimage also parse2002 to show whether it's a reproducible error and if it happens again, keep it in the broken state to debug further?

Dzahn added a comment.Dec 3 2020, 6:02 PM
In T268524#6665888, @MoritzMuehlenhoff wrote:

let's maybe reimage also parse2002 to show whether it's a reproducible error and if it happens again, keep it in the broken state to debug further?

I can just repeat the reimaging process on 2001 as well. But then if it happens again leave it in the broken state on purpose and just downtime everything. But no strong preference.

jijiki added a comment.Dec 3 2020, 6:08 PM
In T268524#6667353, @Dzahn wrote:
In T268524#6665888, @MoritzMuehlenhoff wrote:

let's maybe reimage also parse2002 to show whether it's a reproducible error and if it happens again, keep it in the broken state to debug further?

I can just repeat the reimaging process on 2001 as well. But then if it happens again leave it in the broken state on purpose and just downtime everything. But no strong preference.

Sounds good!

Dzahn added a comment.Dec 3 2020, 6:10 PM

Alright, will do that and start in a couple minutes.

MoritzMuehlenhoff added a comment.Dec 3 2020, 6:22 PM
In T268524#6667353, @Dzahn wrote:
In T268524#6665888, @MoritzMuehlenhoff wrote:

let's maybe reimage also parse2002 to show whether it's a reproducible error and if it happens again, keep it in the broken state to debug further?

I can just repeat the reimaging process on 2001 as well. But then if it happens again leave it in the broken state on purpose and just downtime everything. But no strong preference.

Ack, if it end up broken again, I'll poke at it tomorrow.

Stashbot added a comment.Dec 3 2020, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-03T19:11:12Z] <mutante> depooling parse2001 and repeating auto-reimage to see if ferm issue is repeatable (T268524)

ops-monitoring-bot added a comment.Dec 3 2020, 7:21 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

parse2001.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012031920_dzahn_27732_parse2001_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2020, 8:10 PM

Completed auto-reimage of hosts:

['parse2001.codfw.wmnet']

and were ALL successful.

Dzahn added a comment.EditedDec 3 2020, 9:45 PM

What can I say.. it did not happen this time. Reimaged, ran puppet.. waited a bit, checked Icinga.. all green with the following exceptions:

  • Ensure local MW versions match expected deployment (fixed after running a scap pull because earlier there was a deployment)
  • mediawiki-installation DSH group (will be fixed by pooled=no from pooled=inactive but takes a while)
  • PHP opcache health (not sure about the "below 99.85%" needing any action)

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=parse2001&scroll=242

But either way, no ferm restart involved and things looking good.

MoritzMuehlenhoff added a comment.Dec 4 2020, 8:21 AM
In T268524#6667989, @Dzahn wrote:

What can I say.. it did not happen this time. Reimaged, ran puppet.. waited a bit, checked Icinga.. all green with the following exceptions:

(..)

But either way, no ferm restart involved and things looking good.

Sound good, I'd say we keep it as that, and if we ever run into this error again, keep it broken and ping or John for investigating,

ssastry moved this task from Needs Triage to Non-Parsing-Team Tasks on the Parsoid board.Dec 7 2020, 1:13 AM
Stashbot added a comment.Dec 9 2020, 11:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-09T23:21:54Z] <mutante> repooling parse2001 after buster reimage - T268524

jijiki added a comment.Dec 10 2020, 3:59 PM

@ssastry is there someway we could check that parse2001, which is running on buster now, works as expected?

ssastry added a comment.Dec 10 2020, 4:12 PM

Run a few curl commands like these but while using parse2001 as a proxy. Here is the equivalent for scandium itself:

curl -L -x http://scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Hospet

I tried with parse2001 / parse2002 but that failed to resolve:

ssastry@scandium:~$ curl -L -x http://parse2002.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Hospet
curl: (5) Could not resolve proxy: parse2002.eqiad.wmnet
ssastry@scandium:~$ curl -L -x http://parse2001.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Hospet
curl: (5) Could not resolve proxy: parse2001.eqiad.wmnet

Not sure if I got the server names wrong, but that is what I would try.

For bonus points, verify that the HTML generated on parse2001 and parse2002 is identical. And try on a couple different wikis, (enwiki plus other lang wikis). Can be easily scripted.

ssastry added a comment.Dec 10 2020, 4:20 PM

Ah, because parse2001 and parse2002 are codfw, not eqiad. Anyway, here goes:

ssastry@scandium:~$ curl -L -x http://parse2001.codfw.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Hospet > /tmp/p1.html
...
ssastry@scandium:~$ curl -L -x http://parse2002.codfw.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Hospet > /tmp/p2.html
...
ssastry@scandium:~$ diff /tmp/p1.html /tmp/p2.html
ssastry@scandium:~$
MBinder_WMF edited projects, added Parsoid (Tracking); removed Parsoid.Dec 10 2020, 8:12 PM
gerritbot added a comment.Dec 14 2020, 10:03 PM

Change 648383 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] httpbb: add tests for parsoid servers

https://gerrit.wikimedia.org/r/648383

gerritbot added a project: Patch-For-Review.Dec 14 2020, 10:03 PM
Dzahn added a comment.Dec 14 2020, 10:15 PM

@jijiki @ssastry I wrote the following httpbb tests based on that:

# tests for parsoid appservers
# hosts: parse*, wtp*
http://en.wikipedia.org:
- path: /w/rest.php/en.wikipedia.org/v3/page/html/Hospet
  assert_status: 302 
- path: /w/rest.php/en.wikipedia.org/v3/page/html/Hospet/992733907
  assert_status: 200 
  assert_body_contains: data-parsoid
http://de.wikipedia.org:
- path: /w/rest.php/de.wikipedia.org/v3/page/html/Karnataka
  assert_status: 302 
- path: /w/rest.php/de.wikipedia.org/v3/page/html/Karnataka/206238030
  assert_status: 200 
  assert_body_contains: data-parsoid
http://es.wikipedia.org:
- path: /w/rest.php/es.wikipedia.org/v3/page/html/Bangalore
  assert_status: 302 
- path: /w/rest.php/es.wikipedia.org/v3/page/html/Bangalore/129309635
  assert_status: 200 
  assert_body_contains: data-parsoid
http://it.wikipedia.org:
- path: /w/rest.php/it.wikipedia.org/v3/page/html/Mysore
  assert_status: 302 
- path: /w/rest.php/it.wikipedia.org/v3/page/html/Mysore/112864552
  assert_status: 200 
  assert_body_contains: data-parsoid

So as suggested the "Hospet" page on en.wiki and then some others on de/es/it. The first URL is always a 302 and the second one a 200 where we check if "data-parsoid" is in the body.

They all pass on parse2001.

[deploy1001:~] $ httpbb --hosts parse2001.codfw.wmnet /home/dzahn/test_parse.yaml 
Sending to parse2001.codfw.wmnet...
PASS: 8 requests sent to parse2001.codfw.wmnet. All assertions passed.

If that looks good we can merge the patch that will install it on cumin and deplyment servers for all to use.

Dzahn updated the task description. (Show Details)Dec 14 2020, 10:26 PM
gerritbot added a comment.Dec 15 2020, 9:46 PM

Change 648383 merged by Dzahn:
[operations/puppet@production] httpbb: add tests for parsoid servers

https://gerrit.wikimedia.org/r/648383

Maintenance_bot removed a project: Patch-For-Review.Dec 15 2020, 10:10 PM
Dzahn mentioned this in T245888: Rename wtp* servers to parse* (Parsoid PHP servers).Feb 23 2021, 5:51 PM
Aklapper added a subtask: T245888: Rename wtp* servers to parse* (Parsoid PHP servers).Mar 1 2021, 6:36 AM
Dzahn closed subtask T245888: Rename wtp* servers to parse* (Parsoid PHP servers) as Declined.Mar 2 2021, 12:19 AM
Dzahn updated the task description. (Show Details)Mar 20 2021, 12:14 AM
Dzahn added a subtask: T268248: upgrade scandium to buster.

scandium was upgraded to buster in T268248

taavi mentioned this in T278664: Upgrade deployment-prep MediaWiki clusters to Buster.Mar 29 2021, 5:34 AM
jijiki added a comment.Mar 30 2021, 4:50 PM

@ssastry do we still need parsoid JS running in the parsoid servers? This is a good opportunity to clean this up. I am running into this issue T245757#6953720 when I tried to re-image one of them

ssastry added a comment.Mar 30 2021, 4:52 PM

No.

taavi subscribed.Mar 30 2021, 7:54 PM
jijiki updated the task description. (Show Details)Mar 31 2021, 1:24 PM
gerritbot added a comment.Mar 31 2021, 1:28 PM

Change 676068 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] profile::parsoid: remove parsoid class from parsoid profile

https://gerrit.wikimedia.org/r/676068

gerritbot added a project: Patch-For-Review.Mar 31 2021, 1:28 PM
gerritbot added a comment.Mar 31 2021, 1:33 PM

Change 676071 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] modules: remove parsoidJS puppet module

https://gerrit.wikimedia.org/r/676071

jijiki added a subtask: T279059: Remove parsoidJS leftovers from production.Apr 1 2021, 1:47 PM
Stashbot added a comment.Apr 1 2021, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-01T19:30:03Z] <mutante> depooled parse2001 because on train deployment it caused "MWException: No localisation cache found for English" and then "HTTP CRITICAL: HTTP/1.1 500 Internal Server Error" (T268524)

Stashbot added a comment.Apr 1 2021, 7:37 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-01T19:37:51Z] <mutante> pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524)

gerritbot added a comment.Apr 2 2021, 8:15 AM

Change 676068 abandoned by Effie Mouzeli:

[operations/puppet@production] profile::parsoid: remove parsoid class from parsoid profile

Reason:

in favour of Iab525436f3ae10327a2ac9c31f91009ae451611c

https://gerrit.wikimedia.org/r/676068

Dzahn added a comment.Apr 2 2021, 10:16 PM
In T268524#6966291, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2021-04-01T19:37:51Z] <mutante> pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524)

For this also see T273334

ops-monitoring-bot added a comment.Apr 8 2021, 3:45 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1025.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104081545_jiji_14346.log.

jijiki updated the task description. (Show Details)Apr 8 2021, 3:48 PM
jijiki updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 8 2021, 4:58 PM

Completed auto-reimage of hosts:

['wtp1025.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 7:37 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2002.codfw.wmnet', 'parse2003.codfw.wmnet', 'parse2004.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104130735_jiji_22041.log.

ops-monitoring-bot added a comment.Apr 13 2021, 8:21 AM

Completed auto-reimage of hosts:

['parse2002.codfw.wmnet', 'parse2003.codfw.wmnet', 'parse2004.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 8:51 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2005.codfw.wmnet', 'parse2006.codfw.wmnet', 'parse2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104130850_jiji_8219.log.

ops-monitoring-bot added a comment.Apr 13 2021, 9:34 AM

Completed auto-reimage of hosts:

['parse2005.codfw.wmnet', 'parse2006.codfw.wmnet', 'parse2007.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 11:08 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2008.codfw.wmnet', 'parse2009.codfw.wmnet', 'parse2010.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131108_jiji_17443.log.

ops-monitoring-bot added a comment.Apr 13 2021, 11:51 AM

Completed auto-reimage of hosts:

['parse2008.codfw.wmnet', 'parse2009.codfw.wmnet', 'parse2010.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 1:36 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2011.codfw.wmnet', 'parse2012.codfw.wmnet', 'parse2013.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131336_jiji_29644.log.

ops-monitoring-bot added a comment.Apr 13 2021, 1:38 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1026.eqiad.wmnet', 'wtp1027.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131338_jiji_31386.log.

ops-monitoring-bot added a comment.Apr 13 2021, 2:19 PM

Completed auto-reimage of hosts:

['parse2011.codfw.wmnet', 'parse2012.codfw.wmnet', 'parse2013.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 2:51 PM

Completed auto-reimage of hosts:

['wtp1026.eqiad.wmnet', 'wtp1027.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 3:34 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1028.eqiad.wmnet', 'wtp1029.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131534_jiji_15418.log.

ops-monitoring-bot added a comment.Apr 13 2021, 3:39 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2014.codfw.wmnet', 'parse2015.codfw.wmnet', 'parse2016.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131539_jiji_21218.log.

ops-monitoring-bot added a comment.Apr 13 2021, 4:23 PM

Completed auto-reimage of hosts:

['parse2014.codfw.wmnet', 'parse2015.codfw.wmnet', 'parse2016.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 4:49 PM

Completed auto-reimage of hosts:

['wtp1028.eqiad.wmnet', 'wtp1029.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 5:29 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2017.codfw.wmnet', 'parse2018.codfw.wmnet', 'parse2019.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131728_jiji_26705.log.

ops-monitoring-bot added a comment.Apr 13 2021, 5:29 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1030.eqiad.wmnet', 'wtp1031.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131729_jiji_26775.log.

ops-monitoring-bot added a comment.Apr 13 2021, 6:15 PM

Completed auto-reimage of hosts:

['parse2017.codfw.wmnet', 'parse2018.codfw.wmnet', 'parse2019.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 6:45 PM

Completed auto-reimage of hosts:

['wtp1030.eqiad.wmnet', 'wtp1031.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 6:56 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['parse2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131856_jiji_16703.log.

ops-monitoring-bot added a comment.Apr 13 2021, 6:57 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1032.eqiad.wmnet', 'wtp1033.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104131857_jiji_16778.log.

ops-monitoring-bot added a comment.Apr 13 2021, 7:36 PM

Completed auto-reimage of hosts:

['parse2020.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 13 2021, 8:10 PM

Completed auto-reimage of hosts:

['wtp1032.eqiad.wmnet', 'wtp1033.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 14 2021, 10:31 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1034.eqiad.wmnet', 'wtp1035.eqiad.wmnet', 'wtp1036.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104141031_jiji_25538.log.

ops-monitoring-bot added a comment.Apr 14 2021, 11:47 AM

Completed auto-reimage of hosts:

['wtp1034.eqiad.wmnet', 'wtp1035.eqiad.wmnet', 'wtp1036.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 14 2021, 12:02 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1037.eqiad.wmnet', 'wtp1038.eqiad.wmnet', 'wtp1039.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104141202_jiji_13531.log.

ops-monitoring-bot added a comment.Apr 14 2021, 1:19 PM

Completed auto-reimage of hosts:

['wtp1038.eqiad.wmnet', 'wtp1037.eqiad.wmnet', 'wtp1039.eqiad.wmnet']

and were ALL successful.

Dzahn added a comment.EditedApr 14 2021, 8:47 PM

After the mw train we got some Icinga alerts which caused worry among deployers:

20:15 <+icinga-wm> PROBLEM - Ensure local MW versions match expected deployment on wtp1037 is CRITICAL: CRITICAL: 524 mismatched wikiversions 
                   https://wikitech.wikimedia.org/wiki/Application_servers
20:15 <+icinga-wm> PROBLEM - Ensure local MW versions match expected deployment on wtp1038 is CRITICAL: CRITICAL: 524 mismatched wikiversions 
                   https://wikitech.wikimedia.org/wiki/Application_servers
20:17 <+icinga-wm> PROBLEM - Ensure local MW versions match expected deployment on wtp1039 is CRITICAL: CRITICAL: 524 mismatched wikiversions 
                   https://wikitech.wikimedia.org/wiki/Application_servers

That happens when they are in pooled=inactive during reimage which means not being in scap "dsh" groups and getting deployments.

It's not a real issue as long as we remember running scap pull before repooling but the alerts are making people worry something is broken (and then when they try to ssh to the affected servers they see a key change due to the reimages).

I checked they are depooled and manually ran scap pull, which made the alerts recover.

I think the real issue is that this alert should not be there for hosts that are not pooled, but that may not be so easy to do.

Or it's just about expired or failed downtimes from the reimaging cookbook.

edit: yes, it was just about expired downtimes. Also ACKed 'not in mediawiki-installation dsh group' and set downtimes on mw103[7-9] for 24 hours.

ops-monitoring-bot added a comment.Apr 15 2021, 11:01 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1040.eqiad.wmnet', 'wtp1041.eqiad.wmnet', 'wtp1042.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104151100_jiji_16692.log.

ops-monitoring-bot added a comment.Apr 15 2021, 12:18 PM

Completed auto-reimage of hosts:

['wtp1040.eqiad.wmnet', 'wtp1041.eqiad.wmnet', 'wtp1042.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 15 2021, 12:24 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1043.eqiad.wmnet', 'wtp1044.eqiad.wmnet', 'wtp1045.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104151224_jiji_6464.log.

ops-monitoring-bot added a comment.Apr 15 2021, 1:41 PM

Completed auto-reimage of hosts:

['wtp1043.eqiad.wmnet', 'wtp1044.eqiad.wmnet', 'wtp1045.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 15 2021, 2:02 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['wtp1046.eqiad.wmnet', 'wtp1047.eqiad.wmnet', 'wtp1048.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104151402_jiji_27358.log.

ops-monitoring-bot added a comment.Apr 15 2021, 3:20 PM

Completed auto-reimage of hosts:

['wtp1046.eqiad.wmnet', 'wtp1047.eqiad.wmnet', 'wtp1048.eqiad.wmnet']

and were ALL successful.

jijiki closed this task as Resolved.Apr 15 2021, 3:42 PM
jijiki claimed this task.
jijiki updated the task description. (Show Details)
Dzahn awarded a token.Apr 15 2021, 3:47 PM
Izno removed a subtask: T279059: Remove parsoidJS leftovers from production.Jun 1 2021, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL