This is due to a new feature for git-lfs on Buster, so probably only relevant to the ORES nodes (we are going to upgrade them to Buster). I can take care of the new package, and we can avoid to roll it out everywhere in theory.

Mentioned in SAL (#wikimedia-operations) [2022-04-28T08:31:46Z] <elukey> upload scap 4.7.1 to {buster,stretch,bullseye}-wikimedia apt repos - T306998

hi @elukey, thank you for taking care of this task. This release also has a fix for an issue in scap unrelated to git-lfs, so we would like to have it rolled out to every target host.

@jnuche ack perfect! I'll coordinate with Service Ops for the rollout. I'll try to skip the ORES nodes for the moment since they are still on stretch.

Mentioned in SAL (#wikimedia-operations) [2022-04-28T10:15:07Z] <elukey> update scap to 4.7.1 on A:mw-canary or A:parsoid-canary or A:mw-jobrunner-canary - T306998

I tested scap pull on mwdebug* nodes, and only in codfw I noticed a very long step:

10:15:41 Started scap-cdb-rebuild
10:18:51 Finished scap-cdb-rebuild (duration: 03m 10s)

Tried other times to run scap pulland it took few seconds, not sure if it is normal or not so I am going to add a not inhere for a broader feedback.

Mentioned in SAL (#wikimedia-operations) [2022-04-28T10:23:32Z] <elukey> update scap to 4.7.1 on restbase1016 (canary) - T306998

Deployed also to restbase-dev as highlighted on wikitech, all good. I'll wait some hours and then I'll roll it out to the rest of the nodes, excluding the Ores ones.

@elukey my guess is the long scap-cdb-rebuild step was caused by a change in the l10n files. It's most likely not an issue.

@dancy maybe you can give some more insight.

From the scap logs https://logstash.wikimedia.org/goto/d454018867fa2a63300115ecaf60227c spanning the last 24 hours:

Or split by hosts:

mwdebug1001

mwdebug1002

mwdebug2001

mwdebug2002

The mwdebug1001 and mwdebug1002 regenerated the CDB files yesterday as part of the train (I am guessing).

The mwdebug2001 and mwdebug2002 only got regenerated this morning as part of testing the new scap version. There is no other entries over the last 5 days.

Which leads me to the question: why mwdebug2001 and mwdebug2002 did not trigger a cdb rebuild when we ran the train yesterday. Maybe they are not part of the mediawiki installation targets.

@hashar thanks for all those details.

mwdebug2001 and mwdebug2002 are indeed part of the targets, so that's a good question: why didn't the cdb rebuild trigger yesterday Tuesday during the train?

jnuche@deploy1002:~$ egrep mwdebug200[12] /etc/dsh/group/mediawiki-installation
mwdebug2001.codfw.wmnet
mwdebug2002.codfw.wmnet

Interestingly, the only cdb rebuild happening on mwdebug2001 before today was on the 7th of Feb: https://logstash.wikimedia.org/goto/2504b0b3cbc067990f7197efda5381c2

Also, the list of MW targets was updated yesterday (although I don't know how often that gets updated, probably quite often). From a copy I made this morning:
jnuche@deploy1002:~$ ls -l dshBK/mediawiki-installation
-r--r--r-- 1 jnuche wikidev 7710 Apr 27 21:50 dshBK/mediawiki-installation

Maybe mwdebug200[12] were in the target list at some point, then removed for a while and then readded yesterday? ¯\_(ツ)_/¯

new version rolled out everywhere :)

@elukey thanks!