It's very harmless apart from logspam. If it is too much the fix can be backported.

Change 633704 abandoned by Filippo Giunchedi:
[operations/puppet@production] install_server: use standard partman recipe for nvme cp hosts

I added this back in 2015: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/209380

Change 635532 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Pablo Grass (WMDE)):
[mediawiki/extensions/Wikibase@master] FormatterCacheFactory: add version to cache key(s)

Change 635320 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Pablo Grass (WMDE)):
[mediawiki/extensions/Wikibase@master] FormatterCacheFactory: use a factory for all services

At least there could be better error messages upstream. Also asked Kobiton about the setup between the Mac Mini and the phones, if there could be something special there.

In T266078#6566997, @Urbanecm wrote:

In T266078#6566848, @Xqt wrote:

Unrecognised value for parameter "sites": smnwiki

Something missing on mw api side?

I tested the userinfo call, and it works as expected, so I think it's a PWB bug.

Adding more logs for reference:

1603281834291	geckodriver::android	DEBUG	Launching org.mozilla.firefox/org.mozilla.fenix.IntentReceiverActivity
1603281834392	mozdevice	DEBUG	execute_host_command: >> "host:transport:ZY2242HJDX"
1603281834492	mozdevice	DEBUG	execute_host_command: << []
1603281834492	mozdevice	DEBUG	execute_host_command: >> "shell:am start -W -n org.mozilla.firefox/org.mozilla.fenix.IntentReceiverActivity -a android.intent.action.VIEW -d about:blank --es args --marionette\\ --
profile\\ /mnt/sdcard/org.mozilla.firefox-geckodriver-profile"
1603281838030	mozdevice	DEBUG	execute_host_command: << "Starting: Intent { act=android.intent.action.VIEW dat=about:blank cmp=org.mozilla.firefox/org.mozilla.fenix.IntentReceiverActivity (has extras) }\nStatus: ok\nActivity: org.mozilla.firefox/org.mozilla.fenix.HomeActivity\nThisTime: 1360\nTotalTime: 2284\nWaitTime: 2297\nComplete\n"
1603281838030	geckodriver::marionette	DEBUG	Waiting 60s to connect to browser on 127.0.0.1:64772
1603281838211	geckodriver::android	DEBUG	Force stopping the Android package: org.mozilla.firefox
1603281838211	mozdevice	DEBUG	Force stopping Android package: org.mozilla.firefox
1603281838311	mozdevice	DEBUG	execute_host_command: >> "host:transport:ZY2242HJDX"
1603281838411	mozdevice	DEBUG	execute_host_command: << []
1603281838411	mozdevice	DEBUG	execute_host_command: >> "shell:am force-stop org.mozilla.firefox"
1603281839918	mozdevice	DEBUG	execute_host_command: << ""
1603281839918	webdriver::server	DEBUG	<- 500 Internal Server Error {"value":{"error":"timeout","message":"Socket timeout reading Marionette handshake data: Resource temporarily unavailable (os error 35)","stacktrace":""}}

Recording from IRC for posterity:

11:07 < bblack> so I was checking out https://gerrit.wikimedia.org/r/c/operations/puppet/+/633704 (which is one of the partman cleanup commits, this one 
                affecting our cacheproxy disk layout), and I've fallen back down the rabbithole of swap space considerations
11:07 < bblack> because the new defaults basically take the partman defaults without any specifics, which is apparently going to create a 1G swap partition 
                (at least, until some future change of defaults?)
11:08 < bblack> in the recent past we didn't have swap partitions on the cache boxes
11:08 < bblack> (and of course, modules/base for better or worse has vm.swappiness = 0, along with some other sometimes-questionable tunables)
11:10 < bblack> I get all the arguments for why disabling swap is probably a dumb idea in most common scenarios
11:11 < bblack> but, I think, if we're taking that angle as a reason to configure swap, we'd probably also relax that swapiness=0 setting as well to make it 
                more useful, and perhaps configure something other than a fixed 1G value for wildly varying workloads and phys memory sizes, too
11:11 < bblack> the current setup (yes, do swap, but at a small fixed sizes with swapiness=0) seems to be in some less-ideal intersection of competing ideas
11:12 < bblack> but in any case, rewinding to the cacheproxy case in particular
11:14 < bblack> these boxes have: 384GB of RAM, the bulk of which is hopefully a fast ram cache for http objects, and a 1.6TB super-fast nvme that's used as 
                an http object disk cache as well (it's the backing for the earlier ram cache), and then a mere ~300G of standard-issue (slower) SSDs for this 
                rootfs stuff
11:16 < bblack> we really don't have any hope of a substantially-useful swap config (where e.g. some mostly-idle or inefficient meta-daemons related to 
                monitoring or something might swap out significant RAM that matters to us), the disks we'd potentially use for swap are actually-smaller than 
                the real RAM, and we're very sensitive to the fact that we'd rather risk OOM than have the kernel make a dumb 
11:16 < bblack> decision on swap (e.g. algorithmically make the mistake of swapping out some critical varnishd cache memory and then need to swap it back in 
                during an cache hit for users)
11:17 < bblack> but I don't think our new regime of standard configs allows for a swapless setup?
11:23 < bblack> or: we could make some noswap variants so that it's semi-standardized?
11:24 < bblack> I don't want to derail standardization efforts, and I feel like allowing exceptions can turn into a lot of exceptions over time
11:24 < bblack> really we should revisit the swap question in general, but I think that goes well out of the partman-cleanup scope
11:25 < bblack> (but affects partman, too)
11:30 < bblack> for future sorting out of swap questions in general: we could also make the argument to just never configure swap *partitions*, and have some 
                base/standard puppetization create/manage swap *files* on the rootfs instead, which makes tuning and runtime changes simpler, etc.  I doubt 
                there's a perf diff we'd care about in that particular case.

Change 635506 merged by Karl Wettin (WMSE):
[mediawiki/services/wikispeech/mishkal@master] Add pipeline publish stage

Also I want to clarify: we can reduce the pain as much as possible, but for the duration of the transition phase, it will be somewhat more work than we're used to for this kind of changes. There is no way around that that I can think of.

In T265324#6567761, @jijiki wrote:

Overall, I think we may need to take one step back and consider if an apache container in the pod, is something that might cause more problems than it will solve.

Completed auto-reimage of hosts:

['labtestvirt2003.codfw.wmnet']

I've pushed a changed where I added a timeout of two minutes when we try close the browser to avoid making tests hang. When I looked at the code, that was the only thing I could find. Best case that would at least make sure our tests continuous to run. The root cause will still be there (the browser freezes) but maybe we can live with that.

Change 635530 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] partman: document cacheproxy exceptions

In T265324#6567723, @Joe wrote:

In T265324#6567694, @jijiki wrote:

@Aklapper done :-) thanks for helping to tidy this up!

All sub-tasks have been resolved.

Hi @Tambe great to see you here, I hope you remember me :)

QA done as part of deployment.

Other 2 from this week I can see have that as a parent task so adding...

@Christine_Domgoergen_WMDE: Thanks. In case of https://phabricator.wikimedia.org/project/view/4409/ its three open tasks first need sorting out :)

Change 634210 merged by jenkins-bot:
[mediawiki/extensions/ContentTranslation@master] Unset anon userpage placeholder

Change 635528 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add ldap-replica2003/2004 to conftool-data

@Aklapper This should remain active for non campaign related reportings, but this one could be archived indeed: https://phabricator.wikimedia.org/project/view/4409/

@SrishtiSethi: Assuming that this is resolved as per merged patch. If not, then please reopen.

Change 635527 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Performance: don't check autopromotion if blockautopromote is disabled

https://en.wikipedia.org/wiki/Special:Export ...
Check "Include templates"

@Amire80: No reply and I see nothing actionable in this task, hence closing.

In T265324#6567694, @jijiki wrote:

In T265324#6567095, @Joe wrote:

Regarding the apache httpd container, I am approaching layering as follows:

• one base image, which uses the apache2-bin debian package and just modifies the vanilla configuration to listen on port 8080 (so that the container can run as user www-data).
• one image configured to manage a php-fpm application. This image will be used as a base for both MediaWiki and the shellout service. It will include all modules and base configurations we need, and have a single virtualhost sending all "*.php" files to the fastcgi daemon

How are we planning to solve some minor differences our clusters have in our php configurations? The ones I know for sure are max_execution_time (jobrunners) and apc.ttl = 10 (parsoid). Would use ENV work? Moreover, I think we would like to have the ability to easily tweak those settings

@SrishtiSethi: Assuming that this is resolved as per merged patch. If not, then please reopen.

Change 630896 merged by jenkins-bot:
[operations/mediawiki-config@master] [WikibaseMediaInfo] Add config for related terms API

In T265324#6567095, @Joe wrote:

Regarding the apache httpd container, I am approaching layering as follows:

• one base image, which uses the apache2-bin debian package and just modifies the vanilla configuration to listen on port 8080 (so that the container can run as user www-data).
• one image configured to manage a php-fpm application. This image will be used as a base for both MediaWiki and the shellout service. It will include all modules and base configurations we need, and have a single virtualhost sending all "*.php" files to the fastcgi daemon

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:17:39Z] <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 785404fa2b998947d236aebe481ee1abcbd14220: Disable registrations stat on Special:TranslationStats (T264158) (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:10:31Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s)

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:10:31Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s)

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:10:31Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s)

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:10:31Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s)

Mentioned in SAL (#wikimedia-operations) [2020-10-21T11:10:31Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 11567427c3f7d2908b29046ee56a7b0c0da32c09: Enable ContentTranslation in 5 Wikipedias as a default tool (T264737; T264738; T264739; T264740; T264741) (duration: 01m 30s)

Upgraded db2093 from 10.4.12 to 10.4.15
Rebooted it to pick the new kernels too.

You almost catched them all @Sharvaniharan 👏

Change 635237 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable registrations stat on Special:TranslationStats

Change 635294 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ContentTranslation in 5 Wikipedias as a default tool

Change 635294 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ContentTranslation in 5 Wikipedias as a default tool

Change 635294 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ContentTranslation in 5 Wikipedias as a default tool

Change 635294 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ContentTranslation in 5 Wikipedias as a default tool