We removed shard18 from redis.yaml so to be able to avoid installing redis-server on this server pair (mc1036-mc2036).

• mc1036.eqiad.wmnet is left with puppet being disabled, as it is broken due to the server being absent from redis.yaml
• mc2036.codfw.wmnet has been reimaged to buster without redis-server installed 🎉

@aaron thank you! I updated the task description

@aaron If you have any insights regarding the Redis Lock Manager and file upload, it would be much appreciated (+ T265643)

@Millimetric, after discussing with @ema, traffic feels that those requests should be visible in turnilo (eg webrequests_sampled_128), but we should be able to filter them out easily. Moreover, I gave it some more thought and I think we should add to the condition to have the "x-wikimedia-debug" present as well, along with the X-Analytics: debug=1 header. The way I have written it now, it can be easily used to run requests which would be invisible in turnilo. All our mwdebug servers are small VMs, so sending a lot of traffic towards them would take a lot of time :)

In T243009#6574647, @Joe wrote:

In T243009#6574527, @CDanis wrote:

I would also like for someone to investigate if systemctl reload php7.2-fpm clears opcache.

IIRC it does, but we saw the same kind of corruption happening when issuing a reload than when invalidating it explicitly (that is, much more common than it is now with automatic revalidation).

In T243009#6574527, @CDanis wrote:

I would also like for someone to investigate if systemctl reload php7.2-fpm clears opcache.

In T243009#6569744, @dancy wrote:

Proposal:

Revise /usr/local/bin/safe-service-restart to accept --force to make it skip depool/repool.

Yesterday we had opcache corruptions on 2 servers, mw2252 && I don't know about other times, but for those specific 2 corruptions, I can say that they happened right after opcache restarted because, on these servers it reached its max cached keys:

mw2328:
    "start_time": 1600177590, -> Tuesday, 15 September 2020 
    "last_restart_time": 1603211850, -> Tuesday, 20 October 2020 16:37:30
    "oom_restarts": 0,
    "hash_restarts": 2,

mw2252:
    "start_time": 1600174055, -> Tuesday, 15 September 2020 12:47:35
    "last_restart_time": 1603217533, -> Tuesday, 20 October 2020 18:12:13
    "oom_restarts": 0,
    "hash_restarts": 2,

@LarsWirzenius after discussing it, we decided that for the time being we can't adopt this solution, given that generally there are no issues with the way we currently build debs. You can always use the package_builder instance on cloud VPS tou check if your package build properly. Last time I updated the documentation so I doubt we will have any issues building scap this time. Thank you!

In T265324#6567779, @Joe wrote:

Also I want to clarify: we can reduce the pain as much as possible, but for the duration of the transition phase, it will be somewhat more work than we're used to for this kind of changes. There is no way around that that I can think of.

Hopefully we'll be able to reduce the amount of time needed as much as possible.

I don't think "you also need to do a deploy to k8s" is a bad compromise, but at this point I'd like to hear from others.

In T265324#6567723, @Joe wrote:

In T265324#6567694, @jijiki wrote:

In T265324#6567095, @Joe wrote:

Regarding the apache httpd container, I am approaching layering as follows:

• one base image, which uses the apache2-bin debian package and just modifies the vanilla configuration to listen on port 8080 (so that the container can run as user www-data).
• one image configured to manage a php-fpm application. This image will be used as a base for both MediaWiki and the shellout service. It will include all modules and base configurations we need, and have a single virtualhost sending all "*.php" files to the fastcgi daemon

@Jgiannelos is there any help you would like from serviceops ?

Resolve it since it has not been updated for so long :)

@kostajh Thank you! 💃🏼

In T264698#6547540, @Papaul wrote:

@jijiki disk replaced

@Milimetric Sorry for the late reply, thank you very much! I will move forward with the relevant patch. Do we need to coordinate after merging it?

I like what we do for degraded RAIDs, I think it will help us move forward something like this.

Thank you!

According to netbox, this server is still under warranty.

In T244340#6197415, @Krinkle wrote:

If the local-memcached's blind ttl is around the time we tolerate purges to be delayed for, and if it explicitly excludes mw-wan broadcasts and other WANCache internal keys then this is probably fine to proceed as experiment without further input.

• WANCache:v:* - Fine, these are the bulk of values stored in Memached.
• WANCache:[^v]:- Not fine, but should be minor, these are misc unusual/short-lived keys.
• /*/mw-wan/ - Not fine, but should be minor, these are WANCache tombstones.
• Everything else - Unsure, but should be minor, these are dc-local direct use of BagOStuff without WANCache. The only example that comes to mind are MWs rate limits. Sounds unsafe to split-brain on local app servers?

@aaron Can you confirm the above, and maybe fill in more about the non-WANCache?

@Joe Purge delay tolerance is indeed in the ballpark of 10s, after which makes MW automatically shorten lots of things if exceeded, e.g. refuse storing results of stuff in higher level caches, or use drastically shorter TTLs for them, shortened CDN maxage, eventually throttle edits/read-only etc.

What happens when onhost memcached in unavailable? https://phabricator.wikimedia.org/T244340#6211682 @elukey @aaron

What happens when onhost memcached in unavailable? https://phabricator.wikimedia.org/T244340#6211682 @elukey @aaron