@ReaperDawn @GerardM are you still getting 503s?

@Gilles done, let me know if you need more help.

@Krinkle sorry for the delay in merging this. LGTM now, please reopen if there are issues.

I am stalling this for now until we see how T220811 pans out.

@Krinkle After a little more digging, looks like nginx in hassium is using HTTP/1.0 when forwarding requests to mwdebug*. My theory is that when HHVM returns a 500 and proto version is HTTP/1.0, it add the Content-length: 0 header. Varnish in turn sees Content-length: 0, so it returns its own generic error since it believes there are no data to serve.

After some debugging on hassium, I found some interesting things:

We have upgraded php7 on beta, so now it looks like async jobs are running. We will leave it as is until next week, where we will assess if it worked out.

@Dzahn THANK YOU! 😍

In T220212#5171872, @Joe wrote:

Participants selection should be based on need and not on a fixed quota of participants.

It's extremely sad and frustrating we can't have a all-technical sit-down of any form with all the interested engineers in the movement.

To be clear: the three topics I named above have such a vast impact across the org that I dont' see 50 people tackling them in a satisfactory way.

What is a bit more weird is that this works as expected with PHP7.

It looks like deployment-prep has an older php7.2 version than production, which is something we should fix as well

@Gilles All packages have been rebuilt and added to buster-wikimedia main repo. Please reopen if we have any issues.

@Lucas_Werkmeister_WMDE We will look into it, it only happened on a single server so we believe, for now, that it could not be related to the change per se.

We have pushed https://gerrit.wikimedia.org/r/502986 and (its update) https://gerrit.wikimedia.org/r/505383/ to production. Are we going to proceed with session and mysql settings or should we mark this task as resolved?

@Joe @Krinkle, since we have pushed enable_dl => 0 to production, I am resolving this. Feel free to reopen if you disagree :)

@Dzahn I need to to talk with our team before I green light this, also mentioned in T221132. Is it Possible to revisit this in a week from now? Thank you!

All traffic is served by haproxy. If we have any issues, this can be easily reverted. Closing for now.

@Gilles ping me when you think we are ready to build packages for buster, we would do it soon either way

I am happy to do it, but: I am afraid it will have to wait for next week, as this is a short week for me:)

@Krinkle I have stopped and disabled xenon-log, excimer-log, and apache on mwlog* servers, and I have removed the arclamp-generate-svgs cron job. Let me know if there is anything else related to change 503675

In T221142#5129920, @MoritzMuehlenhoff wrote:

In T221142#5129395, @jijiki wrote:

@faidon @Dzahn user wpao is in LDAP 'ops' group, but not in the admin.yaml 'ops' group which makes cross-validate-accounts script complain. What should we do ?

This will resolve itself once Willy is added to the "datacenter-ops" group (see https://phabricator.wikimedia.org/T221142#5121532), the account check validates that every member of the "ops" LDAP group is either a member of the "ops" or "datacenter-ops" permission group.

@faidon @Dzahn user wpao is in LDAP 'ops' group, but not in the admin.yaml 'ops' group which makes cross-validate-accounts script complain. What should we do ?

@Gilles thank you! I added the relevant codfw ones

I am afraid I do not know much either about the services on this server so to perform any actions. @Krinkle Is there something we can do for the time being? What problems are we having while this server is in this state?

@Gilles This is fixed now, I will though revert back to nginx for the weekend. We do have data we can work with from today.

@CDanis Thank you! I am resolving this for now.

@CDanis tx!

@JoKalliauer Since last week all servers are using librsvg 2.40.20-3 which fixes some svg rendering issues (and possibly introduces others).

@Krinkle yeah we will wait for sure, meanwhile, we are exploring:)