We are planning to move Thumbor to k8s, T233196, thus I am closing this task

In T232412#5497260, @EdErhart-WMF wrote:

@CDanis @ayounsi @Vgutierrez Hello all, happy Monday! WordPress is still investigating what's happening, and they're wondering if we have a more detailed error message that we could share with them. I could also try to cc you on the WordPress ticket, if that would be more helpful. Thanks for your time—y'all are fantastic!

@Dzahn We will be moving Thumbor to k8s T233196, we can repurpose the spare server for something else:)

@thcipriani is it ok if we update php7 to 7.2.22 on deploy* servers? Do you know if there are any dependencies ?

@holger.knust What is the status of this? Just checking if I can help

@crusnov I have downtimed 'Check systemd state' checks until Sept 24 because it was polluting the alerts

I am downtiming the check for a couple of days because it is polluting our alerts

I have disabled this check for now

It has not happened again, Resolving for now.

@thcipriani When a deployer run scap pull on mwdebug, they got sudo: [/usr/local/sbin/check-and-restart-php,: command not found (notice the extra comma), so we rolledback https://gerrit.wikimedia.org/r/531475

We are on our way to finishing migration to PHP7, my opinion is to try the PHP7.2 bandaid rather than upgrading production to PHP7.3 right now, if we all agree of course.

@Mholloway We are getting a number of alerts where scb* servers return 504 to healthchecks, what do you suggest?

@Dzahn Thank you! We'll see how things are since we now have merged https://gerrit.wikimedia.org/r/425027, and continue with moving any transitional changes we did.

@Dzahn I think we should continue by switching all jobs on Monday and see what gives. We followed the same strategy with async jobs, i.e. we migrated a few jobs that stood out along with a few high traffic ones, and eventually we switched what was left with one go. What do you think? Thank you very much for the help!

@Dzahn you are right, I am marking this as invalid.

We switched mw1270 to PHP7 but we came across the following issues

@Papaul closing, sorry this slipped through the cracks.

@Papaul I am marking this as resolved, thank you!

Removing HHVM and any leftovers are now part of T229792, we mark this as resolved 💃

For connection pooling purposes, when we want to access search.svc.eqiad.wmnet from php-fpm, we are doing so via nginx. This nginx is installed on each mw* server listening on *:80. Mediawiki is configured to do so via https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/ProductionServices.php#L147, and trying to resolve localhost. For every request, localhost first resolves to ::1 where nginx is not listening to, and then continues to 127.0.0.1, which is successful. The first failure increases the tcp attemptfails counter.

@Papaul Server is depooled, ping me when do pool it back, many thanks !

@elukey @MoritzMuehlenhoff I have added tmpreaper's removal as an actionable in our HHVM removal process (T229792), shall we mark this as resolved or invalid?

@elukey I am up for attempting to patch it and upload to stretch-wikimedia, I will try to do it next week

Before making changes to profile::mediawiki::common and since we are in the process of removing hhvm from jobrunners/videoscalers (T219148), I think we should also consider the following:

Just a heads up, we are planning to start migrating API servers to serve only via PHP7. For the time being, we have one in each DC.

@Papaul Can you let us know what are our options (if any?)

All async jobs run on PHP7, we will keep an eye for about a week, and then cleanup code leftovers

@Eevans I was under the impression we have more work to be done on the server. Shall we mark this task as resolved?

@holger.knust I accidentally copied the wrong dump to your directory yesterday, I uploaded a new dump today. Sorry for the confusion.

@holger.knust I copied a gzipped dump to a server you have access to, please reopen when you need newer one:)