Fri, Sep 20
Thu, Sep 19
We are planning to move Thumbor to k8s, T233196, thus I am closing this task
Wed, Sep 18
@thcipriani is it ok if we update php7 to 7.2.22 on deploy* servers? Do you know if there are any dependencies ?
@holger.knust What is the status of this? Just checking if I can help
Tue, Sep 17
@crusnov I have downtimed 'Check systemd state' checks until Sept 24 because it was polluting the alerts
I am downtiming the check for a couple of days because it is polluting our alerts
I have disabled this check for now
Fri, Sep 13
It has not happened again, Resolving for now.
Thu, Sep 12
Wed, Sep 11
Aug 22 2019
@thcipriani When a deployer run scap pull on mwdebug, they got sudo: [/usr/local/sbin/check-and-restart-php,: command not found (notice the extra comma), so we rolledback https://gerrit.wikimedia.org/r/531475
We are on our way to finishing migration to PHP7, my opinion is to try the PHP7.2 bandaid rather than upgrading production to PHP7.3 right now, if we all agree of course.
Aug 13 2019
@Mholloway We are getting a number of alerts where scb* servers return 504 to healthchecks, what do you suggest?
Aug 12 2019
Aug 9 2019
@Dzahn I think we should continue by switching all jobs on Monday and see what gives. We followed the same strategy with async jobs, i.e. we migrated a few jobs that stood out along with a few high traffic ones, and eventually we switched what was left with one go. What do you think? Thank you very much for the help!
Aug 8 2019
Aug 7 2019
Aug 6 2019
@Dzahn you are right, I am marking this as invalid.
We switched mw1270 to PHP7 but we came across the following issues
@Papaul closing, sorry this slipped through the cracks.
Aug 5 2019
@Papaul I am marking this as resolved, thank you!
Removing HHVM and any leftovers are now part of T229792, we mark this as resolved 💃
For connection pooling purposes, when we want to access search.svc.eqiad.wmnet from php-fpm, we are doing so via nginx. This nginx is installed on each mw* server listening on *:80. Mediawiki is configured to do so via https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/ProductionServices.php#L147, and trying to resolve localhost. For every request, localhost first resolves to ::1 where nginx is not listening to, and then continues to 127.0.0.1, which is successful. The first failure increases the tcp attemptfails counter.
@Papaul Server is depooled, ping me when do pool it back, many thanks !
Jul 31 2019
Jul 30 2019
@elukey I am up for attempting to patch it and upload to stretch-wikimedia, I will try to do it next week
Jul 29 2019
Before making changes to profile::mediawiki::common and since we are in the process of removing hhvm from jobrunners/videoscalers (T219148), I think we should also consider the following:
Just a heads up, we are planning to start migrating API servers to serve only via PHP7. For the time being, we have one in each DC.
Jul 26 2019
@Papaul Can you let us know what are our options (if any?)
Jul 25 2019
All async jobs run on PHP7, we will keep an eye for about a week, and then cleanup code leftovers
@Eevans I was under the impression we have more work to be done on the server. Shall we mark this task as resolved?
Jul 24 2019
@holger.knust I accidentally copied the wrong dump to your directory yesterday, I uploaded a new dump today. Sorry for the confusion.
Jul 23 2019
@holger.knust I copied a gzipped dump to a server you have access to, please reopen when you need newer one:)