Wed, Nov 25
Regarding your question in mediawiki IRC: at the end of scap sync-world or scap sync-l10n, scap calls tasks.clear_message_blobs() which runs mwscript extensions/WikimediaMaintenance/refreshMessageBlobs.php
Also, my concern with the unnecessary MBS clearing is that it (presumably) increases the load on the appservers due to the subsequent cache misses. I don't know what the magnitude of that impact is though.
Tue, Nov 24
Mon, Nov 23
A burst of 116 of these errors occurred in the last 15 minutes.
wmf.18 has been rolled back out to group1.
Fri, Nov 20
Thu, Nov 19
The train has been rolled back to group0 (i.e., group0 = 1.36.0-wmf.18, group1 and group 2 = 1.36.0-wmf.16).
Wed, Nov 18
Tue, Nov 17
@kostajh https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 is deployed to group0.
@kostajh I will include https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 in the group0 rollout.
Mon, Nov 16
Thanks for the update @Andrew.
Tue, Nov 10
Mon, Nov 9
Execution of wmf-config/CommonSettings.php requires an MW installation.
Fri, Nov 6
Oct 23 2020
Oct 21 2020
Revise /usr/local/bin/safe-service-restart to accept --force to make it skip depool/repool.
Definitely opcache. Resolved by restarted php-fpm on the affected server.
Noting for the record in all cases where there's a bogus changed character in a string, the bad character is always one less than what it should be (i.e., it's not a bit flip, it's a decrement).
Oct 20 2020
And the errors are only happening on mw2328
Reported T266052 as a blocker.
Oct 16 2020
Talked to @Catrope yesterday and he confirmed that the early MBS cache clear is unnecessary.
Oct 15 2020
Oct 13 2020
I'm looking forward to seeing the results of this stress test. Does the docker registry have any controls to limit the number of connections and/or total bandwidth?
Oct 8 2020
Oct 7 2020
Oct 6 2020
This needs to be completed to allow T259586 to move forward.
I'm changing the status of this task to Declined. We don't need Gerrit to test/develop scap in train-dev. We can revive this task if needed in the future.
Oct 1 2020
Deploy promote works well in train-dev now.
Sep 29 2020
I asked about this in #wikimedia-cloud and Lucas pointed me to https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances
Sep 24 2020
ApiQueryGlobalUsage::execute() has been modified to defend against the undefined index error but the root cause has not been resolved.
Hi @DannyS712. @thcipriani instructed me to remove the non-blocking tasks from the subtasks list. An empty subtasks lists is the indication that there are no outstanding train blockers. Hope that helps!
Sep 23 2020
Saw a few of these error messages today for https://zh.wikipedia.org/wiki/User_talk:Franklin_Yu
360 of these (PHP Notice: Undefined property: stdClass::$dsr) in the last 4 hours.
We can either leave this task open for dealing with Scribunto, or a separate task can be created for that and this one can be closed.
The commented out deprecation warning has been deployed to production so I removed the Wikimedia-production-error tag. I changed the priority from UBN to High.
The easiest thing to do is to enlarge the virtual disk or add an additional vdisk to house /srv. How do we move forward with one of those options?
Sep 22 2020
Sep 14 2020
Sep 11 2020
I have been built and rebuilt the train-dev environment a zillion times without issues related to SSH host keys so I'm closing this ticket as a non-issue.
Sep 10 2020
Deployed via T261234. Closing