I've edited the setup section of the train docs to use ssh rather than http token auth. https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Setup

@AlvaroMolina and @Stryn I see the problem with your user accounts. The tl;dr is our Gerrit instance has ldap.localUsernameToLowerCase = true but both of your accounts are stored in the database as uppercase only.

re-ran here successfully https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/18998/

In T218209#5020861, @Krinkle wrote:

Presumably due to https://gerrit.wikimedia.org/r/#/c/integration/config/+/496162/ which was just deployed.

I do see the message in its entirety in logstash

In case it's useful, here's what CI does: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/quibble/+/master/quibble/mediawiki.d

In T217830#5009234, @thcipriani wrote:

How do we fix this? We could set the mtime way in the future in scap, I guess. This system is really hard to reason about as it is and I'd be happier with a different solution.

I think I have a theory, and I think it could explain how this keeps happening.

Lowering priority since touching InitialiseSettings.php has been happening since T60618: [scap] Touch InitialiseSettings.php if syncing (circa 2014)

I would delete it right away in the web UI but I have no idea whether https://releases-jenkins.wikimedia.org/ uses JJB and if so where the config files are stored :/

I have disabled beta-scap-eqiad since it's now failing constantly.

Now that 3.9.2-1 is everywhere, this should be resolved.

I'm not sure why it's showing None but the actual problem is:

In T217539#4999025, @hashar wrote:

I thought about a LDAP failure, I could not find any evidence in deployment-deploy01 /var/log though :-(

In T215918#4967428, @Jrbranaa wrote:

@EvanProdromou, that sounds like a good thing to pursue. Definitely something worth investigating. Let me poke Tyler/Dan to see the feasibility for using K8s for that today. I know that there's some work in this space that could be leveraged.

We currently have 2 paths to an image being built and pushed:

Did this manually. Disk space issue should be resolved. This time I did this the same way I did T177867#4516303. There's a patch to do this via Jenkins that I'm going to review now.

I think this just needs to be deployed at this point (which looking at /srv/deployment maybe hasn't happened in a while).

The l10nupdate is currently disabled in production. Resetting priority.

I think @dduvall accomplished everything outlined on this task. Closing.

Thanks for the merge @jijiki !

Title of task was a bit ambiguous. To release the new version of scap, we still need to merge the puppet patch: https://gerrit.wikimedia.org/r/491879

@Ottomata I backported your fix and deployed it for wmf.18 yesterday before I rolled-forward the train to group1 wikis. This error hasn't shown up in the logs since.

Some random background on what's happening here:

In T216689#4971987, @Jdforrester-WMF wrote:

In T216689#4971832, @Nikerabbit wrote:

Yeah, this has been failing a lot of late.

I haven't seen a working build for CX since yesterday, so I would say it is always failing now.

And yet a bunch of things have passed these tests and merged in the gate, so it's not 100%:

Lowering priority and removing as train blocker since it did not regress last week.

In T216086#4965559, @Anomie wrote:

This doesn't seem to be new in wmf.17, the earliest hit I see is on 2019-02-04 on enwiki running wmf.14 (XFfLDgpAAEMAAFssDuIAAAAR). The rates don't really start taking off until the 9th.

Also all the hits in Kibana[1] have phpversion "7.2.8-1+0~20180725124257.2+stretch~1.gbp571e56". So it seems more likely this is something to do with PHP 7 rather than any recent MediaWiki change.

[1]: Except for 5 on 2019-02-04 and 2019-02-05, all from mw1272, that don't have a phpversion field at all.

@dduvall removed that last week after one of the cloudvirt machines had issues. Rather than rebuild the machine there was some discussion with @Mholloway in IRC about whether or not it was necessary to rebuild IIRC the conclusion was it was no longer being used(?).

Move scap pull sooner

This could expand on the configuration outline in T210267: The continuous release pipeline should support more than one service per repo