In T227529#5344660, @akosiaris wrote:

In T227529#5335623, @bd808 wrote:

In T227529#5333480, @akosiaris wrote:

I am resolving this, feel free to reopen is something is amiss

The cn and sn for uid=waldir,ou=people,dc=wikimedia,dc=org are both waldyrious (lower case w). Wikitech will never be able to authenticate as MediaWiki will canonicalize the username to start with a capital letter W and wikitech is configured to enforce same case matching for user account lookup.

Changed to Waldyrious (upper case) per comment above. @waldyrious could you please try again logging into wikitech?
@thcipriani The gerrit error seems to be the same as T216605, mind running that script?

For that particular image I can recreate locally:

In T228328#5342778, @thcipriani wrote:

refreshMessageBlobs was added in T222539
One of two solutions:

• Install scap::scripts on all appservers rather than canary appservers
• rethink how this is included in scap

refreshMessageBlobs was added in T222539

We use /mnt/docker now and it's got a lot of space.

This looks to be released as a feature on contint1001 now.

In T227814#5327808, @Lucas_Werkmeister_WMDE wrote:

So did we figure out what happened? (The change @thcipriani linked is by me, and I also think it feels suspect, so I hope this is not my fault 😬)

In T227814#5327808, @Lucas_Werkmeister_WMDE wrote:

So did we figure out what happened? (The change @thcipriani linked is by me, and I also think it feels suspect, so I hope this is not my fault 😬)
Edit: And if this was only fixed by reverting the train, and the issue persists on Test Wikidata, shouldn’t the task remain open?

In T193824#5316795, @daniel wrote:

In T193824#5316023, @Legoktm wrote:

I've been operating under the assumption that at some stage we're going to have some kind of containers with just some extensions installed, and we'll need to resolve those dependencies for testing or whatever.

I'm not sure that this would actually be very useful. AS far as I understand, goal here is to test the interoperability of extensions. Just enabling all of them and running all their tests in the combined environment doesn't seem like a great approach for that.

Ran in another terminal window after I got the message 22:10:59 Updating ExtensionMessages-1.34.0-wmf.13.php from scap

mergeMessageFileList which is what scap shells out to to handle building ExtensionMessages does a require_once on each of the files listed in the extension-list file. It then dumps $wgMessagesDirs into the ExtensionMessages file.

In T227814#5326488, @Jdforrester-WMF wrote:

Nothing obvious in the repo's logs.

hrm, so WikibaseLib is in ExtensionMessages for 1.34.11, but not in ExtensionsMessages for 1.34.13

UBN since it's a deployment blocker.

Nothing looks to have changed with the actual json in the extension:

Hrm. That is bizarre.

Hi @Shirayuki I believe your issue should be resolved. Please re-open this ticket if you ware still unable to login.

didn't mean to change priority

In T207707#5302763, @hashar wrote:

So I think we can just:

• stick to overlay2
• reconfigure Docker to use /mnt/docker
• restart Docker
• run docker-pkg and verify it actually pulls all images / does not rebuild any
• archive /var/lib/docker and ultimately delete it

In T208259#5314362, @Gehel wrote:

I ran into this issue again when deploying WDQS today. Some of the binaries were owned by the previous deployer. My workaround was to reset ownership to myself, but that's obviously not a step that I would like to de every time we switch the deployer.
@thcipriani: Any idea how we could fix this for the longer term?

This should now be resolved in production as of Tue, Jun 25, 7:45 AM when the new scap version (3.10.0-1) was released.

Hi @Shirayuki sorry for the delay

@CDanis I merged in the changes to the release branch and pushed up the debian/3.11.0-1 tag: you should be good to release from the scap side.

In T207707#5302044, @Dzahn wrote:

Hi Hashar, at this point i think it makes sense to assign back to to you to check if it seems sane and then for your next step:

and change its config to point there.

Applied patch and reindexed accounts 2019-07-02 17:53 UTC.

In T227111#5300995, @thcipriani wrote:

In T227111#5300974, @greg wrote:

09:47:13 <James_F> I just got a large number of e-mails from gerrit all at once.
09:47:19 <James_F> Possibly a deadlock got resolved?

In T227111#5300974, @greg wrote:

09:47:13 <James_F> I just got a large number of e-mails from gerrit all at once.
09:47:19 <James_F> Possibly a deadlock got resolved?

In T227111#5300903, @hashar wrote:

Only thing I can imagine is that hmm Zuul lost its connection to Gerrit some how :-\

Assigning to @jeena since she already has an initial patchset. Adding serviceops-radar since folks on their team will likely be reviewing (thanks @akosiaris for the current review on the patchset)

In T216605#5265588, @akosiaris wrote:

• what's stopping people from making the same case mistakes and causing the issue again? Won't new accounts face that same problem? Or do we expect LocalUsernamesToLowerCase to mitigate that?

In T226724#5297154, @jcrespo wrote:

Not manager by SREs as discussed on SRE meeting, some sres do not even have the admin rights on gerrit AFAIK.

Investigated this a bit today. I was hoping with 3 incidents in one day that the trigger for this event might be obvious.

In T224448#5284141, @thcipriani wrote:

This happened twice in the past 24 hours.

Spoke too soon. Gerrit 2.15.14 caused a lot of SendEmail locks (T224448: Gerrit http threads stuck behind sendemail thread). I have to rollback to 2.15.13. I will update wikitech as well.

This happened twice in the past 24 hours.

In T225308#5243509, @mmodell wrote:

@thcipriani: the fix is turning on http passwords.

Config change has been deployed.

In T224857#5277649, @Joe wrote:

In T224857#5274663, @thcipriani wrote:

Should we be working to implement symlink swapping in scap? Sounds like it's currently a blocker for this task (although opcache exhaustion may be blocking both).
If so the key is that we need the realpath for each deployment needs to be unique, correct? Atomic deployments are a side effect of the unique real path?

No, I don't think that's what we should focus on.
We want to solve both the atomicity problem and the opcache exhaustion problem, and the symlink swapping won't solve the latter, which is by far the more urgent matter.
As I said, rolling restarts on each deployment solve both problems, but for now given the doubts I've seen floating around we could find a middle ground as follows:

• we only run the rolling restart when a signficant code path change happens, so when we run the train

this would unblock the php7 transition at the very least. Is that acceptable to you?

In T224857#5268558, @Joe wrote:

To summarize the situation a bit, we have 2 problems right now that need to be solved:

1. Deploys are not atomic
2. Opcache gets progressively exhausted by deploying code

Se can solve 1) using the symlink swapping and mod_realdoc or switching to nginx on the frontend, but the second problem can only be solved via regular resets or restarts of the opcache.
Given we don't trust the opcache resets at all at this point, the rolling restart of the daemons seem like the best option to solve both issues.

In T224915#5271635, @jijiki wrote:

@thcipriani I have packaged the newer version and uploaded to stretch-wikimedia, I will upgrade the servers soon, time permits.

In T207707#5239428, @hashar wrote:

The new disks can be shown as sdc and sdd.
Currently I think we have 3 RAID 1 arrays, with LVM on the largest one (md2):
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/md0: 46.5 GiB, 49965694976 bytes, 97589248 sectors
Disk /dev/md1: 953.4 MiB, 999751680 bytes, 1952640 sectors
Disk /dev/md2: 883.9 GiB, 949069283328 bytes, 1853650944 sectors
Disk /dev/mapper/contint1001--vg-data: 883.9 GiB, 949066137600 bytes, 1853644800 sectors
Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
I don't know what are ops best practices for disks. I guess we will need SRE to create a new RAID 1 array over the two disks, create a new LVM volume group and then we can do some partitioning.
For Docker I guess we can start with a 500GB partition on the new disks? Then mount that to /srv/docker and change its config to point there.

In T217830#5178490, @Krinkle wrote:

I think the simpler solution would be to do the same as what we generally to avoid this problem in production code, e.g. in MediaWiki core. Which is to have the verification mechanism part of the value instead of as some fragile indirect side-effect of the value.
In other words, either

1. Make the fragile side-effect not a side effect and not fragile, by having the PHP code in wmf-config create the cache with the explicit mtime set to the value it was creating it for.

In T224857#5253300, @Joe wrote:

In T224857#5247276, @thcipriani wrote:

Checking my understanding of what you're saying (please correct me where I'm misunderstanding): for rollback/emergencies, scap does opcache_reset, syncs the file, then calls the smart script to do a rolling restart (11 minutes -- 350 servers 2 seconds/server). By sending the opcache_reset first means that the change goes live immediately, and the restart is just for cache sanity.

We'd sync the files, reset the opcache (which would actually make the rollback effective) and then do the rolling restart afterwards.

In T224939#5257358, @brennen wrote:

...

2. Python (present or installable on most systems?)

Thanks for the enthusiasm :)

In T216605#5224338, @Mhurd wrote:

Happening to me too:

My theory here is that there is some ssh timeout on the phab side for pushing to gerrit (or maybe some error on the phab side(?)). I don't see anything in the gerrit error logs related to phab/editquality.

Looks like master is showing a newer version:

@brennen made the mistake of showing interest in this task, assigning accordingly :)

Guessing this is the aborted clean from https://tools.wmflabs.org/sal/log/AWq2FV6OOwpQ-3Pk_Reb

In T224857#5243133, @Joe wrote:

Regarding rollbacks: I guess that we can add a switch that, just for rollbacks, sends out the opcache reset first, and does the rolling restart afterwards, thus reducing the window of time in which the issues are present. We'll have a small window of time in which an opcache corruption would be possible, but only in case of emergency.
@thcipriani would that work for addressing your concerns?

In T213198#4961595, @LarsWirzenius wrote:

https://wikitech.wikimedia.org/wiki/Blubber has been rewritten (by @thcipriani), is there anything specific that still needs to be improved for this ticket to be done?

In T225308#5243509, @mmodell wrote:

@thcipriani: the fix is turning on http passwords.

This is due to gerrit using cn as the UI login (internally in gerrit this is the gerrit schema), while using uid as the ssh/api login (internally in gerrit this is the username schema).

gate-and-submit-swat is for immediate-to-prod code. Yes, mw-config is in its own queue, but it should also have priority…

In T225064#5240522, @mobrovac wrote:

In T225064#5240480, @hashar wrote:

Nice, thank you for the explanation :-] Left to figure out in a different task is how to test Citoid together with Zotero, but I guess that is for another task.

Euh, no, that's what this task is for :) We were able to build images before, now we are not.

I have gc info from right when this happened: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDYvNS8tLWp2bV9nYy5nZXJyaXQubG9nLjcuY3VycmVudC0tMjMtMjgtNTE=&channel=WEB

The pod that is running in the ci namespace of the staging cluster seems to be logging 504 errors:

Rerunning the test on contint1001 the failure message I see in the logs is:

In T224857#5232389, @jijiki wrote:

If there are not any better short-term solutions/ideas, depooling-deploy-restarting-pooling looks like the only option we have. We will have to sacrifice speed for platform stability, because at the end of the day, that's what our users expect.

What affect have the opcode_invalidate calls for specific files via sync-file had? Do we only see this corruption for opcache_reset?

Deployed on beta, the old method took ~25 seconds, new method takes ~5 seconds. I'll prep a new prod scap release.

In T224448#5216643, @dcausse wrote:

In T224448#5216526, @thcipriani wrote:

In T224448#5216469, @dcausse wrote:

Here SendEmail-1 is not "blocked" it's just waiting for jobs, however the dump says:
Locked ownable synchronizers:- <0x00000001c13617f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync).
Meaning that this SendEmail-1 thread holds the ReentrantLock lock which is blocking Http threads.
So something happened previously in that thread that caused the lock to still be held.

Is it also possible that it's some kind of blocking read causing a deadlock that doesn't show up in dumps? i.e., https://dzone.com/articles/java-concurrency-hidden-thread

Here it's clear that Send-Email1 has held the lock but failed to release it. Also it's the responsability of guava LocalCache to not allow this to happen. Since Send-Email-1 is clearly not in a zone where the lock can legitimately be held I suspect a bug in guava or in the loading method of the Account info (if LocalCache does not allow hard failures).

In T224448#5216469, @dcausse wrote:

Here SendEmail-1 is not "blocked" it's just waiting for jobs, however the dump says:
Locked ownable synchronizers:- <0x00000001c13617f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync).
Meaning that this SendEmail-1 thread holds the ReentrantLock lock which is blocking Http threads.
So something happened previously in that thread that caused the lock to still be held.

Restarted Gerrit 2019-05-27T23:10,