Interestingly this may have been a kernel bug: https://gerrit-review.googlesource.com/c/gerrit/+/153090/4#message-10cc8f42c24c28b1cb29c441c9aff13684555654

Going to boldly close this as resolved. It was mentioned on IRC that things look better since we've moved to new hardware with more resources.

Since we moved to new hardware (with increased resources) things have looked much better! Threads have been much lower and we haven't had to restart for over 3 weeks!

Yeh, though that site is hosted on gerrit-test (not gerrit-test5/gerrit-test6). Also that issue appears to be in the clouds proxy since i didn't change the apache config when that issue first started (a while ago).

Hi, yes some of our users wikis link to commons thumbnails.

So i'm not entirly sure how gerrit2001 managed to grow like this. I wonder if we should delete on gerrit2001 and just replicate a fresh copy to see.

I think we're planning to go with 2.16 instead of doing this then 2.16. Since a tentative date has been decided. (Also this month is busy for releng).

+1 to that @Reedy

1. Is not possible, it's either you can create repos for all namespaces or you carn't :)

This is totally expected. Gerrit dosen't allow you to do that when the auth type is LDAP.

Turns out we were missing setWrapperAttributes from HTMLForm.php

Fails on the master branch too https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/27780/console

I found doing https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ArticleRatings/+/547564/ fixed the issue.

All the instances run stretch :) (upgraded using dist-upgrade). Also the project is being replaced with T236309.

Is this still a issue? We have a more powerful machine with more ram/cpu power now!

@LarsWirzenius it's useful! (in matter of fact your latest report helped me to conclude that this is java gc as we restarted gerrit only last night).

So from the git and phabricator project:

@Ladsgroup sometimes users may forget to press it. It’s easily forgettable.

Did you all click the remember me box?

Please let me know if there is anything else!

@Mholloway should be resolved now, @thcipriani started the online reindexer.

@MarcoAurelio it takes a few minutes due to the cache. (It’s not instant once your edit that page)

All I did to fix this was set ‘profile::backup::ferm_directors: []’ in hiera.

https://gerrit.wikimedia.org/r/#/admin/groups/1698,members

@Osnard hi! I’ve gone ahead and created “ bluespice-admin” I doint think I have the perms to touch the bluespice repos but you should be able to. All you need to do is use Gerrits ui, go to Access Control, click add new reference (if refs/* or refs/heads/* is not there) then choose the group and then scroll down the list to “Push” (it’s a drop down).

This happened to me twice this week. I believe this is related to T224448. As this only happens to me when gerrit has to be restarted due to that.

Is this normal though? Having lists.wikimedia.org blocked by yahoo in my opinion is pretty high priority.

When i sent a email to wikitech-i last night, it failed (seems to be because yahoo blacklisted lists.wikimedia.org).

Sent https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092664.html