Seems to me like this is 2 issues:

a) ldapauth-gitldap.wmflabs.org is not working as an LDAP provider (outside of the Gerrit instance)
b) https://gerrit.devtools.wmflabs.org/ is currently down (different from invalid certificate but what we need to check first)

Mentioned in SAL (#wikimedia-cloud) [2023-02-13T21:58:45Z] <mutante> rebooting instance gerrit-prod-1001 which can't be reached T329444

b) is probably related to T329535

step 1.. VM gerrit-prod-1001 (which that host name points to) could not be reached via SSH (maybe because of the general cloud outage today) and rebooting the instance brought that back

https://gerrit.devtools.wmflabs.org/ is back with the described TLS error.

step 2: puppet was disabled on the instance about 17 days ago with reason "gerrit deploy". I reactivated that.

Mentioned in SAL (#wikimedia-cloud) [2023-02-13T22:18:21Z] <mutante> install package python3-certbot-apache on gerrit-prod-1001 - T329444

I fixed the certificate issue by:

• installing package python3-certbot-apache. This is a the plugin to do the renewal challenge via apache httpd. In the past this was not a separate package.

• running certbow renew --apache as root

https://gerrit.devtools.wmflabs.org/r/q/status:open+-is:wip is back

Mentioned in SAL (#wikimedia-cloud) [2023-02-13T22:22:10Z] <mutante> certbot renew --apache fixed cert issue - https://ldapauth-gitldap.wmflabs.org/ does not exist unrelatedly - T329444

Change 888808 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] devtools: change gerrit hostname to use wmcloud, not wmflabs

https://gerrit.wikimedia.org/r/888808

@Ameisenigel While there are some other cleanups we should do here.. can you try what happens if you click the "Sign Up" link now that it's back?

it links me to https://wikitech.wikimedia.org/w/index.php?title=Special:CreateAccount&returnto=Gerrit/NewUser

so the normal wikitech signup page. and "ldapauth-gitldap.wmflabs.org" seems to be entirely gone.

So ideally that just works with your normal "developer account" (wikitech user) and we don't need that extra LDAP anymore.

Yes, I can sign in with my regular developer account.

@Ameisenigel Unfortunately it's against TOU if we do that, so I had to shut it down for the moment.

Can I ask what you wanted to test with the login?

Well, you have written that it should be possible to use the regular account. I just wanted to try out if it works.

@Ameisenigel Yes, I am sorry about that. I was informed myself after the previous comment that we can't do that. Would the gerrit test instance still be helpful for you if it's up but you can't login? I fixed the other issues, that the VM was down and that it had the cert errors.

We still need to edit the docs you pointed to.

Also we should replace "wmflabs.org" with "wmcloud.org". I already expanded the SSL cert to contain both and created a new proxy in Horizon.

It would be great if one would be able to perform actions at the test instance, but if this is not possible I will look out for another solution.

@Ameisenigel If you can give us another week or so that would be great. I have brought up the topic today but it needs a little time to discuss with others/other team. Then we are going to decide on how to move forward with this. There are several options.

I can say though that the "to learn Gerrit" part is outdated. It would be for a bit different purpose, to test changes before deploying to production.

Also, did you have any specific thing you wanted to test or learn about Gerrit? And did you know Wikimedia also has local Gitlab now?

The issue is that we can't use Wikitech for auth, so either we'd have to setup a separate LDAP server for this or just disable auth or use another auth method.

There is no need for a hurry. I just wanted to try out Gerrit without messing around at the real Gerrit instance. Yes, I know that we also have GitLab, but I have not yet actually used it.

I am not sure gerrit-prod-1001 is in a workable state currently. It has some mixed changes in it to prepare for some operations (changing the user, moving files around) and might be using an obsolete version compared to what we use in production.

As for logging in, the documentation at https://wikitech.wikimedia.org/wiki/Gerrit/test_instance is probably obsolete. Looks like https://ldapauth-gitldap.wmflabs.org/ was a MediaWiki instance with LDAP support used to create a LDAP entry, then the Gerrit test instance could have been made to authenticate against it. Thus essentially login on the test Gerrit is not possible.

If you want to try a few things with Gerrit, you can use the test/gerrit-ping repository on the production Gerrit. https://gerrit.wikimedia.org/r/test/gerrit-ping . Example of past changes: https://gerrit.wikimedia.org/r/q/project:test/gerrit-ping

Specifically the instance is down and can't keep using the current auth scheme. I am going to bring this up in today's IC meeting and I created follow-up task T330248. This should not be simply declined, considerable effort was already put into it. We either need to fix or or remove it.

Change 888808 merged by Dzahn:

[operations/puppet@production] devtools: change gerrit hostname to use wmcloud, not wmflabs

https://gerrit.wikimedia.org/r/888808