unrelatedly: yea, would also be nice to reinstall cobalt as gerrit1001, wouldn't it? subtask?

We can't move them behind LVS. Unlike Phabricator, which uses a separate hostname for the SSH service, Gerrit exposes them over the same domain. Last time we talked about this, LVS wasn't able to support multiple ports per the same host. I also once brought up the idea of SRV records, but AIUI they're not super well supported by all clients.

I'm not opposed to this, I'm just genuinely curious how it can be done.

In T165631#3272935, @demon wrote:

We can't move them behind LVS. Unlike Phabricator, which uses a separate hostname for the SSH service, Gerrit exposes them over the same domain. Last time we talked about this, LVS wasn't able to support multiple ports per the same host. I also once brought up the idea of SRV records, but AIUI they're not super well supported by all clients.

I'm not opposed to this, I'm just genuinely curious how it can be done.

I believe it can, we're doing multiple ports for the same service ip for e.g. logstash, unless I misunderstood the problem

Also in a master/slave configuration are the ssh host keys exposed by gerrit the same on both machines? Only slightly related to lvs but it just occurred to me

I was under the impression we couldn't do port-based LVS to the same domain. But I'm gladly willing to be wrong ☺️

And yes, same host key for the ssh daemon.

Decent transition might be to:

1. Create public LVS VIP
2. Test it on the Gerrit instance (in public)
3. If all good, decom the current (non-LVS) VIP

Later one:

1. Move gerrit2001 to the private vlan (re-image)
2. If all good, do the same with gerrit1001

@Dzahn other than the outdated host names, is this task still relevant?

@LSobanski good find! Hmm.. probably not but maybe let's talk about it for a minute before declining it.

@Dzahn, why is this not relevant anymore?

@ayounsi It would mean a considerable effort to recreate an entire LVS service, which we just recently shut down for Phabricator in a lenghty decom process (git-ssh.wm.org), just to have a relatively small improvement, that users can use the "nice" port 22 instead of 29418.

So for the group of existing users it would actually introduce a change to their existing (and working) setup while the group of new users are encouraged to use gitlab instead of Gerrit anyways.

Further WMF as a whole has decided to move towards gitlab and isn't planning to keep 2 code review systems around in the long term (not my personal decision).

So basically it'd be quite some work for relatively little improvement and resources are used for the new system rather than putting effort into the old system at this point.

Noted, thanks for the explanation!

In T165631#8485123, @Dzahn wrote:

@ayounsi It would mean a considerable effort to recreate an entire LVS service, which we just recently shut down for Phabricator in a lenghty decom process (git-ssh.wm.org), just to have a relatively small improvement, that users can use the "nice" port 22 instead of 29418.

FWIW, we've been able to run Gerrit off 22 since the beginning, we just never did. All we would have to do is run the system sshd on something like 2222 to free up 22 for the Gerrit daemon.

But it's not really worth it at this point.