This ticket is to replace https://gerrit-replica.wikimedia.org with https://gerrit.wikimedia.org in various repos that might or might not use it instead of the main host.
reasoning per list thread here:
and commit message here:
https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919925
also, from today's SRE collab team meeting it was TODO to create this very ticket, even before the mail thread brought it up again the same day
Change 919925 had a related patch set uploaded (by Dzahn; author: Dzahn):
[labs/codesearch@master] switch to main gerrit server instead of using the replica
Change 919925 merged by jenkins-bot:
[labs/codesearch@master] switch to main gerrit server instead of using the replica
I deployed the codesearch change. So it shouldn't be affected by the maint window. Maybe eventually we should have three hosts, one for replicas (codesearch and others put a lot of pressure on gerrit) and one for standby. Just thinking.
Change 920243 had a related patch set uploaded (by Hashar; author: Hashar):
[labs/codesearch@master] Revert "switch to main gerrit server instead of using the replica"
I am replying here to the wikitech-l message, I think it will make it easier to find it later in the future. It came up in preparation of a network operation which affected access to the Gerrit replica and scheduled for 30 minutes. My reply are inline
This means codesearch will be affected (and won't get updated) and possibly even will be down during that time.
More or less, Codesearch indexes repositories via a service named Hound which we have configured to poll every 90 minutes. If it can't reaches the repositories, I guess it will bails out indexing for that time tick and try again on the next poll tick 90 minutes later. Once the index is populated, the frontend serves the searches from the index and do not need any access to the repositories. At least that is my understanding.
We, at least in my team, would like to switch codesearch (and other clients) back to just use gerrit.wikimedia.org and not the replica directly.
Just today we agreed to make a new ticket for specifically this, because soon we have to reimage the replica to bullseye and add more downtime.
I don't think it causes much troubles. The major services we offloaded to the replica are system that routinely hammer every single repositories (and sometime multiple branches) but can otherwise work fine without live updates. From the top of my mind:
That has offloaded 60% of the git traffic (roughly 4000 requests per minute) off of the primary Gerrit (which also serves the REST API requests and anything that requires live). When the replica get reimaged the CodeSearch (and others) will be delayed by a few hours, and I don't think it is any problematic. The primary might be to handle it, though I don't know how it will behave given it will have to serve traffic from all repositories which discard caches and might well affect the latency of more important use cases.
The reason we did the split in the past was to reduce load on the main Gerrit server but meanwhile first the issue has been fixed in newer Gerrit versions and then also just a few days ago we switched to brand new hardware.
So now if anything it should be beefier than before and even without
that it seemed already a thing of the past.
Possibly yes, then that still adds load for the application and I imagine it would add a bit of latency to requests made to the primary. It can potentially be measured by pointing gerrit-replica.wikimedia.org to the primary (which might just work, IIRC the ssh host key is shared between the primary and the replica) and then observes any change in the service quality (latency primarily which is affected by caches hit ratio, Java garbage collection and whatever race contention / locking mechanisms in the app).
And we pay for this with this issue that the replica becomes a second production system, with the need for downtimes. It complicates fail-over scenarios too and in a way means there is never a passive host when we do DC switch-over.
The replica is a hot spare, it continuously receives updates from the primary which lets one switch to it having all the data up-to-date thanks to the replication. I wanted to try switching over to it back in August when the server got changed and would have proposed to follow that route late April but I was in vacations in both cases. Regardless what I have found out is would have missing data:
I think we should invest in a high availability setup which is T217174 (I declined it as part of a routine cleanup of Phabricator tasks). What it entitles? I don't know to be fair, but that is how most are running Gerrit this days, that lets one have multiple primaries geographically distributed, and when one goes down (maintenance, upgrade whatever) traffic shift to the others. That should make operations easier and reduce maintenance occurred downtime which is always stressful.
Until we go that route, there are a few changes in the pipeline to make it easier to switch over the service (use a fixed UID to ease rysnc and move files around to have all of them under a single directory).
So yea, I suggest we change the config of codesearch now to use the main gerrit unless you have concerns about that.
I'd like to keep the bot git traffic on the replica and proposed a revert of the CodeSearch change: https://gerrit.wikimedia.org/r/920243
It's more complicated than that (I'm one of the maintainers of codesearch). gerrit-replica being down for extended period of time can bring down codesearch fully. I distinctly remember one case of that.
As maintainer of codesearch: I don't care if you want to revert the patch after the maintenance or keep it as-is. Just point to the main gerrit during the maintenance since it's not bringing gerrit down even though it's making a lot of requests.
As an engineer: I think there is difference in opinion on what gerrit-replica is or should be. Either make an extra host that's a hot stand-by (and switch them around as needed). Or switch DNS of gerrit-replica to point to the main gerrit during maint on gerrit-replica. or make it official gerrit-replica is not stable and allow codesearch to use main gerrit in the times.
Change 920243 abandoned by Hashar:
[labs/codesearch@master] Revert "switch to main gerrit server instead of using the replica"
Reason:
We had a couple meetings with Daniel and others and clarified a few things. In short the server is beefer, codesearch is not that much traffic and we can revisit later. Overall it is probably good to keep offloading some traffic off the primary, but this revert does not change much.
This caused T337263: full disk on codesearch8 on the codesearch side (I honestly wouldn't have caught that ahead of time either, so not trying to blame!!).
Yep, this is what should happen and will happen in most cases - it'll just keep running, with an outdated index. There are some (IMO) edge cases that it could not start back up, but being realistic, codesearch is not a production service and if people can't live without it for a few hours or a day, that's an entirely different problem.
I guess we don't need to switch back to gerrit-replica? In any case, if we eventually do, we need to take care to remove all the old git repos first before cloning the new ones. Something like:
cd /srv mv hound hound.bak mkdir hound chown codesearch:codesearch hound systemctl start codesearch-write-config # in a separate terminal rm -rf hound.bak
I guess we don't need to switch back to gerrit-replica?
The agreement so far was "don't switch back until after the reimage of gerrit-replica (tomorrow, Thu May 25) and then we ultimately decide (if it stays on main gerrit forever or goes back). I feel a tendency to keep it on main gerrit but we should still make a formal decision afaict.
P.S. it also seems possible in the future, with regular 6 month DC-switchover, we would consider flipping what is gerrit and what is gerrit-replica between eqiad and codfw, which might or might not factor into this. The whole point of NOT pointing clients to gerrit-replica is that it makes the "passive" / hot spare a second production host. So if we don't have to use it I think we should not use it.
@LSobanski so.. summarizing the status here:
codesearch was switched to main gerrit in https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919925
extdist though stayed on gerrit-replica, see last comment on abandoned patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/920761
Other than that, looking at codesearch for things that use gerrit-replica.wikimedia.org I can't see specific clients using it, at least not in what is covered by codesearch of course.
https://codesearch.wmcloud.org/search/?q=gitlab-replica.wikimedia.org&files=&excludeFiles=&repos=
It seems that we either have to say "extdist stays on replica forever" and then decline this ticket but also accept that it still does this one thing for production and both gerrit machines are therefore production... or we have to still switch extdist to main gerrit now and resolve this.
If switching extdist to main gerrit doesn't kill it that seems the better solution to me by far.
codesearch was switched to main gerrit in https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919925
We should switch it back to gerrit-replica :]
I gave some overview of the replica usage T336710#8857046 , the missing LARGE user which can't be found in codesearch is Phabricator which is constantly pulling every single repositories. I'd like one day to make it a Gerrit replica and push updates instead, but that is moare work.
stalled since we have no consensus which direction to go
if both gerrit and gerrit-replica are needed at all times then we have to accept there exists no failover machine or create more machines
I am reopening since I apparently missed following up on that task. Codesearch must go back to the replica.
Change #920243 restored by Hashar:
[labs/codesearch@master] Revert "switch to main gerrit server instead of using the replica"
A thing that changed here since the previous comments is that now we have one new hardware machine to fail-over to (one per DC but shared between phab and gerrit).
Codesearch was moved from the replica to the primary because of a network operation which would result in a 30 minutes downtime. I imagine most services passing through that switch got moved to avoid a cascading failure. On the same day, I have send a change to move it back to the replica has it has been pointed to since August 2019.
You have pushed back here on this task, on this change, we had a video call and eventually I gave up and followed along with the hardware being newer to support the traffic. The thing is whenever I have to dig into the Gerrit log, investigate an outgoing issue, that code search traffic adds to the background noise and that is frustrating.
Please kindly switch CodeSearch back to the Gerrit replica using the change I have proposed last year https://gerrit.wikimedia.org/r/c/labs/codesearch/+/920243 and which I have updated with the instruction by Amir above T336710#8877525
Hello Antoine, so the reason I hadn't just merged that was that I wanted to discuss it in meeting but we didn't get to that yet. Partially because the managers were out last week.
My concerns from back then have been mitigated mostly by the existence of the new third machine we have now and are in the middle of setting up.
So let me merge for now as requested. Just with the option to get back to this and talk about which replica/server to use later. We might have 2 replicas etc. cheers!
BTW, last time we did the change, the hound storage filled up because it couldn't understand that gerrit.wm.o and gerrit-replica.wm.o are the same so it reindexed everything in extra directories. I don't know if we put mitigations in place to fix that or not but something to keep an eye on.
Indeed and you gave some indications at T336710#8877525 which I have integrated in the refreshed commit message https://gerrit.wikimedia.org/r/c/labs/codesearch/+/920243
@Dzahn You would need someone from codesearch project or a WMCS root to clear the existing index on the codesearch9.codesearch.eqiad1.wikimedia.cloud instance (see commit message for the commands).
Change #920243 merged by jenkins-bot:
[labs/codesearch@master] Revert "switch to main gerrit server instead of using the replica"
I had already clicked +2 on that change yesterday and it had said "starting gate and submit" but for some reason it had not actually submitted.
It did now after I sent my +2 again.
I do have access to codesearch project and currently running those commands you listed. It's just slow...
This is now deployed.
I did run these commands. but after initial trouble: stopped all hound services, moved /srv/hound, started all hound services, started write-config service, waited a long while, deleted /srv/hound.back,
I can now see in the various /srv/hound/*/config.json files the URL is gerrit-replica rather than gerrit.
Health page shows they are slowly coming back one after another:
From the Apache access logs https://logstash.wikimedia.org/app/dashboards#/view/825c5c80-8aef-11eb-8ab2-63c7f3b019fc , I can confirm the codesearch instance is no more hitting gerrit.wikimedia.org:
Thanks!