gerrit2003 is new hardware and on bookworm
Change #1070683 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add backup::host, gerrit::migration etc to insetup role
Change #1070683 merged by Dzahn:
[operations/puppet@production] gerrit: add backup::host, gerrit::migration etc to insetup role
Change #1072323 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit::proxy profile to insetup::gerrit role
Change #1072323 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit::proxy profile to insetup::gerrit role
Change #1073305 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit::proxy: files managed under /var/www/ require httpd
Change #1073308 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit::proxy: fix link target for gerrit logo
Change #1073308 merged by Dzahn:
[operations/puppet@production] gerrit::proxy: fix link target for gerrit logo
Change #1073305 merged by Dzahn:
[operations/puppet@production] gerrit::proxy: ensure /var/www/ exists before files under it
Change #1074275 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] acme_chief: authorize new machine gerrit2003 to fetch gerrit certs
Change #1074275 merged by Dzahn:
[operations/puppet@production] acme_chief: authorize new machine gerrit2003 to fetch gerrit certs
Change #1074477 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: include gerrit profile in insetup::gerrit for testing
Change #1074498 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add acme_chief snippet to gerrit-setup role
Change #1074498 merged by Dzahn:
[operations/puppet@production] gerrit: add acme_chief to gerrit-setup role
gerrit2003 now has a working apache-based gerrit::proxy with certs, no puppet errors and everything.
except the actual gerrit application and we avoided adding any service IP
Change #1077781 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: make it possible to not bind the service IP on a gerrit server
Change #1077781 merged by Dzahn:
[operations/puppet@production] gerrit: make it possible to not bind the service IP on a gerrit server
Change #1074477 merged by Dzahn:
[operations/puppet@production] gerrit: include gerrit profile in insetup::gerrit for testing
Change #1078748 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: move passwords include from role to profile
Change #1078748 merged by Dzahn:
[operations/puppet@production] gerrit: move passwords include from role to profile
Change #1078752 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: sync lfs data also to new machine
Mentioned in SAL (#wikimedia-operations) [2024-10-08T21:34:38Z] <mutante> gerrit2003 - sudo -u gerrit-deploy /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (for some reason this fails in puppet but works manually) T372804 T257317 T317412
Change #1078759 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: avoid duplicate declaration error on first setup
Change #1078759 merged by Dzahn:
[operations/puppet@production] gerrit: avoid duplicate declaration error on first setup
Change #1079026 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: comment out creation of site dir in migration profile
Change #1079026 merged by Dzahn:
[operations/puppet@production] gerrit: comment out creation of site dir in migration profile
For the first time puppet runs just fine on the new hardware now, before it is in production.
Also gerrit is deployed there already. Everything is in place minus "no service IP is bound to the NIC", we don't sync lfs-data yet and ?.
Notably this also means gerrit on bookworm seems to work. Since no more puppet issues, app deployed, same Java version.
Mentioned in SAL (#wikimedia-operations) [2024-10-10T07:32:23Z] <hashar> Stopped gerrit service on gerrit2003.codfw.wmnet since it is not starting up properly | T372804
The gerrit process on gerrit2003 does not start properly and is flapping:
Oct 10 07:29:21 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19328. Oct 10 07:29:30 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19329. Oct 10 07:29:38 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19330. Oct 10 07:29:47 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19331. Oct 10 07:29:56 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19332. Oct 10 07:30:04 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19333. Oct 10 07:30:13 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19334.
I tried to stop it manually, but of course Puppet bring it back up. That is causing alerts as the service goes up and down continuously. The service and its monitoring should be disabled until it s ready.
As a side track, I don't know what gerrit2003 is for. Is that a hardware refresh for gerrit2002?
Change #1079206 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079206 abandoned by Hashar:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Reason:
Thank you for having set the downtime!
Change #1079206 restored by Dzahn:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079206 merged by Dzahn:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079358 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: set Hiera keys for nist_keys, nftables
Change #1079358 merged by Dzahn:
[operations/puppet@production] gerrit: set Hiera keys for nist_keys, nftables
Change #1079363 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit2003: move bind_serviceIP Hiera key host name level
Change #1079363 merged by Dzahn:
[operations/puppet@production] gerrit2003: move bind_service_ip Hiera key host name level
Change #1078752 merged by Dzahn:
[operations/puppet@production] gerrit: sync lfs data also to new machine
Change #1063893 merged by Dzahn:
[operations/puppet@production] site: apply gerrit role on gerrit2003
Change #1080379 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: remove Hiera keys on host gerrit2003 that are applied by role
Change #1080379 merged by Dzahn:
[operations/puppet@production] gerrit: remove Hiera keys on host gerrit2003 that are applied by role
Basically done.
All that is missing is we haven't assigned a service IP to this machine.
In puppet it is just set to not bind a service IP and the service is masked.
But it could be enabled now whenever we like. Also lfs data is already synced from prod server.
To be determined how exactly we use it next.
One way would be to assign like "gerrit-replica-b" to it.
Another would be to fail-over to this machine as the new production gerrit (since it's newer hardware and also newer OS version!) and then make the previous prod server a replica.
Change #1074488 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: delete temp gerrit setup role
Change #1074488 merged by Dzahn:
[operations/puppet@production] gerrit: delete temp gerrit setup role
Discussed in today's team meeting. This server is up and running with the production role on it now, so the task to setup the service on this machine is considered done.
There is just no service name name attached to it and the gerrit service is masked. Monitoring is disabled.
How to use it will be determined in the upcoming work regarding Gerrit failover strategies.
But as has been pointed out by @Jelto the current Gerrit prod server is not that old either. The current replica is older.
It might make the most sense to turn this into a new replica, leave the current prod host as is and also keep using the current old replica, to have 3 Gerrit machines at the same time.
LFS data syncing from the prod server is also already setup and happened via the puppetized timer:
root@gerrit2003:/srv/gerrit/data/lfs# du -hs . 31G
Change #1087967 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add chown parameter to lfs data rsync, ensure daemon_user is used
Change #1087967 merged by Dzahn:
[operations/puppet@production] gerrit: add chown parameter to lfs data rsync, ensure daemon_user is used
We need a follow-up task to _acutally start using_ this new server and failover gerrit to it.
Change #1140520 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add a second replica, start replicating to gerrit2003
Change #1140520 merged by Dzahn:
[operations/puppet@production] gerrit: add a second replica, start replicating to gerrit2003
Change #1152782 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add ssh_host_rsa public key for gerrit2003
Change #1152782 merged by Dzahn:
[operations/puppet@production] gerrit: add ssh_host_rsa public key for gerrit2003
Change #1152810 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: introduce second daemon_user name
Change #1152819 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: replace gerrit2003 RSA host key with ed25519 host key
I have reverted the replica configuration since that broke GitHub replication and the new gerrit2003 host was misconfigured ( T395887#10879856 )
Change #1153159 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/puppet@production] Revert "gerrit: add a second replica, start replicating to gerrit2003"
Change #1152819 merged by Dzahn:
[operations/puppet@production] gerrit: replace gerrit2003 RSA host key with ed25519 host key
Change #1152810 abandoned by Dzahn:
[operations/puppet@production] gerrit: introduce second daemon_user name
Reason:
will make a new simpler patch rebased on top of a re-revert of the replication config change
Since yesterday we are now replicating to the new machine gerrit2003 again.
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1153265
@ABran-WMF I wonder if you have thoughts on my original question on this ticket, back in August 2024 I said:
"determine if this is resolved once it's a warm standby host or if we switch production to this because it's newer hardware".
At this stage, I think we'll still wait a for a while:
Change #1167838 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/puppet@production] gerrit: enable gerrit.service and monitoring
Change #1167838 merged by Arnaudb:
[operations/puppet@production] gerrit: enable gerrit.service and monitoring
I would argue that this is already resolved because clearly a gerrit exists on gerrit2003.
But we agreed to close this out once it's actually the production gerrit.
@ABran-WMF is working on this right now: --> https://gerrit.wikimedia.org/r/c/operations/puppet/+/1188351
I feel like this ticket is resolved regardless of fail-over or not. Clearly we have a gerrit service running on this machine and on the new OS .. if we had not we would not have even attempted to make it prod.
I am happy to be convinced otherwise and if you want to reopen it that's not a big deal to me. But all the remaining work has been happening on other tickets more specific to the failover anyways.