gerrit2003 is new hardware and on bookworm
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Jhancock.wm | T369670 Q1:rack/setup/install gerrit2003 | |||
| Resolved | Dzahn | T372804 setup gerrit2003 with gerrit service (gerrit on bookworm) | |||
| Open | Dzahn | T338470 Rename gerrit2 unix user to gerrit and assign a fixed uid | |||
| Resolved | Dzahn | T379213 PuppetFailure - gerrit2003 | |||
| Open | None | T333143 Move Gerrit data out of root partition |
Change #1063893 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site: (WIP) try applying gerrit role on gerrit2003
Change #1063896 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: temp set a gerrit IP for testing
Change #1063896 merged by Dzahn:
[operations/puppet@production] gerrit: temp set a gerrit IP for testing, gerrit2003 only
Change #1063898 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: on gerrit2003, set firewall provider, admin groups, team owner
Change #1063898 merged by Dzahn:
[operations/puppet@production] gerrit: on gerrit2003, set firewall provider, admin groups, team owner
Change #1063904 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: create a temp insetup role to test java install in bookworm
Change #1063904 merged by Dzahn:
[operations/puppet@production] gerrit: create a temp insetup role to test java install in bookworm
Change #1064412 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add firewall, java, scap, mail settings to Hiera for gerrit1004
Change #1064412 merged by Dzahn:
[operations/puppet@production] gerrit: add firewall, java, scap, mail settings to Hiera for gerrit1004
Change #1070680 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] site: add insetup-gerrit role to gerrit2003, remove gerrit1004 hiera
Change #1070680 merged by Dzahn:
[operations/puppet@production] site: add insetup-gerrit role to gerrit2003, remove gerrit1004 hiera
Change #1070683 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add backup::host, gerrit::migration etc to insetup role
Change #1070683 merged by Dzahn:
[operations/puppet@production] gerrit: add backup::host, gerrit::migration etc to insetup role
Change #1072323 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add gerrit::proxy profile to insetup::gerrit role
Change #1072323 merged by Dzahn:
[operations/puppet@production] gerrit: add gerrit::proxy profile to insetup::gerrit role
Change #1073305 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit::proxy: files managed under /var/www/ require httpd
Change #1073308 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit::proxy: fix link target for gerrit logo
Change #1073308 merged by Dzahn:
[operations/puppet@production] gerrit::proxy: fix link target for gerrit logo
Change #1073305 merged by Dzahn:
[operations/puppet@production] gerrit::proxy: ensure /var/www/ exists before files under it
Change #1074275 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] acme_chief: authorize new machine gerrit2003 to fetch gerrit certs
Change #1074275 merged by Dzahn:
[operations/puppet@production] acme_chief: authorize new machine gerrit2003 to fetch gerrit certs
Change #1074477 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: include gerrit profile in insetup::gerrit for testing
Change #1074498 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add acme_chief snippet to gerrit-setup role
Change #1074498 merged by Dzahn:
[operations/puppet@production] gerrit: add acme_chief to gerrit-setup role
gerrit2003 now has a working apache-based gerrit::proxy with certs, no puppet errors and everything.
except the actual gerrit application and we avoided adding any service IP
Change #1077781 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: make it possible to not bind the service IP on a gerrit server
Change #1077781 merged by Dzahn:
[operations/puppet@production] gerrit: make it possible to not bind the service IP on a gerrit server
Change #1074477 merged by Dzahn:
[operations/puppet@production] gerrit: include gerrit profile in insetup::gerrit for testing
Change #1078748 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: move passwords include from role to profile
Change #1078748 merged by Dzahn:
[operations/puppet@production] gerrit: move passwords include from role to profile
Change #1078752 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: sync lfs data also to new machine
Mentioned in SAL (#wikimedia-operations) [2024-10-08T21:34:38Z] <mutante> gerrit2003 - sudo -u gerrit-deploy /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (for some reason this fails in puppet but works manually) T372804 T257317 T317412
Change #1078759 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: avoid duplicate declaration error on first setup
Change #1078759 merged by Dzahn:
[operations/puppet@production] gerrit: avoid duplicate declaration error on first setup
Change #1079026 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: comment out creation of site dir in migration profile
Change #1079026 merged by Dzahn:
[operations/puppet@production] gerrit: comment out creation of site dir in migration profile
For the first time puppet runs just fine on the new hardware now, before it is in production.
Also gerrit is deployed there already. Everything is in place minus "no service IP is bound to the NIC", we don't sync lfs-data yet and ?.
Notably this also means gerrit on bookworm seems to work. Since no more puppet issues, app deployed, same Java version.
Mentioned in SAL (#wikimedia-operations) [2024-10-10T07:32:23Z] <hashar> Stopped gerrit service on gerrit2003.codfw.wmnet since it is not starting up properly | T372804
The gerrit process on gerrit2003 does not start properly and is flapping:
Oct 10 07:29:21 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19328. Oct 10 07:29:30 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19329. Oct 10 07:29:38 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19330. Oct 10 07:29:47 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19331. Oct 10 07:29:56 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19332. Oct 10 07:30:04 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19333. Oct 10 07:30:13 gerrit2003 systemd[1]: gerrit.service: Scheduled restart job, restart counter is at 19334.
I tried to stop it manually, but of course Puppet bring it back up. That is causing alerts as the service goes up and down continuously. The service and its monitoring should be disabled until it s ready.
As a side track, I don't know what gerrit2003 is for. Is that a hardware refresh for gerrit2002?
Change #1079206 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079206 abandoned by Hashar:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Reason:
Thank you for having set the downtime!
Change #1079206 restored by Dzahn:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079206 merged by Dzahn:
[operations/puppet@production] Disable gerrit monitoring on gerrit2003
Change #1079358 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: set Hiera keys for nist_keys, nftables
Change #1079358 merged by Dzahn:
[operations/puppet@production] gerrit: set Hiera keys for nist_keys, nftables
Change #1079363 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit2003: move bind_serviceIP Hiera key host name level
Change #1079363 merged by Dzahn:
[operations/puppet@production] gerrit2003: move bind_service_ip Hiera key host name level
Change #1078752 merged by Dzahn:
[operations/puppet@production] gerrit: sync lfs data also to new machine
Change #1063893 merged by Dzahn:
[operations/puppet@production] site: apply gerrit role on gerrit2003
Change #1080379 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: remove Hiera keys on host gerrit2003 that are applied by role
Change #1080379 merged by Dzahn:
[operations/puppet@production] gerrit: remove Hiera keys on host gerrit2003 that are applied by role
Basically done.
All that is missing is we haven't assigned a service IP to this machine.
In puppet it is just set to not bind a service IP and the service is masked.
But it could be enabled now whenever we like. Also lfs data is already synced from prod server.
To be determined how exactly we use it next.
One way would be to assign like "gerrit-replica-b" to it.
Another would be to fail-over to this machine as the new production gerrit (since it's newer hardware and also newer OS version!) and then make the previous prod server a replica.
Change #1074488 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: delete temp gerrit setup role
Change #1074488 merged by Dzahn:
[operations/puppet@production] gerrit: delete temp gerrit setup role
Discussed in today's team meeting. This server is up and running with the production role on it now, so the task to setup the service on this machine is considered done.
There is just no service name name attached to it and the gerrit service is masked. Monitoring is disabled.
How to use it will be determined in the upcoming work regarding Gerrit failover strategies.
But as has been pointed out by @Jelto the current Gerrit prod server is not that old either. The current replica is older.
It might make the most sense to turn this into a new replica, leave the current prod host as is and also keep using the current old replica, to have 3 Gerrit machines at the same time.
LFS data syncing from the prod server is also already setup and happened via the puppetized timer:
root@gerrit2003:/srv/gerrit/data/lfs# du -hs . 31G
Change #1087967 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add chown parameter to lfs data rsync, ensure daemon_user is used
Change #1087967 merged by Dzahn:
[operations/puppet@production] gerrit: add chown parameter to lfs data rsync, ensure daemon_user is used