Page MenuHomePhabricator

Gerrit is down
Closed, ResolvedPublic

Description

Gerrit is down

While running the switchover process, we faced a race condition on the final steps of the process; where the puppet change required to switch Gerrit instances over was not visible to all Gerrit instances. That issue came from the fact that Gerrit was seen as unavailable for the puppetserver instances that tried to git fetch the puppet change, despite the DNS cache wipe. Once the Puppet change had been merged on the primary instance where it was initially fetched at the start of the process, it was impossible to re-apply that change anywhere. At this stage, we chose to revert to the previous state and use the emergency DNS procedure.

Details

Event Timeline

corto changed the task status from Open to In Progress.
corto triaged this task as High priority.
corto created this object with visibility "acl*security (Project)".
corto created this object with edit policy "acl*security (Project)".
LSobanski changed the task status from In Progress to Open.Oct 14 2025, 11:26 AM
Jdforrester-WMF subscribed.

SHould we now mark this as Resolved? Or it is worth keeping open?

ABran-WMF claimed this task.
ABran-WMF subscribed.

marking it resolved, I'll follow up in T387833: Gerrit switchover process. Feel free to reopen if needed!

hnowlan changed the visibility from "acl*security (Project)" to "Public (No Login Required)".