Standardize failover procedures for Collab services
Open, MediumPublic
Actions

Assigned To

None

Authored By

	LSobanski
	Mar 4 2025, 8:51 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

LSobanski

Description

Todo:

Done:

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T393239 ProbeDown
Open	None	T387831 Standardize failover procedures for Collab services
Open	None	T387830 Contint failover process
Resolved	ABran-WMF	T387833 Gerrit switchover process
Open	None	T388507 Investigate missing commit from old changes
Resolved	ABran-WMF	T260666 Create a cookbook to automate gerrit's switchover
Resolved	ABran-WMF	T393050 ProbeDown
Resolved	ABran-WMF	T338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
Resolved	Dzahn	T379213 PuppetFailure - gerrit2003
Open	None	T417642 Rename gerrit2 Gerrit user to gerrit in the administrator group
Resolved	MatthewVernon	T392186 Grant Gerrit admin to arnaudb
Resolved	LSobanski	T393034 Investigate out of date refs following gerrit switchover
Resolved	ABran-WMF	T395440 Gerrit read-only plugin test
Resolved	ABran-WMF	T406334 Gerrit switchover between secondary instances
Resolved	LSobanski	T406482 SystemdUnitFailed (jenkins.service on contint1002)
Resolved	hashar	T406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
Resolved	hashar	T411583 Gerrit backups are growing
Resolved	hashar	T430774 Compact Gerrit H2 databases to reclaim disk
Open	None	T430775 Validate Gerrit backup recoverability after the fileset exclusions
Open	None	T433926 Backup Gerrit with LVM snapshots and Bacula pre/post job hooks
Resolved	hashar	T406856 Reduce size of analytics/superset/deploy.git Gerrit repo
Resolved	ABran-WMF	T407200 Gerrit is down
Resolved	ABran-WMF	T412779 Gerrit failover process
Resolved	ABran-WMF	T417246 Reimage gerrit1003
Resolved	dancy	T417767 scap error on Gerrit first setup
Resolved	ABran-WMF	T417777 SystemdUnitFailed - apache2.service on gerrit1003:9100
Duplicate	None	T417968 ProbeDown
Duplicate	ABran-WMF	T417966 GerritHAProxyBackendUnavailable
Resolved	ABran-WMF	T417247 Reimage gerrit2002
Resolved	ABran-WMF	T418242 SystemdUnitFailed - rsync-lfs_replica_sync.service on gerrit2002:9100
Resolved	ABran-WMF	T418264 Fix gerrit-restart cookbook
Resolved	ABran-WMF	T418266 GerritHAProxyServiceUnavailable
Resolved	ABran-WMF	T418240 GerritReplicationUnavailable
Resolved	ABran-WMF	T417279 Harmonize DNS on all gerrit instances
Resolved	ABran-WMF	T417565 GerritHAProxyBackendUnavailable
Resolved	ABran-WMF	T417566 GerritHAProxyServiceUnavailable
Resolved	Dzahn	T391578 Releases failover process
Resolved	Dzahn	T410422 SystemdUnitFailed - rsync-srv-org-wikimedia-releases
		Restricted Task
Resolved	Arnoldokoth	T412016 Mailman failover process
Open	None	T433268 Zuul failover process

Event Timeline

LSobanski created this task.Mar 4 2025, 8:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2025, 8:51 AM

LSobanski renamed this task from Standardizing failover procedures for Collab services to Standardize failover procedures for Collab services.Mar 4 2025, 8:55 AM

LSobanski added a subtask: T387830: Contint failover process.

LSobanski moved this task from Incoming to Work in Progress (Tracking tasks) on the Collaboration-Services board.

LSobanski updated the task description. (Show Details)Mar 4 2025, 9:00 AM

LSobanski added a subtask: T387833: Gerrit switchover process.

ABran-WMF changed the status of subtask T387833: Gerrit switchover process from Open to In Progress.Mar 4 2025, 9:01 AM

LSobanski triaged this task as Medium priority.Nov 24 2025, 12:37 PM

Dzahn closed subtask T391578: Releases failover process as Resolved.Dec 4 2025, 11:20 PM

Arnoldokoth created subtask T412016: Mailman failover process.Dec 8 2025, 12:53 PM

Arnoldokoth updated the task description. (Show Details)

ABran-WMF closed subtask T387833: Gerrit switchover process as Resolved.Feb 16 2026, 2:57 PM

Arnoldokoth closed subtask T412016: Mailman failover process as Resolved.Feb 25 2026, 1:38 PM

LSobanski created subtask T433268: Zuul failover process.Jul 27 2026, 3:43 PM