Maniphest T91068

Set up a schedule for doing failover exercises for toollabs
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	yuvipanda
	Feb 27 2015, 2:33 PM

Tags

Referenced Files

None

Subscribers

Description

Once T90542 is filled, we should have hotspares for everything, and should do a simulated failover every 3-4 months. This shouldn't affect users at all.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Declined		None	T91068 Set up a schedule for doing failover exercises for toollabs
		Resolved		Andrew	T90542 Make sure that toollabs can function fully even with one virt* host fully down
		Resolved		coren	T90546 Test and verify that OGE master/shadow failover works as expected
		Resolved		yuvipanda	T89995 ToolLabs web proxy tolerate the failure of virt host
		Declined		yuvipanda	T91484 Monitor that the redundant webproxies have same state in terms of what they are proxying to whom
		Resolved		yuvipanda	T90557 Generic services nodes should be redundant so OGE can reschedule them onto another machine if one goes down
		Resolved		yuvipanda	T91065 Move uwsgi jobs to be run on generic hosts, retire uwsgi hosts
		Resolved		yuvipanda	T91066 Retire 'tomcat' node, make Java apps run on the generic webgrid
		Resolved		bd808	T91072 Move toollabs instances around to minimize damage from a single downed virt* host
		Resolved		yuvipanda	T99347 Migrate tools-checker-02 away from labvirt1003
		Resolved		yuvipanda	T101635 Write an icinga check to ensure that toollabs instances are appropriately distributed across labvirt** hosts
		Declined		yuvipanda	T91237 Have bigbrother run on multiple nodes to provide redundancy against tools-submit failure
		Resolved		yuvipanda	T91239 Setup a redis slave for toollabs as backup / redundancy
		Resolved		yuvipanda	T96966 Make tools-static redundant
		Declined		None	T96967 Make tools-mail redundant
		Duplicate		None	T101636 Move tools-shadow away from labvirt1004

Event Timeline

yuvipanda created this task.Feb 27 2015, 2:33 PM

yuvipanda raised the priority of this task from to Needs Triage.

yuvipanda updated the task description. (Show Details)

yuvipanda added projects: Toolforge, Cloud-Services.

yuvipanda added subscribers: yuvipanda, coren, Andrew.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 27 2015, 2:33 PM

Well, if it'd affect users, the exercise wasn't successful :-).

yuvipanda added a project: ToolLabs-Goals-Q4.Mar 25 2015, 9:19 PM

yuvipanda moved this task from Backlog to Redundancy on the ToolLabs-Goals-Q4 board.Mar 25 2015, 9:36 PM

scfc triaged this task as Medium priority.Apr 6 2015, 7:48 AM

scfc added a subtask: T90542: Make sure that toollabs can function fully even with one virt* host fully down.

scfc moved this task from Backlog to Ready to be worked on on the Toolforge board.Apr 6 2015, 8:23 AM

Luke081515 moved this task from Triage to Backlog on the Cloud-Services board.Mar 25 2016, 4:12 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:54 PM

• GTirloni edited projects, added cloud-services-team (Kanban); removed ToolLabs-Goals-Q4, Cloud-Services.Mar 21 2019, 8:44 PM

Andrew closed subtask T90542: Make sure that toollabs can function fully even with one virt* host fully down as Resolved.Nov 27 2019, 4:31 PM

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 9:47 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Will reopen if needed/when we focus on this.