Configure phabricator clustering for daemons and repositories
Open, Stalled, MediumPublic
Actions

Assigned To

None

Authored By

	• mmodell
	Aug 16 2016, 11:31 PM

Description

Now that we have phab2001.codfw.wmnet up and running, we need to properly configure phabricator for repository replication and clustered task daemons.

Clustering configuration is well documented upstream in https://secure.phabricator.com/book/phabricator/article/cluster_repositories/ and https://secure.phabricator.com/book/phabricator/article/cluster_daemons/
Benefits
- The main benefit will be in redundancy of git repositories, allowing us to quickly recover from a failure of the primary phabricator server (or the primary data center)
- An additional (minor) benefit comes from running the phabricator task daemons on the backup server. This should reduce latency of commit parsing and email notifications, among other things.
Considerations
- The setup needs to be done during a maintenance window because it requires us to temporarily disable new repository creation.
- The main blocker, currently, is T143363: networking: allow ssh between iridium and phab2001
- When this task is resolved, we need to undo f528c2779e84 so that phd will run on both phabricator nodes.

Revisions and Commits

rOPUP Wikimedia Puppet
	rOPUPf528c2779e84 phabricator: don't run phd on inactive server yet

Related Objects
Search...

Status	Assigned	Task
Resolved	Joe	T154658 Prepare and improve the datacenter switchover procedure
Resolved	LSobanski	T156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services
Stalled	None	T143175 Configure phabricator clustering for daemons and repositories
Resolved	Dzahn	T143363 networking: allow ssh between iridium and phab2001

Event Timeline

• mmodell created this task.Aug 16 2016, 11:31 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 16 2016, 11:31 PM

• mmodell mentioned this in T137928: Deploy phabricator to phab2001.codfw.wmnet.Aug 16 2016, 11:32 PM

When cluster config is done we need to undo f528c2779e84 so that phd will run on both phabricator nodes.

• mmodell added a subtask: T143363: networking: allow ssh between iridium and phab2001.Aug 26 2016, 4:46 PM

• mmodell updated the task description. (Show Details)Aug 26 2016, 4:56 PM

When I initially attempted to set this up, I ran into an issue with almanac: it's not possible to remove devices from service bindings using the almanac UI. You have to use bin/remove destroy $PHID with a phid obtained from the almanac.search conduit api call.

Disabling clustered repo service entirely involves doing the same bin/remove trick but with the service phid rather than the binding phid.

Inquiring about this issue resulted in this upstream bug report: https://secure.phabricator.com/T11534

• mmodell updated the task description. (Show Details)Aug 26 2016, 5:01 PM

greg moved this task from To Triage to Misc on the Phabricator board.Sep 26 2016, 10:19 PM

jayvdb subscribed.Nov 15 2016, 11:56 AM

Patch to enable separate config files per environment: https://gerrit.wikimedia.org/r/#/c/321654

Dzahn closed subtask T143363: networking: allow ssh between iridium and phab2001 as Resolved.Nov 18 2016, 1:59 AM

• mmodell lowered the priority of this task from High to Medium.Apr 11 2017, 12:47 PM

greg added a project: Release-Engineering-Team (Kanban).May 16 2017, 3:55 PM

• mmodell changed the task status from Open to Stalled.Jun 6 2017, 9:09 AM

• mmodell moved this task from Backlog to Blocked (externally) on the Release-Engineering-Team (Kanban) board.

Following @Dzahn last message on https://gerrit.wikimedia.org/r/#/c/324841/3 , continuing here.

I've quickly reviewed Phabricator documentation and the change and the sudo requirement seems to be limited to run ssh and git as the Phabricator daemon-user, not as root. The commit message was a bit misleading and I might have missed the phd_user there, apologies.

Anyway can we also make it use keyholder to use an ssh key with passphrase and removing access to the private key to the Phabricator users?
Keyholder is the standard in our infrastructure for intra-host ssh, AFAIK.

As per the obvious concerns about www-data being able to ssh as the Phabricator daemon_user, that seems to be a hard requirement on Phabricator side in order to enable the clustering unfortunately. I'd like @MoritzMuehlenhoff to comment on this too, in the end is a trade off between security and high availability.
Is not fully clear to me right now in the single host configuration what kind of access has www-data to the repositories data. The sudo line seems limited to git-upload-pack and git-receive-pack.

In T143175#3374349, @Volans wrote:

Is not fully clear to me right now in the single host configuration what kind of access has www-data to the repositories data. The sudo line seems limited to git-upload-pack and git-receive-pack.

@Volans: also git-http-backend:

www-data ALL=(phd) SETENV: NOPASSWD: /usr/local/bin/git-http-backend

Additionally, www-data has direct (read-only) access to /srv/repos/ as the files are world readable on disk.