Completely remove Beta Cluster dependency on NFS
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Jun 18 2015, 2:11 PM

Description

In an effort to bring deployment-prep back up while the NFS cluster was completely down, we've disabled NFS from it completely. That broke some things, which is good. We are now removing our dependency on NFS.

Please add blockers to this task.

Details

	Subject	Repo	Branch	Lines +/-
	labs: Remove nfs for deployment-prep \o/	operations/puppet	production	+0 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• AlexMonk-WMF	T102953 Completely remove Beta Cluster dependency on NFS
Resolved	• bd808	T102962 Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd)
Resolved	thcipriani	T102963 Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage
Resolved	hashar	T103300 deployment-bastion: Cannot create /home/l10nupdate/.ssh; parent directory /home/l10nupdate does not exist
Resolved	• AlexMonk-WMF	T64835 Setup a Swift cluster on beta-cluster to match production
Resolved	fgiunchedi	T123512 beta swift labs instances requirements
Resolved	• AlexMonk-WMF	T140220 "UnderflowException from ... FancyCaptcha.class.php: Ran out of captcha images" at Special:CreateAccount
Resolved	hashar	T102169 Disable NFS home directories on deployment-prep
Resolved	hashar	T103731 deployment-parsoid01-test fails puppet: Could not find class role::parsoid
Resolved	yuvipanda	T125624 Disable /data/project for instances in deployment-prep that do not need it

Event Timeline

yuvipanda created this task.Jun 18 2015, 2:11 PM

yuvipanda raised the priority of this task from to Needs Triage.

yuvipanda updated the task description. (Show Details)

yuvipanda added projects: Beta-Cluster-Infrastructure, Cloud-Services.

yuvipanda subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2015, 2:11 PM

+ Continuous-Integration-Infrastructure in case it is impacted

Krenair subscribed.Jun 18 2015, 2:17 PM

hashar updated the task description. (Show Details)Jun 18 2015, 2:23 PM

hashar removed a project: Continuous-Integration-Infrastructure.

hashar set Security to None.

zhuyifei1999 subscribed.Jun 18 2015, 2:24 PM

zhuyifei1999 moved this task from Triage to Tracking on the Cloud-Services board.Jun 18 2015, 2:27 PM

Uploads/thumbnails seem broken (some stuff still seems to be around, but probably due to cache? I think it would all be from NFS, so...)

Logstash-beta is broken, HTTP 500 ISE due to:

[Thu Jun 18 14:53:57 2015] [error] [client 10.68.16.65] (2)No such file or directory: Could not open password file: /data/project/logstash/.htpasswd, referer: https://logstash-beta.wmflabs.org/

@Krenair please fill subtasks as needed. This is a tracking bug :-}

Logstash filled as T102962
UploadWizard as T102963

In T102953#1378766, @hashar wrote:

UploadWizard as T102963

Just turned that into a generic upload issues task.

Bugreporter triaged this task as High priority.Jun 18 2015, 4:32 PM

Bugreporter added a subtask: T102925: Tools rely on NFS.

Bugreporter subscribed.

Krenair removed a subtask: T102925: Tools rely on NFS.Jun 18 2015, 4:34 PM

• bd808 closed subtask T102962: Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd) as Resolved.Jun 19 2015, 9:42 PM

hashar closed subtask T103300: deployment-bastion: Cannot create /home/l10nupdate/.ssh; parent directory /home/l10nupdate does not exist as Resolved.Jun 22 2015, 10:07 AM

thcipriani closed subtask T102963: Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage as Resolved.Jun 22 2015, 7:35 PM

Seems all NFS related breakages have been fixed now.

By NFS being up, sure... Shouldn't we be trying to make it not depend on NFS?

This is not resolved unless nothing in beta is expected to break next time NFS does.

Thanks @Krenair.

Well NFS is gone from the deployment-prep. Its only use now are the upload/thumbnails. We need to migrate to swift which is T64835 blocking T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).

hashar added a subtask: T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).Jun 22 2015, 7:53 PM

hashar removed hashar as the assignee of this task.Jun 22 2015, 7:56 PM

hashar moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.

greg renamed this task from Things broken by betacluster suddenly being moved off NFS to Completely remove Beta Cluster dependency on NFS.Jun 24 2015, 10:25 PM

greg updated the task description. (Show Details)

greg added a subtask: T64835: Setup a Swift cluster on beta-cluster to match production.

greg removed a subtask: T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).

greg added a subtask: T102169: Disable NFS home directories on deployment-prep.

hashar closed subtask T102169: Disable NFS home directories on deployment-prep as Resolved.Jun 25 2015, 10:54 AM

Waiting for T64835: Setup a Swift cluster on beta-cluster to match production

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptOct 6 2015, 1:31 PM

hashar changed the status of subtask T64835: Setup a Swift cluster on beta-cluster to match production from Open to Stalled.Oct 8 2015, 10:27 AM

Luke081515 awarded a token.Nov 3 2015, 6:40 PM

hashar changed the status of subtask T64835: Setup a Swift cluster on beta-cluster to match production from Stalled to Open.Jan 20 2016, 2:14 PM

hashar added a parent task: T125624: Disable /data/project for instances in deployment-prep that do not need it.Feb 11 2016, 8:10 AM

hashar mentioned this in T125624: Disable /data/project for instances in deployment-prep that do not need it.

yuvipanda removed a parent task: T125624: Disable /data/project for instances in deployment-prep that do not need it.Feb 16 2016, 11:18 PM

hashar added a subtask: T125624: Disable /data/project for instances in deployment-prep that do not need it.Feb 17 2016, 12:53 PM

yuvipanda reopened subtask T125624: Disable /data/project for instances in deployment-prep that do not need it as Open.Mar 2 2016, 8:17 PM

yuvipanda closed subtask T125624: Disable /data/project for instances in deployment-prep that do not need it as Resolved.Mar 3 2016, 10:31 PM

hashar mentioned this in T129586: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config.Mar 29 2016, 11:03 AM

I'm far enough into T64835: Setup a Swift cluster on beta-cluster to match production now that I think we can almost call this done.

• AlexMonk-WMF closed subtask T64835: Setup a Swift cluster on beta-cluster to match production as Resolved.Jul 13 2016, 8:37 PM

Given that the only place NFS is mounted in deployment-prep is deployment-upload, which is shutoff and to be deleted at some point, I think we can call this done.

Change 299123 had a related patch set uploaded (by Yuvipanda):
labs: Remove nfs for deployment-prep \o/

https://gerrit.wikimedia.org/r/299123

gerritbot added a project: Patch-For-Review.Jul 15 2016, 10:28 AM

yuvipanda mentioned this in rOPUPad31e422b036: labs: Remove nfs for deployment-prep \o/.Jul 15 2016, 10:33 AM

yuvipanda mentioned this in rOPUP96f9177e54cd: labs: Remove nfs for deployment-prep \o/.Jul 15 2016, 1:20 PM

Change 299123 merged by Yuvipanda:
labs: Remove nfs for deployment-prep \o/

https://gerrit.wikimedia.org/r/299123

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:49 PM