overhaul labstore setup [tracking]
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• chasemp
	Feb 5 2016, 11:03 PM

Description

Container for labstore and nfs improvements and investigation.

Details

Subject	Repo	Branch	Lines +/-
labstore: misc don't shadow remounts default	operations/puppet	production	+0 -2
labstore100[4\|5]: define initial labstore::drbd::resource	operations/puppet	production	+21 -0
labstore: drbd framework for software based HA	operations/puppet	production	+244 -0
labstore secondary role cleanup and remove 00[12] specifics	operations/puppet	production	+4 -28
Match class assignment to file for labstore::fileserver::secondary	operations/puppet	production	+1 -1
assign labs::nfs::secondary to labstore100[45]	operations/puppet	production	+20 -0
labs nfs scratch and dumps to soft mode	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• Bstorm	T126083 overhaul labstore setup [tracking]
Resolved		None	T85606 Replicate data between codfw and eqiad
Resolved		coren	T85605 Set storage service up in codfw
Resolved		coren	T93740 Upgrade labstore2001 to Jessie
Declined		yuvipanda	T85608 Process for user backups
Resolved		coren	T93792 Sync up the new labs NFS project filesystem with the live one
Resolved		None	T104881 NFS exports broken for new projects which want NFS
Resolved		• Bstorm	T122508 Prevent overly-large log files
Open		None	T97861 [toolforge.infra] Provide centralized logging (logstash) for Toolforge
Open		None	T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge webservices and bots
Declined		None	T127368 Estimate hardware requirements for Toolforge logging elastic cluster
Open		None	T152235 Simple logrotate service for users of Tools as stopgap before central logging
Resolved	Feature	Raymond_Ndibe	T302211 toolforge-jobs: merge stdout/stderr output
Duplicate	Feature	Raymond_Ndibe	T304421 Allow customizing the out/err files with toolforge-jobs
Resolved	Feature	Raymond_Ndibe	T301901 Allow specifying the path for log files for jobs executed on the new toolforge Jobs framework
Resolved		None	T327165 toolforge-jobs: add logrotate
Resolved	Feature	taavi	T330715 Allow TJF job logs to go to Kuberenetes output buffer rather than disk
Resolved		taavi	T336057 Add commands to `webservice` and `jobs` to query logs from Kubernetes
Invalid		None	T91806 Change the way manage-nfs-volumes is monitored
Resolved		None	T95766 One instance hammering on NFS should not make it unavailable to everyone else
Resolved		None	T91822 New instances don't always get their /data and /home project mounted from NFS
Resolved		None	T98652 Clean up huge logs on toollabs
Resolved		scfc	T100572 wikiviewstats rapidly fills error.log
Declined		None	T121890 Add step in start-nfs to ask operator to consider dropping some snapshots
Declined		• chasemp	T121905 Reinstall labstore1002 to ensure consistency with labstore1001
Resolved		• Cmjohnson	T123740 Replace labstore1002 RAID card
Declined		None	T124991 evaluate possibility for nscd use with useldap
Resolved		• chasemp	T122743 labs NFS slowness / high load
Resolved		None	T124877 user `marcmiquel` gzipping 41G file on NFS
Resolved		• chasemp	T124822 Find a method for `templatetiger` to sort large files
Declined		None	T126085 change nfs-exports job to only run on changes to /etc/exports.d
Resolved		RobH	T126089 new labstore hardware for eqiad
			Unknown Object (Task)
Resolved		• Cmjohnson	T127490 testing: r430 server / h800 controller / md1200 shelf
			Unknown Object (Task)
			Unknown Object (Task)
Resolved		• chasemp	T133397 eqiad: Rack and setup new labstore
Resolved		None	T126090 change labstore1001/1002 to cfq io scheduler
Resolved		• chasemp	T126237 make network saturation alert on labstore1003 sane
Resolved		• madhuvishy	T126623 track growth of project and home directories
Resolved		• Bstorm	T127559 Soft mount option via hiera for NFS
Resolved		• chasemp	T127560 Soft mount /data/dumps
Resolved		• chasemp	T127561 Soft mount /data/scratch
Resolved		• madhuvishy	T127567 revise/fix labstore replicate backup jobs
Resolved		Papaul	T128083 Figure out what labstore hardware is viable in codfw
Invalid		None	T117453 Unable to connect both redundant labstores to the shelves in parallel
Declined		None	T121629 Investigate better way of deferring activation of Labs LVM volumes (and corresponding snapshots) until after system boot
Resolved		• madhuvishy	T134896 move nfs /scratch to labstore1003
Resolved		• chasemp	T140483 Create an NFS mount manager
Resolved		• Cmjohnson	T144183 Connect secondary nic for labstore1004 and labstore1005
Resolved		• madhuvishy	T144255 Sync data for tools-project from labstore1001 to labstore1004/5
Resolved		• madhuvishy	T144633 Set up monitoring for secondary labstore HA cluster
Resolved		• chasemp	T146153 Performance test new secondary labstore HA cluster
Resolved		• chasemp	T146154 Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking]
Resolved		• chasemp	T149946 Add config option in tools webservice debian package to write logs to /dev/null
Resolved		• madhuvishy	T147657 Move maps share to labstore1003
Resolved		• madhuvishy	T154336 Migrate misc to secondary labstore HA cluster
Declined		None	T153728 Duplicate /proc/mounts entries on labstore1001
Resolved		• madhuvishy	T149870 Set up backups of tools and misc data from labstore1004/5 in labstore2003/4
Resolved		• madhuvishy	T156982 Cleanup tools nfs share on labstore1004/5
Resolved		• madhuvishy	T158196 Reimage labstore1001 and labstore1002 for DRBD storage setup
Resolved		• madhuvishy	T158913 Move labstore1002 and labstore1002-array1 and labstore1002-array2 to different rack (currently in C3)
Invalid		• madhuvishy	T162090 Investigate alternative RAID strategies for labstore1001/2
Resolved		• Bstorm	T158883 Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories')
Declined		None	T161834 Undo special tools-home and tools-project share definitions for NFS
Resolved		• Bstorm	T161835 Convert labstore cluster configuration to hiera and profiles
Resolved		• Bstorm	T171394 Better monitoring for labstore backup crons
Resolved		• madhuvishy	T171623 Split up labstore external shelf storage available in codfw between labstore2001 and 2

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 289903 had a related patch set uploaded (by Rush):
labs nfs scratch and dumps to soft mode

https://gerrit.wikimedia.org/r/289903

gerritbot added a project: Patch-For-Review.May 20 2016, 7:22 PM

Change 289903 merged by Rush:
labs nfs scratch and dumps to soft mode

https://gerrit.wikimedia.org/r/289903

• chasemp mentioned this in rOPUP6d1c8000106f: labs nfs scratch and dumps to soft mode.May 20 2016, 7:37 PM

• chasemp mentioned this in rOPUP62e0107519ed: introduce fileserver::secondary role.Jun 1 2016, 8:18 PM

Change 292558 had a related patch set uploaded (by Rush):
assign labs::nfs::secondary to labstore100[45]

https://gerrit.wikimedia.org/r/292558

Change 292558 merged by Rush:
assign labs::nfs::secondary to labstore100[45]

https://gerrit.wikimedia.org/r/292558

Change 292559 had a related patch set uploaded (by Rush):
Match class assignment to file for labstore::fileserver::secondary

https://gerrit.wikimedia.org/r/292559

• chasemp mentioned this in rOPUP55e58a226e16: assign labs::nfs::secondary to labstore100[45].Jun 3 2016, 1:59 PM

Change 292559 merged by Rush:
Match class assignment to file for labstore::fileserver::secondary

https://gerrit.wikimedia.org/r/292559

• chasemp mentioned this in rOPUP837beefa2943: Match class assignment to file for labstore::fileserver::secondary.Jun 3 2016, 2:08 PM

Change 292561 had a related patch set uploaded (by Rush):
labstore secondary role cleanup and remove 00[12] specifics

https://gerrit.wikimedia.org/r/292561

Change 292561 merged by Rush:
labstore secondary role cleanup and remove 00[12] specifics

https://gerrit.wikimedia.org/r/292561

• chasemp mentioned this in rOPUPa859bac59753: labstore secondary role cleanup and remove 00[12] specifics.Jun 3 2016, 2:20 PM

• chasemp mentioned this in rOPUP0aa7405d13e1: labstore secondary role cleanup and remove 00[12] specifics.Jun 17 2016, 6:06 PM

• chasemp mentioned this in rOPUP1da14b7a5206: introduce fileserver::secondary role.

• chasemp mentioned this in rOPUP29005a43e058: introduce fileserver::secondary role.

• chasemp mentioned this in rOPUP01188dd7e831: introduce fileserver::secondary role.

• chasemp mentioned this in rOPUP96da5285cb71: introduce fileserver::secondary role.

• chasemp mentioned this in rOPUP627d42133c49: labs nfs scratch and dumps to soft mode.

• chasemp mentioned this in rOPUP98cd6a36a82c: labstore::fileserver remove use_ldap option.

• chasemp closed subtask T127561: Soft mount /data/scratch as Resolved.Jul 15 2016, 2:53 PM

• chasemp closed subtask T127560: Soft mount /data/dumps as Resolved.Jul 15 2016, 2:56 PM

• chasemp created subtask T140483: Create an NFS mount manager.Jul 15 2016, 5:36 PM

• chasemp closed subtask T140483: Create an NFS mount manager as Resolved.Aug 17 2016, 9:00 PM

Change 305663 had a related patch set uploaded (by Rush):
labstore: drbd framework for software based HA

https://gerrit.wikimedia.org/r/305663

• chasemp mentioned this in rOPUP1ff7eba4acf8: labstore: drbd framework for software based HA.Aug 19 2016, 5:15 PM

• chasemp mentioned this in rOPUP9a4963ac6ef1: labstore: drbd framework for software based HA.Aug 19 2016, 7:09 PM

• chasemp mentioned this in rOPUP165b4ed5b4b2: labstore: drbd framework for software based HA.Aug 19 2016, 7:20 PM

Change 305663 merged by Rush:
labstore: drbd framework for software based HA

https://gerrit.wikimedia.org/r/305663

• chasemp mentioned this in rOPUP854a2680709d: labstore: drbd framework for software based HA.Aug 19 2016, 8:53 PM

Change 305738 had a related patch set uploaded (by Rush):
labstore100[4|5]: define initial labstore::drbd::resource

https://gerrit.wikimedia.org/r/305738

• chasemp mentioned this in rOPUP7e8fc843b417: labstore100[4|5]: define initial labstore::drbd::resource.Aug 19 2016, 9:15 PM

• chasemp mentioned this in rOPUPaf80afd5536f: labstore100[4|5]: define initial labstore::drbd::resource.

Change 305738 merged by Madhuvishy:
labstore100[4|5]: define initial labstore::drbd::resource

https://gerrit.wikimedia.org/r/305738

• chasemp mentioned this in rOPUP39e295f40ec5: labstore100[4|5]: define initial labstore::drbd::resource.Aug 19 2016, 9:47 PM

• chasemp mentioned this in rOPUPc949d7c994d4: labstore100[4|5]: define initial labstore::drbd::resource.

• chasemp mentioned this in T144183: Connect secondary nic for labstore1004 and labstore1005.Aug 29 2016, 2:15 PM

• chasemp created subtask T144183: Connect secondary nic for labstore1004 and labstore1005.

• madhuvishy created subtask T144255: Sync data for tools-project from labstore1001 to labstore1004/5.Aug 30 2016, 5:39 AM

• Cmjohnson closed subtask T144183: Connect secondary nic for labstore1004 and labstore1005 as Resolved.Aug 30 2016, 3:07 PM

• madhuvishy closed subtask T134896: move nfs /scratch to labstore1003 as Resolved.Aug 31 2016, 10:08 PM

• madhuvishy created subtask T144633: Set up monitoring for secondary labstore HA cluster.Sep 2 2016, 7:51 PM

Change 309311 had a related patch set uploaded (by Rush):
labstore: misc don't shadow remounts default

https://gerrit.wikimedia.org/r/309311

Change 309311 merged by Rush:
labstore: misc don't shadow remounts default

https://gerrit.wikimedia.org/r/309311

• madhuvishy created subtask T146153: Performance test new secondary labstore HA cluster.Sep 20 2016, 3:05 PM

• madhuvishy created subtask T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking].

• madhuvishy created subtask T147657: Move maps share to labstore1003.Oct 7 2016, 5:24 PM

• chasemp closed subtask T91806: Change the way manage-nfs-volumes is monitored as Invalid.Oct 7 2016, 8:39 PM

• chasemp closed subtask T144255: Sync data for tools-project from labstore1001 to labstore1004/5 as Resolved.Nov 16 2016, 3:30 PM

• chasemp closed subtask T127567: revise/fix labstore replicate backup jobs as Resolved.

• chasemp closed subtask T146153: Performance test new secondary labstore HA cluster as Resolved.Nov 16 2016, 3:43 PM

• chasemp closed subtask T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] as Resolved.

• chasemp reopened subtask T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] as Open.Nov 16 2016, 3:46 PM

• chasemp closed subtask T147657: Move maps share to labstore1003 as Resolved.

• chasemp closed subtask T126085: change nfs-exports job to only run on changes to /etc/exports.d as Declined.Nov 16 2016, 3:50 PM

• chasemp closed subtask T146154: Migrate tools to secondary labstore HA cluster (Scheduled on 11/14) [tracking] as Resolved.Nov 22 2016, 3:21 PM

• chasemp closed subtask T124822: Find a method for `templatetiger` to sort large files as Resolved.Dec 2 2016, 6:46 PM

• madhuvishy created subtask T154336: Migrate misc to secondary labstore HA cluster.Dec 30 2016, 9:48 PM

• chasemp added a subtask: T153728: Duplicate /proc/mounts entries on labstore1001.Jan 5 2017, 3:39 PM

• madhuvishy closed subtask T144633: Set up monitoring for secondary labstore HA cluster as Resolved.Jan 25 2017, 10:13 PM

• chasemp added a subtask: T149870: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4.Jan 25 2017, 10:20 PM

• madhuvishy closed subtask T154336: Migrate misc to secondary labstore HA cluster as Resolved.Jan 27 2017, 5:27 PM

• madhuvishy created subtask T156982: Cleanup tools nfs share on labstore1004/5.Feb 2 2017, 12:29 AM

• chasemp created subtask T158196: Reimage labstore1001 and labstore1002 for DRBD storage setup.Feb 15 2017, 3:05 PM

• chasemp closed subtask T121890: Add step in start-nfs to ask operator to consider dropping some snapshots as Declined.Feb 15 2017, 3:07 PM

• chasemp closed subtask T121905: Reinstall labstore1002 to ensure consistency with labstore1001 as Declined.

• chasemp closed subtask T121629: Investigate better way of deferring activation of Labs LVM volumes (and corresponding snapshots) until after system boot as Declined.

• chasemp closed subtask T153728: Duplicate /proc/mounts entries on labstore1001 as Declined.

• madhuvishy closed subtask T126623: track growth of project and home directories as Resolved.Feb 23 2017, 11:42 PM

• madhuvishy closed subtask T149870: Set up backups of tools and misc data from labstore1004/5 in labstore2003/4 as Resolved.Mar 14 2017, 6:14 PM

• chasemp added a subtask: T158883: Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories').Mar 30 2017, 7:02 PM

• chasemp created subtask T161834: Undo special tools-home and tools-project share definitions for NFS.Mar 30 2017, 7:08 PM

• chasemp created subtask T161835: Convert labstore cluster configuration to hiera and profiles.Mar 30 2017, 7:10 PM

• madhuvishy closed subtask T156982: Cleanup tools nfs share on labstore1004/5 as Resolved.Jul 23 2017, 12:21 AM

• madhuvishy created subtask T171394: Better monitoring for labstore backup crons.Jul 23 2017, 12:28 AM

bd808 edited projects, added Data-Services; removed Patch-For-Review, Cloud-Services.Jul 23 2017, 11:34 PM

bd808 moved this task from Backlog to Shared Storage on the Data-Services board.

• madhuvishy created subtask T171623: Split up labstore external shelf storage available in codfw between labstore2001 and 2.Jul 25 2017, 5:29 PM

• madhuvishy closed subtask T171623: Split up labstore external shelf storage available in codfw between labstore2001 and 2 as Resolved.Aug 30 2017, 5:43 AM

• madhuvishy closed subtask T158196: Reimage labstore1001 and labstore1002 for DRBD storage setup as Resolved.Jan 5 2018, 6:52 PM

• Bstorm closed subtask T171394: Better monitoring for labstore backup crons as Resolved.Aug 24 2018, 6:12 PM

• Bstorm reopened subtask T171394: Better monitoring for labstore backup crons as Open.Aug 31 2018, 2:26 PM

• Bstorm closed subtask T171394: Better monitoring for labstore backup crons as Resolved.Sep 5 2018, 8:05 PM

• GTirloni subscribed.Oct 31 2018, 6:06 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:31 PM

• Bstorm closed subtask T122508: Prevent overly-large log files as Resolved.Mar 14 2019, 6:06 PM

• GTirloni unsubscribed.Mar 21 2019, 9:11 PM

• GTirloni added a project: cloud-services-team (Kanban).Mar 23 2019, 9:25 PM

• chasemp closed subtask T124991: evaluate possibility for nscd use with useldap as Declined.Jan 2 2020, 9:09 PM

• Bstorm closed subtask T161834: Undo special tools-home and tools-project share definitions for NFS as Declined.Jun 11 2020, 10:15 PM

• Bstorm closed subtask T158883: Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories') as Resolved.Jun 11 2020, 10:18 PM

Only two of the subtasks are still open and on the workboard. They are very important and will be pursued soon. That said, the next NFS project will not be this particular overhaul, and a new epic will be needed. I'll mark this one resolved for now.

• Bstorm closed subtask T127559: Soft mount option via hiera for NFS as Resolved.Jun 16 2020, 9:33 PM

• Bstorm closed subtask T161835: Convert labstore cluster configuration to hiera and profiles as Resolved.Jun 22 2020, 11:19 PM

overhaul labstore setup [tracking]Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

overhaul labstore setup [tracking]
Closed, ResolvedPublic
Actions

Related Objects
Search...