Page MenuHomePhabricator
Create Task
Maniphest T105720

Labs team reliability goal for Q1 2015/16
Closed, ResolvedPublic
Actions

Description

Tracking task for Labs team reliability goal for Q1 2015/16.

  1. Meet or exceed 99.5% uptime for each Labs infrastructure service
  2. Remove all Labs support host SPOFs, using redundancy or hot spares
  3. Finish NFS migration to RAID10 storage, and implement NFS sharding
  4. Audit Labs projects on NFS dependencies and support migration to alternatives where appropriate

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedyuvipandaT105720 Labs team reliability goal for Q1 2015/16
 ResolvedAndrewT105723 Eliminate SPOFs in Labs infrastructure
 DuplicatecorenT88716 Make sure tools-db is backed up in some form
 Resolved• jcrespoT88718 Make sure tools-db is replicated somewhere
 ResolvedyuvipandaT98556 Make labs domainproxies fully redundant
 ResolvedAndrewT106141 labnet1001 is a spof
 ResolvedAndrewT106142 holmium is a spof
 ResolvedRobHT106147 New server: labservices1001
 Resolved• jcrespoT114159 New db grants for labservices1001
 DeclinedNoneT104193 Find replacements for various things that people were using NFS for but should not have been (Tracking)
 Resolved• chasempT104194 Simple method to have a per-project debian repository
 DeclinedNoneT102173 implement a simple way to share dotfiles across Cloud VPS project instances
 DuplicateNoneT104206 Provide a simple way to backup arbitrary files from instances
 ResolvedAndrewT102240 Audit projects' use of NFS, and remove it where not necessary
 DuplicateNoneT102368 Remove NFS completely from bastion project
 ResolvedyuvipandaT95924 Move all labs bastions to jessie and disable NFS on them
DeclinedNoneT102369 Remove NFS completely from project-proxy project
 ResolvedKrinkleT102370 Investigate reducing use of NFS in cvn project
 ResolvedyuvipandaT102371 Investigate wether NFS mounts are really necessary for HHVM project
 ResolvedyuvipandaT102372 Disable NFS mounts in extdist project
 ResolvedyuvipandaT102373 Audit and potentially remove all NFS mounts in grantreview project
 ResolvedyuvipandaT102379 Remove NFS mounts from project Akosiaristests
 ResolvedyuvipandaT102380 Remove NFS from project chasetest
 ResolvedyuvipandaT102381 Remove NFS mounts from project cephtest
 ResolvedyuvipandaT102383 Investigate and potentially disable NFS mounts on contintcloud project
 ResolvedyuvipandaT102384 Investigate and disable NFS on 'dns' project
 ResolvedyuvipandaT102385 Turn off NFS for varnish project
 ResolvedyuvipandaT102386 Investigate and potentially turn off NFS in 'design' project
 ResolvedyuvipandaT102388 Investigate and disable NFS on security-tools project
 ResolvedhasharT102169 Disable NFS home directories on deployment-prep
 ResolvedhasharT103731 deployment-parsoid01-test fails puppet: Could not find class role::parsoid
 ResolvedyuvipandaT102391 Audit get rid of NFS usage in social-tools project
 ResolvedyuvipandaT102392 Disable NFS on staging project
 OpenNoneT102402 Find alternative solutions for video project's use of NFS
 DeclinedyuvipandaT102468 Remove and unmount /data/project from projects where it is empty
 ResolvedyuvipandaT102519 Investigate and potentially remove logstash project's use of NFS
 ResolvedNoneT102663 Investigate and remove NFS from editor-engagement project
 ResolvedyuvipandaT102668 Kill NFS mounts in shinken project
 ResolvedAndrewT102672 Kill NFS in openstack project
 ResolvedyuvipandaT102673 Investigate NFS use in services and services-testbed project
 ResolvedyuvipandaT102678 Investigate and potentially get rid of NFS usage in shiny-r project
 ResolvedyuvipandaT102679 Kill NFS mounts in shinken project
 ResolvedAndrewT102680 Investigate and remove NFS mounts in the snuggle project
 ResolvedyuvipandaT102682 Identify (and potentially) help move mwoffliner off NFS
 ResolvedyuvipandaT102688 Investigate and potentially move off NFS in the 'visualeditor' project
 ResolvedyuvipandaT102695 Disable NFS for 'planet' project
 ResolvedyuvipandaT102696 Get rid of NFS on 'nginx' project
 ResolvedyuvipandaT102697 Disable NFS for ogvjs-integration project
 ResolvedyuvipandaT102700 Disable NFS for 'Hadoop' project
 ResolvedyuvipandaT102702 Kill NFS mounts from graphite project
 ResolvedNegative24T102703 Get rid of NFS in the phabricator Labs project
 ResolvedyuvipandaT102704 Kill NFS in scrumbugz project
 ResolvedyuvipandaT102813 Verify NFS is dead on icinga project
 ResolvedyuvipandaT102843 Kill NFS mounts for monitoring project
 ResolvedyuvipandaT102844 Get rid of NFS from the mailman project
 ResolvedyuvipandaT102845 Investigate and get rid of NFS on the etcd project
 ResolvedyuvipandaT102847 Investigate and disable NFS on the swift project
 ResolvedhasharT90610 Continuous integration should not depend on labs NFS
 ResolvedhasharT103312 Cant ssh to integration-slave-jessie-1001.integration.eqiad.wmflabs
 ResolvedyuvipandaT103098 Investigate and disable NFS on mobile project
 ResolvedyuvipandaT103126 Disable NFS on multimedia project
 ResolvedyuvipandaT103130 Investigate if NFS is needed on the language project
 ResolvedyuvipandaT103137 Disable NFS on the orgcharts project
 ResolvedyuvipandaT103148 Investigate NFS alternatives to the wikistream project
 DeclinedNoneT103156 Investigate account-creation-assistance project's use of NFS and look for replacements
 ResolvedyuvipandaT103750 Re-evaluate use of NFS in WMT project
 ResolvedyuvipandaT103755 Investigate and remove NFS from embed-sandbox project
 ResolvedyuvipandaT103757 Investigate and reduce NFS use in maps-team project
 ResolvedyuvipandaT103759 Investigate and remove netflow project's NFS
 ResolvedyuvipandaT103760 Disable NFS in puppet3-diffs project
 ResolvedyuvipandaT103812 Get rid of NFS in the packaging project
 ResolvedyuvipandaT103824 Disable NFS from performance project
 ResolvedyuvipandaT103840 Investigate and disable NFS in the ttmserver project
 ResolvedyuvipandaT103864 Disable NFS for dwl project
 ResolvedyuvipandaT103865 Remove NFS from fundraising project
 ResolvedcorenT104256 Disable NFS for the toolserver-legacy project
 ResolvedyuvipandaT105704 Remove pdbhandler project
 ResolvedyuvipandaT128816 Get rid of NFS from the wikistats project
 Resolved• jkrollT208400 Check whether catgraph project requires NFS or not
 ResolvedKrinkleT208401 Check whether cvn project requires NFS or not
 DeclinedAndrewT208402 Check whether dumps project requires NFS or not
 Resolved• BstormT255628 VPS Project dumps is using 2.4 TB at /data/project on NFS
 ResolvedAndrewT208403 Check whether etytree project requires NFS or not
 ResolvedAndrewT208404 Check whether fastcci project requires NFS or not
 ResolvedAndrewT208405 Check whether huggle project requires NFS or not
 ResolvedAndrewT208406 Check whether maps project requires NFS or not
 ResolvedKrenairT208407 Check whether maps-team project requires NFS or not
 ResolvedAndrewT208408 Check whether math project requires NFS or not
 Resolved• BstormT208409 Check whether mwoffliner project requires NFS or not
 ResolvedAndrewT208410 Check whether osmit project requires NFS or not
 ResolvedNoneT208411 Check whether quarry project requires NFS or not
 ResolvedAndrewT208412 Check whether testlabs project requires NFS or not
 Resolvedjsn.shermanT208413 Check whether twl project requires NFS or not
 ResolvedAndrewT208414 Check whether utrs project requires NFS or not
 ResolvedAndrewT208415 Check whether wikidumpparse project requires NFS or not
 ResolvedAndrewT208416 Check whether wikidata-dev project requires NFS or not
 ResolvedAndrewT208417 Check whether wikidata-primary-source-tool project requires NFS or not
 ResolvedSmalyshevT208418 Check whether wikidata-query project requires NFS or not
 ResolvedAndrewT106144 Make an evacuation plan for labs instances
 ResolvedAndrewT106145 Fix up cold instance migration scheme
 DeclinedNoneT106146 Get instance block-migration working reliably; script and document
 ResolvedAndrewT90821 Upgrade labs cluster to Trusty
 ResolvedAndrewT90822 Upgrade Labs Compute nodes to Trusty
 ResolvedAndrewT90823 Upgrade labs network node to trusty
 ResolvedAndrewT99701 Setup/Install/Deploy labnet1002
 ResolvedRobHT98740 labnet1002
 Resolved• CmjohnsonT99702 relocate WMF3153 (now labnet1002) from row C to B
 Resolved• CmjohnsonT102131 check labnet1002 for pci slot space
 ResolvedAndrewT109329 Try to fail over to labnet1002
 ResolvedAndrewT109653 move nova api to labnet1002
 ResolvedAndrewT90824 Upgrade labs controller to Trusty
 ResolvedAndrewT98226 Glance image files are only stored on virt1000
 ResolvedAndrewT99738 Investigate kernel issues on labvirt** hosts
 ResolvedcorenT106474 Make continuous backups of NFS data to codfw
 InvalidNoneT106871 paramiko (python SSH implementation) needs older hashes for host authentication
 ResolvedcorenT106479 Ensure that labstore machine is 'known good' hardware
 ResolvedcorenT95293 Inspect and diagnose labstore1001's H800 controler
 DeclinedcorenT94607 Test labstore switchover
 ResolvedNoneT85604 Storage capacity & redundancy expansion (tracking)
 ResolvedNoneT85606 Replicate data between codfw and eqiad
 ResolvedcorenT85605 Set storage service up in codfw
 ResolvedcorenT93740 Upgrade labstore2001 to Jessie
 DeclinedyuvipandaT85608 Process for user backups
 ResolvedcorenT93792 Sync up the new labs NFS project filesystem with the live one
 ResolvedcorenT85607 Increase storage available to labs NFS server
 ResolvedcorenT91640 Upgrade labstore1002 to Jessie
 Resolved• CmjohnsonT91677 labstore1002 fails to enter PERC bios, hangs on detecting devices
 Resolved• chasempT98183 labstore1002 issues while trying to reboot
 Resolved• chasempT101741 Locate and assign some MD1200 shelves for proper testing of labstore1002
 ResolvedcorenT96063 Migrate Labs NFS storage from RAID6 to RAID10
 ResolvedcorenT101011 Rsync live labstore filesystem to local eqiad copy
 ResolvedmarkT101010 Make a block-level copy of the codfw mirror of labstore1001 to eqiad
 Resolved• CmjohnsonT101743 Locate spare H800 PERC in case it is necessary to switch labstore1002's
Unknown Object (Task)
 ResolvedcorenT107038 Switch NFS server back to labstore1001
 ResolvedcorenT102478 labstore has multiple unpuppetized files/scripts/configs
 ResolvedcorenT107574 Reinstall labstore1001 and make sure everything is puppet-ready
 ResolvedfaidonT107507 Investigate whether to use Debian's jessie-backports
 DeclinedfaidonT108941 Make certain that jessie-backports is disabled fleetwide.
 ResolvedcorenT106590 Build proper monitoring for making sure that processes that need to run only once on one labstore only are running only once on one labstore only
 ResolvedAndrewT107058 Have catchpoint checks for all labs services
 ResolvedyuvipandaT105721 Identify user facing services labs provides
ResolvedAndrewT97748 Setup a tools checker service that can check all internal services for availability
 ResolvedAndrewT107449 Setup an availability checker for all labsdb hosts
 ResolvedAndrewT107450 Have a checkpoint check for labs proxies
 DeclinedAndrewT107451 Have checkpoint check for public labs DNS
 ResolvedAndrewT107453 Setup checkpoint check for private DNS
ResolvedAndrewT107454 Create a catchpoint check for labs LDAP
 DeclinedyuvipandaT107455 Have checkpoint check for external network access from labs
 ResolvedAndrewT107456 Create a catchpoint check for labs puppetmaster
 ResolvedAndrewT107457 Have checkpoint check for Wikitech availability
Restricted Task
 ResolvedNoneT107066 Measure capacity and utilization of labs services (Tracking)
 ResolvedAndrewT107067 Measure capacity and utilization of labvirt**** servers
 DeclinedcorenT106476 Setup monitoring and reporting for disk space usage of each project on NFS
 Resolved• jcrespoT107070 Measure capacity and utilization of labsdb*** boxes

Event Timeline

yuvipanda created this task.Jul 13 2015, 6:14 PM
yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Cloud-Services.
yuvipanda subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 13 2015, 6:14 PM
yuvipanda added a subtask: T104193: Find replacements for various things that people were using NFS for but should not have been (Tracking).Jul 13 2015, 6:23 PM
yuvipanda added a subtask: T102240: Audit projects' use of NFS, and remove it where not necessary.
Ricordisamoa subscribed.Jul 13 2015, 6:32 PM
yuvipanda added subscribers: mark, coren.Jul 13 2015, 6:54 PM

Not sure about the NFS sharding one - @mark / @coren is that just the tools / others / maps being on different arrays? Or is there more to that? :)

Andrew added a subtask: T99701: Setup/Install/Deploy labnet1002.Jul 17 2015, 2:52 PM
Andrew edited subtasks, added: T106141: labnet1001 is a spof; removed: T99701: Setup/Install/Deploy labnet1002.Jul 17 2015, 2:59 PM
Andrew added a subtask: T106142: holmium is a spof.Jul 17 2015, 3:05 PM
Andrew removed subtasks: T106142: holmium is a spof, T106141: labnet1001 is a spof.
Andrew added a subtask: T106144: Make an evacuation plan for labs instances.Jul 17 2015, 3:08 PM
coren added a comment.Jul 21 2015, 7:18 PM

@yuvipanda: It's keeping the filesystem reasonably small (and operations on them more parallelizable) by spliting along project lines, yes. So right now we've spun off tools and maps with everything else together, but making sure that we locate outliers and split them as an ongoing thing is part of this conceptually.

yuvipanda added a subtask: T101742: start-nfs script warning message is scary and wrong.Jul 22 2015, 9:04 AM
yuvipanda added a subtask: T102478: labstore has multiple unpuppetized files/scripts/configs.Jul 22 2015, 9:07 AM
coren closed subtask T101742: start-nfs script warning message is scary and wrong as Invalid.Jul 27 2015, 2:37 PM
yuvipanda removed a subtask: T105721: Identify user facing services labs provides .Jul 27 2015, 5:53 PM
• MZMcBride subscribed.Jul 27 2015, 11:21 PM
coren closed subtask T106474: Make continuous backups of NFS data to codfw as Resolved.Aug 3 2015, 5:37 PM
coren reopened subtask T106474: Make continuous backups of NFS data to codfw as Open.
coren closed subtask T102478: labstore has multiple unpuppetized files/scripts/configs as Resolved.
yuvipanda removed a subtask: T101742: start-nfs script warning message is scary and wrong.Aug 3 2015, 5:56 PM
coren closed subtask T106474: Make continuous backups of NFS data to codfw as Resolved.Aug 12 2015, 1:19 PM
coren closed subtask T106476: Setup monitoring and reporting for disk space usage of each project on NFS as Declined.Sep 14 2015, 4:37 PM
Andrew closed subtask T105723: Eliminate SPOFs in Labs infrastructure as Resolved.Oct 5 2015, 2:56 PM
coren closed subtask T106479: Ensure that labstore machine is 'known good' hardware as Resolved.Oct 28 2015, 4:02 PM
Andrew closed subtask T107058: Have catchpoint checks for all labs services as Resolved.Oct 28 2015, 4:52 PM
• chasemp triaged this task as High priority.Nov 30 2015, 4:25 PM
• chasemp set Security to None.
coren closed subtask T106590: Build proper monitoring for making sure that processes that need to run only once on one labstore only are running only once on one labstore only as Resolved.Dec 1 2015, 4:17 PM
yuvipanda closed this task as Resolved.Dec 16 2015, 10:33 PM
yuvipanda claimed this task.

I guess?

• Phabricator_maintenance added a project: Goal.Aug 13 2016, 8:39 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:49 PM
Andrew closed subtask T106144: Make an evacuation plan for labs instances as Resolved.Oct 23 2019, 3:57 PM
• Bstorm closed subtask T104193: Find replacements for various things that people were using NFS for but should not have been (Tracking) as Declined.Jun 11 2020, 10:56 PM
bd808 closed subtask T107066: Measure capacity and utilization of labs services (Tracking) as Resolved.Jun 11 2020, 11:43 PM
Andrew closed subtask T102240: Audit projects' use of NFS, and remove it where not necessary as Resolved.Oct 19 2020, 8:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits