Misc (so far known as others) constitutes all labs projects nfs shares that aren't tools or maps. This data is still served from labstore1001 and the next step is to move this to the secondary nfs cluster (labstore1004/5).
Prior to the maintenance window
- Announce to labs-l, labs-announce, engineering/wikitech-l(?) tech-news(?)
- Mount misc from labstore1001 and labstore-secondary simultaneously (https://gerrit.wikimedia.org/r/329711 - Tested on nfs-test.testlabs and works fine)
- Sync others 24 hours before migration to labstore-secondary and labstore200*
- Check up on these nodes with others
Puppet run issues in these nodes
- dumps-3.dumps.eqiad.wmflabs -> Var is full? [FIXED]
- relic-relics.testlabs.eqiad.wmflabs -> Var is full? [FIXED]
- labs-dnsrecursor-test.openstack.eqiad.wmflabs -> Puppet errors
- labs-dynamicproxy-test.openstack.eqiad.wmflabs -> Puppet errors
SSH Auth fails for following instances prior to migration:
- fastcci-puppetmaster.fastcci.eqiad.wmflabs [Can log in as root, but Puppet Errors]
- phab-test.contributors.eqiad.wmflabs [Yuvi can't log in either]
- wmt-exec.wmt.eqiad.wmflabs (only instance in project) [Yuvi can't log in either]
- wsexport.wikisource-tools.eqiad.wmflabs (only instance in project) [Yuvi can't log in either]
- wtui-new.wikidata-topicmaps.eqiad.wmflabs (only instance in project) [Can log in as root, Puppet Errors]
During migration:
- Update lists/irc channel on start of migration
- Silence shinken alarms (shinken-01 instance kill ircecho and disable puppet)
- Silence icinga on labstore1001, and 1004
- Disable puppet across misc
- Make others share on labstore1001 and labstore1004 (because dual mount) Read Only (export as ro)
- service nfs-exportd stop
- staged a copy of /etc/export.d as of 1/17 for all things ro at /root/export_d_ro
- exportfs -ra (will activate a new config)
- Keep puppet disabled
- Snapshot and latest sync of the others/misc share (Last one took ~50 minutes)
- Merge gerrit patch to remove the mount from 1001 (Removing defn from nfs-mount.yaml), and symlink mount path - /mnt/nfs/labstore-secondary-project to /data/project and /mnt/nfs/labstore-secondary-home to /home on tools
- https://gerrit.wikimedia.org/r/#/c/332735/
- Run nfs-exportd on labstore-secondary to make sure the mount is exported to all labs hosts <== comes w/ nfs-manage up
- For each instance - enable and run puppet twice (If necessary, reboot any problematic nodes after running puppet)
Post Migration
- Follow up on lists/irc
- Consider cleaning up following files/directories /srv/others
volumes-without-projects
orphan-volumes
analytics --> not nfs anymore
cvresearch --> deleted project
fatg --> deleted project
deployment-prep --> not nfs anymore
gsoc2014-fonttailor-demo --> deleted project
maps-team --> only scratch currently mounted
wikidata-quality --> deleted project
wikispy --> deleted project
wikidata-query --> only scratch and dumps mounted
wikistats --> not nfs anymore
ircnotifier --> deleted project
megacron --> deleted project?
Success criteria:
- All labs instances in the list below have /data/project and /home(if mounted) symlinked and mounted from the secondary NFS cluster
Rollback strategy:
- Disable puppet across tools
- Merge gerrit patch that will make mounts from labstore1001 present on nfsclient.pp
- Export misc nfs share as rw on labstore1001
- Run puppet to have the share from 1001 symlinked to /home and /data/project on clients
- List of misc projects with nfs mounted
- catgraph
- account-creation-assistance
- contributors
- wikidata-topicmaps
- sugarcrm
- wikidumpparse
- video
- openstack
- testlabs
- quarry
- huggle
- editor-engagement
- utrs
- wmt
- cvn
- fastcci
- toolsbeta
- project-proxy
- dumps
- bots
- snuggle
- math
- wikisource-tools
List of nodes with nfs mounted
- mwdiffstuff.catgraph.eqiad.wmflabs
- fridolin.catgraph.eqiad.wmflabs
- fishbone.catgraph.eqiad.wmflabs
- sylvester.catgraph.eqiad.wmflabs
- accounts-dbslave.account-creation-assistance.eqiad.wmflabs
- accounts-db3.account-creation-assistance.eqiad.wmflabs
- accounts-mwoauth.account-creation-assistance.eqiad.wmflabs
- accounts-appserver4.account-creation-assistance.eqiad.wmflabs
- accounts-db2.account-creation-assistance.eqiad.wmflabs
- phab-test.contributors.eqiad.wmflabs
- contributors-metrics.contributors.eqiad.wmflabs
- wtui-new.wikidata-topicmaps.eqiad.wmflabs
- officetools.sugarcrm.eqiad.wmflabs
- wigi.wikidumpparse.eqiad.wmflabs
- videodev.video.eqiad.wmflabs
- gfg01.video.eqiad.wmflabs
- encoding01.video.eqiad.wmflabs
- encoding02.video.eqiad.wmflabs
- encoding03.video.eqiad.wmflabs
- video-redis.video.eqiad.wmflabs
- t104588-test.openstack.eqiad.wmflabs
- labs-traefikbuild.openstack.eqiad.wmflabs
- labs-dynamicproxy-new2.openstack.eqiad.wmflabs
- labs-dnsrecursor-test.openstack.eqiad.wmflabs
- labs-bootstrapvz-jessie.openstack.eqiad.wmflabs
- labs-dynamicproxy-test.openstack.eqiad.wmflabs
- labs-vmbuilder-trusty.openstack.eqiad.wmflabs
- labs-vmbuilder-precise.openstack.eqiad.wmflabs
- saltreactortest.testlabs.eqiad.wmflabs
- nfs-test-02.testlabs.eqiad.wmflabs
- util-abogott.testlabs.eqiad.wmflabs
- relic-relics.testlabs.eqiad.wmflabs
- quarry-runner-01.quarry.eqiad.wmflabs
- quarry-runner-02.quarry.eqiad.wmflabs
- quarry-main-01.quarry.eqiad.wmflabs
- huggle-d2.huggle.eqiad.wmflabs
- huggle-pg.huggle.eqiad.wmflabs
- huggle-win32.huggle.eqiad.wmflabs
- huggle.huggle.eqiad.wmflabs
- deferred-changes.editor-engagement.eqiad.wmflabs
- flow-tests.editor-engagement.eqiad.wmflabs
- docs.editor-engagement.eqiad.wmflabs
- mwui.editor-engagement.eqiad.wmflabs
- utrs-secondary.utrs.eqiad.wmflabs
- utrs-primary.utrs.eqiad.wmflabs
- wmt-exec.wmt.eqiad.wmflabs
- cvn-app7.cvn.eqiad.wmflabs
- cvn-app6.cvn.eqiad.wmflabs
- cvn-apache8.cvn.eqiad.wmflabs
- fastcci-puppetmaster.fastcci.eqiad.wmflabs
- fastcci-main.fastcci.eqiad.wmflabs
- fastcci-master.fastcci.eqiad.wmflabs
- fastcci-worker1.fastcci.eqiad.wmflabs
- toolsbeta-puppetdb-03.toolsbeta.eqiad.wmflabs
- toolsbeta-puppetdb-02.toolsbeta.eqiad.wmflabs
- toolsbeta-valhallasw-puppet-compiler-4.toolsbeta.eqiad.wmflabs
- toolsbeta-valhallasw-puppet-compiler-3.toolsbeta.eqiad.wmflabs
- toolsbeta-puppetmaster7.toolsbeta.eqiad.wmflabs
- test-spm-1.project-proxy.eqiad.wmflabs
- novaproxy-02.project-proxy.eqiad.wmflabs
- novaproxy-01.project-proxy.eqiad.wmflabs
- bugzilla.dumps.eqiad.wmflabs
- dumps-3.dumps.eqiad.wmflabs
- dumps-2.dumps.eqiad.wmflabs
- dumps-1.dumps.eqiad.wmflabs
- dumps-stats.dumps.eqiad.wmflabs
- botbot.bots.eqiad.wmflabs
- wm-bot.bots.eqiad.wmflabs
- snuggle-enwiki-01.snuggle.eqiad.wmflabs
- snuggle-en.snuggle.eqiad.wmflabs
- hadoop000.math.eqiad.wmflabs
- hadoop002.math.eqiad.wmflabs
- hadoop003.math.eqiad.wmflabs
- drmf2017.math.eqiad.wmflabs
- drmf-beta.math.eqiad.wmflabs
- drmf2016.math.eqiad.wmflabs
- math-de.math.eqiad.wmflabs
- mathosphere.math.eqiad.wmflabs
- mathoid.math.eqiad.wmflabs
- math-ru.math.eqiad.wmflabs
- drmf.math.eqiad.wmflabs
- mathoid2.math.eqiad.wmflabs
- wsexport.wikisource-tools.eqiad.wmflabs