Page MenuHomePhabricator

Remove puppet class role::labs::lvm::mnt
Open, NormalPublic

Description

WMCS instances have some extra disk space which is not mounted by default. It can be mounted on /srv using role::labs::lvm::srv. There is a legacy class role::labs::lvm::mnt (which mount it on /mnt) which is deprecated and should be removed entirely.

Continuous integration was among the early adopters of /mnt. They all got migrated to /srv via T146381.

Labs instances should be switched to role::labs::lvm::srv @hashar wrote a migration guide on https://gerrit.wikimedia.org/r/#/c/330412/ :

  • switch from role::labs::lvm::mnt to role::labs::lvm::srv
  • run puppet
  • kill anything that might be using /mnt (lsof /mnt)
  • manually umount /mnt
  • ensure /srv is mounted
  • VERIFY /mnt is no more refered to in /etc/fstab
    • failure to do so would cause the instance to be deadlocked on the next reboot

Instances https://tools.wmflabs.org/openstack-browser/puppetclass/role::labs::lvm::mnt

As of October 20 2017:

Project: deployment-prep Beta-Cluster-Infrastructure - DONE Oct 30th

  • deployment-logstash2.deployment-prep.eqiad.wmflabs

Project: tools PAWS

  • tools-paws-*

Project: wikidata-ldf Wikidata

  • wikidata-ldf.wikidata-ldf.eqiad.wmflabs

Event Timeline

hashar created this task.Oct 20 2017, 9:55 PM

deployment-logstash2.deployment-prep.eqiad.wmflabs has three mounts:

/dev/vd/second-local-disk  /srv                    ext4  defaults  0  0
/dev/vd/second-local-disk  /var/lib/elasticsearch  ext4  defaults  0  0
/dev/vd/second-local-disk  /mnt                    ext4  defaults  0  0
hashar updated the task description. (Show Details)Oct 30 2017, 10:55 AM

Mentioned in SAL (#wikimedia-releng) [2017-10-30T10:56:20Z] <hashar> deployment-logstash2 removed puppet class role::labs::lvm::mnt, replacing with role::labs::lvm::srv . /srv is already mounted. Unmounting /mnt and restarting elastcisearch - T178722

Hello @CristianCantoro

Looks like you have created the instance wikidata-ldf.wikidata-ldf.eqiad.wmflabs. It is apparently mounting the instance extended disk space on /mnt and I would like to get it normalized to use /srv. That is due to the puppet class role::labs::lvm::mnt being applied to it. This task has some instructions to do the migration, but you would also need to adjust any code/settings that refers to /mnt.

Would it be possible to migrate it? I am available to assist.

hashar triaged this task as Normal priority.Nov 9 2017, 2:01 PM

PAWS uses role::labs::lvm::mnt with the mount_point being directed at /var/lib/docker instead of /mnt. Role role::labs::lvm::srv does not allow for that.
To me it seems the sanest path to remove role::labs::lvm::mnt would be develop a PAWS specific role that includes either a mountpoint definition in that path or, ideally, a PAWS specific role that installs docker, requires ::profile::labs::lvm::srv and adds -g /srv/docker to the docker default file.

hashar removed a subscriber: hashar.Mar 22 2019, 11:41 PM