Page MenuHomePhabricator
Create Task
Maniphest T432587

Migrate to dumps-nfs.w.o in Cloud VPS
Open, Needs TriagePublic
Actions

Description

This task tracks the deployment/migration from individual clouddumps mounts to mounting dumps-nfs.w.o and automatic client recovery. Similar to T432212, however there's a few cases to consider:

  1. Puppet-only VMs that mount clouddumps, those are covered by turning on dumps_use_nfs_lb: true
  2. Puppet + k8s VMs, e.g. Toolforge. For Toolforge those will be covered by T432325, there might be others that will need auditing in this task
  3. Magnum clusters. For PAWS, those will be covered by T432583, there might be others that will need auditing in this task
  4. Unpuppetized VMs, to be audited

Similar to T432212, there might be also code that uses /mnt/nfs/clouddumps* paths as opposed to canonical locations. If such code is indexed by codesearch then we can proactively do something, if it is not then it'll be harder to spot usages.

Also a cloud-announce post is warranted with heads up for this work and timeline, once the deployment is expanded to more projects / is the default. I was thinking of the following:

  • make sure /mnt/nfs/clouddumps* paths are not hardcoded, use /public/dumps instead
  • write and point to documentation for folks needing dumps-nfs mounted inside containers (e.g. for docker --mount type=bind,source=/mnt/nfs,target=/...,bind-propagation=rslave)

Cloud VPS VMs connected to clouddumps* (excluding toolforge and paws) as of Jul 20th

dumps-10.dumps
generator01.dumpstorrents
etytree-b.etytree
services-new.glamwikidashboard
webapp-new.glamwikidashboard
language-lab.language
lpl-cx-sx2.language
lpl-mleb-master.language
lpl-mleb-stable.language
lpl-recommend.language
lpl-services.language
mint.language
maps-nfs-2.maps
maps-osmdb.maps
maps-warper4.maps
maps-wma2.maps
math-docker-2.math
math24.math
osmit-estratti.osmit
osmit-podoma.osmit
osmit-tm4.osmit
mentorship-search.recommendation-api
research-opensearch.recommendation-api
semantic-search.recommendation-api
trending-indicators.recommendation-api
wikinav-bookworm.research-collaborations-api
bullseye.spi-tools
opensearch2.spi-tools
nfs-client-2.testlabs
larynx-be-01.text-to-speech
mimic-be-01.text-to-speech
encoding01.video
encoding02.video
encoding03.video
encoding04.video
encoding05.video
encoding06.video
video-dev-bookworm.video
video-nfs-2.video
video-redis-bookworm.video
backend.wikicommunityhealth
humaniki-prod-bw.wikidumpparse
manager.wikiqlever
worker-1.wikiqlever
worker-2.wikiqlever

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
hieradata: enable dumps-nfs.w.o usageFilippo Giunchedioperations/puppetproduction+4 -0
networktests: use dumps canonical pathsFilippo Giunchedioperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Mon, Jul 20, 10:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Jul 20, 10:32 AM
fgiunchedi updated the task description. (Show Details)Mon, Jul 20, 11:49 AM
fgiunchedi added a comment.Wed, Jul 22, 7:32 AM

I audited the hosts above looking for running containers, and if any were mounting clouddumps nfs. None did which is good news since no action will be required for such containers.

fgiunchedi updated the task description. (Show Details)Wed, Jul 22, 7:36 AM
gerritbot added a comment.Wed, Jul 22, 8:13 AM

Change #1313830 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] networktests: use dumps canonical paths

https://gerrit.wikimedia.org/r/1313830

gerritbot added a project: Patch-For-Review.Wed, Jul 22, 8:13 AM
gerritbot added a comment.Tue, Jul 28, 7:21 AM

Change #1313830 merged by Filippo Giunchedi:

[operations/puppet@production] networktests: use dumps canonical paths

https://gerrit.wikimedia.org/r/1313830

Maintenance_bot removed a project: Patch-For-Review.Tue, Jul 28, 7:31 AM
gerritbot added a comment.Wed, Jul 29, 10:06 AM

Change #1319053 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: enable dumps-nfs.w.o usage

https://gerrit.wikimedia.org/r/1319053

gerritbot added a project: Patch-For-Review.Wed, Jul 29, 10:06 AM
gerritbot added a comment.Thu, Jul 30, 8:21 AM

Change #1319053 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: enable dumps-nfs.w.o usage

https://gerrit.wikimedia.org/r/1319053

Maintenance_bot removed a project: Patch-For-Review.Thu, Jul 30, 8:31 AM
fgiunchedi added a comment.EditedThu, Jul 30, 10:13 AM

Change to default to dumps-nfs.w.o is deployed !

There are a few stragglers that didn't switch:

These hosts run docker and bind-mount the dumps directories. For these a switch to --mount type=bind,source=/mnt/nfs,target=TARGET,bind-propagation=rslave is needed. Note that we switch to mounting /mnt/nfs as a whole with propagation host -> container. Another possibility is to stop bind-mounting dumps if not needed.
For higher level container orchestration, this is the equivalent idea for k8s:

volumeMounts:
  - name: mnt-nfs
    mountPath: /mnt/nfs
    readOnly: true
    mountPropagation: HostToContainer

and of course

volumes:
  - name: mnt-nfs
    hostPath:
      path: /mnt/nfs
  • backend.wikicommunityhealth
  • wikinav-bookworm.research-collaborations-api
  • manager.wikiqlever
  • worker-1.wikiqlever
  • worker-2.wikiqlever
fgiunchedi updated the task description. (Show Details)Thu, Jul 30, 10:14 AM
fgiunchedi created subtask T433667: Change mount propagation for /mnt/nfs in backend.wikicommunityhealth.Fri, Jul 31, 8:52 AM
fgiunchedi created subtask T433668: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api.Fri, Jul 31, 9:00 AM
fgiunchedi created subtask T433671: Change mount propagation for /mnt/nfs in wikiqlever.Fri, Jul 31, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits