Page MenuHomePhabricator
Create Task
Maniphest T309346

Replace labstore100[67] with clouddumps100[12]
Closed, ResolvedPublic
Actions

Description

The dumps workloads on labstore1006 and 7 can be moved to the new clouddumps hosts. The new hosts have lots of built-in storage so this move will let us decom the old servers and disk shelves used by 1006/1007

Details

SubjectRepoBranchLines +/-
Dumps: remove a bunch of references to labstore1006 and labstore1007operations/puppetproduction+2 -14
rsync-via-primary.sh: replace labstore with clouddumpsoperations/puppetproduction+4 -4
Dumps: stop mounting the old labstore100x servers on VMsoperations/puppetproduction+15 -1
Dumps: switch to using clouddumps hosts rather than the old labstores.operations/puppetproduction+2 -3
openstack network tests: switch to check clouddumps1001 mountsoperations/puppetproduction+2 -2
Dumps: switch to using clouddumps hosts rather than the old labstores.operations/puppetproduction+3 -3
Move dumps from labstore1006 to clouddumps1001operations/dnsmaster+1 -1
Add dummy keytabs for new clouddumps100* serverslabs/privatemaster+0 -0
put clouddumps100[12] into serviceoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.May 26 2022, 7:16 PM
Restricted Application edited projects, added cloud-services-team (Kanban); removed Patch-For-Review, cloud-services-team (Hardware). · View Herald TranscriptMay 26 2022, 7:16 PM
Andrew reassigned this task from Andrew to ArielGlenn.May 26 2022, 7:18 PM
Andrew edited subscribers, added: ArielGlenn; removed: Jclark-ctr, cmooney, Volans and 3 others.

@ArielGlenn these two new servers should be ready; I'm hoping that you have the time to move the data and workload over.

Currently each has a gigantic sdb1 lvm partition that isn't mounted anywhere. If you want me to mess with partman more and get these set up some different way I'm happy to give it a go, but I'm pretty sure that moving ahead with puppetized or by-hand lvm commands is the right next step.

Hokwelum subscribed.May 30 2022, 1:46 PM
ArielGlenn removed ArielGlenn as the assignee of this task.May 30 2022, 2:05 PM

Not sure who should get this next but it's not Hannah or I :-) I was never involved in the configuration and setup of the old labstore boxes, so I wouldn't have any idea how they need to be set up. I guess that was probably all Brooke's work.

When the servers are ready for rsync, you'll want to grab the data off of the old boxes, and then let me know and we can change a couple names and sync to them instead of the current labstore1006,7. It's just flipping a few variable names in puppet. But if we want to minimize extra syncing, the main rsync from the labstores should be run a few times for the bulk of the data and one final time on say the 17th of the month when the sql/xml dumps are complete, then I can swap our end over to point to those with almost no additional copying needed.

For the fetches from stats hosts and such, I guess those are separate settings in puppet that can just be swapped as well on your end, once the main rsyncs have gotten done.

Note that there's the open task about putting dumps.wikimedia.org behind our cdn T306550 related to this task.

• MoritzMuehlenhoff triaged this task as Medium priority.May 30 2022, 5:42 PM
• Cmjohnson removed a project: DC-Ops.May 31 2022, 4:41 PM

This task does not require DC-OPs tag, once you have moved the data, please decommission labstores and create a task for DC-OPs,

gerritbot added a comment.Jun 2 2022, 7:50 PM

Change 802600 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] put clouddumps100[12] into service

https://gerrit.wikimedia.org/r/802600

gerritbot added a project: Patch-For-Review.Jun 2 2022, 7:50 PM
wiki_willy assigned this task to • Cmjohnson.Jun 8 2022, 8:43 PM
Andrew claimed this task.Jun 8 2022, 10:58 PM
Andrew added a subscriber: • Cmjohnson.

I think this should be assigned to me, to put the new hosts into service. That's currently blocked by a lot of partman nonsense that Papaul is helping me with; that's documented on T302981

gerritbot added a comment.Jun 12 2022, 7:07 PM

Change 802600 merged by Andrew Bogott:

[operations/puppet@production] put clouddumps100[12] into service

https://gerrit.wikimedia.org/r/802600

wiki_willy removed a project: ops-eqiad.Jun 30 2022, 4:46 PM
ayounsi removed a project: netops.Jul 4 2022, 7:38 AM
gerritbot added a comment.Aug 18 2022, 2:51 PM

Change 824513 had a related patch set uploaded (by Btullis; author: Btullis):

[labs/private@master] Add dummy keytabs for new clouddumps100* servers

https://gerrit.wikimedia.org/r/824513

gerritbot added a comment.Aug 18 2022, 2:52 PM

Change 824513 merged by Btullis:

[labs/private@master] Add dummy keytabs for new clouddumps100* servers

https://gerrit.wikimedia.org/r/824513

BTullis mentioned this in rLPRI3809fc0dd3a4: Add dummy keytabs for new clouddumps100* servers.Aug 23 2022, 6:34 PM
Andrew closed subtask T316123: Auth extremely slow on clouddumps100[12] as Resolved.Aug 30 2022, 7:15 PM
gerritbot added a comment.Aug 30 2022, 10:01 PM

Change 828102 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Dumps: switch to using clouddumps hosts rather than the old labstores.

https://gerrit.wikimedia.org/r/828102

gerritbot added a comment.Aug 30 2022, 10:01 PM

Change 828103 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Dumps: stop mounting the old labstore100x servers on VMs

https://gerrit.wikimedia.org/r/828103

Andrew added a comment.Aug 30 2022, 10:02 PM

Useful context for this at https://wikitech.wikimedia.org/wiki/Dumps/XML-SQL_Dumps#A_labstore_host_dies_(web_or_nfs_server_for_dumps)

Andrew closed subtask T310451: hdfs client packages for debian Bullseye as Resolved.Sep 1 2022, 8:19 PM
gerritbot added a comment.Sep 6 2022, 4:38 PM

Change 830200 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/dns@master] Move dumps from labstore1006 to clouddumps1001

https://gerrit.wikimedia.org/r/830200

gerritbot added a comment.Sep 6 2022, 4:44 PM

Change 828102 merged by Andrew Bogott:

[operations/puppet@production] Dumps: switch to using clouddumps hosts rather than the old labstores.

https://gerrit.wikimedia.org/r/828102

gerritbot added a comment.Sep 6 2022, 4:54 PM

Change 830200 merged by Andrew Bogott:

[operations/dns@master] Move dumps from labstore1006 to clouddumps1001

https://gerrit.wikimedia.org/r/830200

gerritbot added a comment.Sep 6 2022, 7:41 PM

Change 830246 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack network tests: switch to check clouddumps1001 mounts

https://gerrit.wikimedia.org/r/830246

gerritbot added a comment.Sep 6 2022, 7:44 PM

Change 830246 merged by Andrew Bogott:

[operations/puppet@production] openstack network tests: switch to check clouddumps1001 mounts

https://gerrit.wikimedia.org/r/830246

gerritbot added a comment.Sep 26 2022, 1:57 PM

Change 835192 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Dumps: switch to using clouddumps hosts rather than the old labstores.

https://gerrit.wikimedia.org/r/835192

bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Sep 27 2022, 9:29 PM
gerritbot added a comment.Oct 3 2022, 12:51 PM

Change 835192 merged by Andrew Bogott:

[operations/puppet@production] Dumps: switch to using clouddumps hosts rather than the old labstores.

https://gerrit.wikimedia.org/r/835192

gerritbot added a comment.Oct 3 2022, 2:12 PM

Change 828103 merged by Andrew Bogott:

[operations/puppet@production] Dumps: stop mounting the old labstore100x servers on VMs

https://gerrit.wikimedia.org/r/828103

Andrew added a comment.Oct 3 2022, 2:28 PM

This is now done. I'm going to gradually dismantle the old dumps servers but will probably leave their data intact for a few weeks before decom.

Andrew closed subtask T317144: toolforge/paws k8s containers need to know about clouddumps100[12] as Resolved.Oct 3 2022, 2:28 PM
Andrew added a subtask: T319217: decommission labstore100[67].wikimedia.org.Oct 3 2022, 5:07 PM
BTullis subscribed.Oct 5 2022, 4:46 PM

Thanks for all your work on this @Andrew.

I'm going to do a fleet-wide check to see if anything still references the labstore servers in /etc/fstab or if they're still mounted anywhere. I have a feeling that one or two servers might still have the mount active.

Also you might like to update this section when convenient: https://wikitech.wikimedia.org/wiki/Dumps/Dump_servers#Hardware

ArielGlenn added a comment.Oct 5 2022, 5:01 PM

Note that labstore1006 has some html dumps that didn't make it around to the other boxes, so please don't reimage until I get that sorted. Thanks!

• nskaggs added a comment.Oct 7 2022, 3:26 PM

Given the new machines much larger capacity, I believe any pending requests for more space can now be reconsidered. @ArielGlenn do you know of any pending needs? Shall we discuss on a separate ticket any potential use cases for the extra space?

ArielGlenn added a comment.Oct 10 2022, 8:39 PM
In T309346#8300585, @nskaggs wrote:

Given the new machines much larger capacity, I believe any pending requests for more space can now be reconsidered. @ArielGlenn do you know of any pending needs? Shall we discuss on a separate ticket any potential use cases for the extra space?

There's the Kiwix request to mirror more files; see T57503 which I believe you're aware of. I'm not sure of anything else at the moment.

Andrew added a comment.Oct 11 2022, 8:12 PM

Also you might like to update this section when convenient: https://wikitech.wikimedia.org/wiki/Dumps/Dump_servers#Hardware

  • done
BTullis added a comment.Oct 24 2022, 4:23 PM

The directory /srv/deployment/analytics had incorrect ownership on the new hosts, so our deployment failed.
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery#Deploying_to_clouddumps*_hosts

I did the following manual fix on clouddumps100[1-2]

btullis@clouddumps1001:/srv/deployment/analytics$ ls -ld
drwxr-xr-x 3 root root 4096 Aug 23 01:47 .
btullis@clouddumps1001:/srv/deployment/analytics$ sudo chown analytics-deploy .
btullis@clouddumps1001:/srv/deployment/analytics$ ls -ld
drwxr-xr-x 3 analytics-deploy root 4096 Aug 23 01:47 .
btullis@clouddumps1001:/srv/deployment/analytics$

I will follow up with a patch to configure this in puppet.

gerritbot added a comment.Oct 25 2022, 11:07 PM

Change 849192 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Dumps: remove a bunch of references to labstore1006 and labstore1007

https://gerrit.wikimedia.org/r/849192

gerritbot added a comment.Oct 25 2022, 11:07 PM

Change 849193 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] rsync-via-primary.sh: replace labstore with clouddumps

https://gerrit.wikimedia.org/r/849193

gerritbot added a comment.Oct 26 2022, 6:19 PM

Change 849193 merged by Andrew Bogott:

[operations/puppet@production] rsync-via-primary.sh: replace labstore with clouddumps

https://gerrit.wikimedia.org/r/849193

gerritbot added a comment.Oct 26 2022, 6:21 PM

Change 849192 merged by Andrew Bogott:

[operations/puppet@production] Dumps: remove a bunch of references to labstore1006 and labstore1007

https://gerrit.wikimedia.org/r/849192

Andrew closed this task as Resolved.Oct 27 2022, 10:37 PM
Jclark-ctr closed subtask T319217: decommission labstore100[67].wikimedia.org as Resolved.Nov 1 2022, 4:28 PM
Andrew reopened subtask T319217: decommission labstore100[67].wikimedia.org as Open.Nov 10 2022, 4:00 PM
Andrew closed subtask T319217: decommission labstore100[67].wikimedia.org as Resolved.Nov 15 2022, 11:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits