Page MenuHomePhabricator
Create Task
Maniphest T295584

eqiad: 2 VMs for cloudbackup-dev
Closed, ResolvedPublic
Actions

Description

Cloud VPS Project Tested: ---
Site/Location: EQIAD
Number of systems: 2
Service: cloudbackup-dev
Networking Requirements: internal
Processor Requirements: 2
Memory: 4G
Disks: 5G for base operating system 20G in a separate drive for LVM to store cinder backups
Other Requirements:

rationale

We're developing a new backup solution in Cloud VPS for cinder volumes, that will unblock migration of NFS servers into the cloud realm (see parent task).

We have several other cloudbackup servers, including 2 in codfw:

  • cloudbackup2001.codfw.wmnet (200TB storage)
  • cloudbackup2002.codfw.wmnet (200TB storage)

These 2 new VMs (cloudbackupXXXX-dev.eqiad.wmnet) are meant to serve as the mirror of the cloudbackup servers on codfw.

In other words:

  • the openstack deployment eqiad1 will use cloudbackup servers in codfw to store backup data <-- not related to this ticket
  • the openstack deployment codfw1dev will use cloudbackup-dev servers in eqiad to store fake backup data (we only need like 20G storage to tests the system) <-- this ticket

Details

SubjectRepoBranchLines +/-
cloud: codfw1dev: hiera update for new backup serversoperations/puppetproduction+26 -6
cloudbackup1002-dev: mark as spare for nowoperations/puppetproduction+10 -1
cloud: cinder-backups: use main ceph cinder keyringoperations/puppetproduction+1 -9
cloud: introduce cloudbackup1002-dev.eqiad.wmnetoperations/puppetproduction+32 -0
cloud: cinder-backups: fix configuration valuesoperations/puppetproduction+30 -23
cloud: cinder-backup: refresh hiera defaults for eqiad1operations/puppetproduction+6 -1
cloud: cinder-backups: refresh node referencesoperations/puppetproduction+19 -5
cloudbackup: fix disk allocationoperations/puppetproduction+1 -1
cloudbackup: require the lvm packageoperations/puppetproduction+2 -6
cloudbackup: specify administrative contactoperations/puppetproduction+1 -0
cloudbackup: introduce base profilesoperations/puppetproduction+3 -0
cloudbackup: introduce ceph rbd configurationoperations/puppetproduction+47 -0
hiera: cloudbackup1001-dev: relocate ceph auth configlabs/privatemaster+9 -4
hiera: cloud: fix ceph hiera key name for cinder-backups @ codfw1devlabs/privatemaster+1 -1
hiera: cloud: add more secrets for cinder-backups @ codfw1devlabs/privatemaster+4 -0
hiera: cloud: update secrets for cinder-backups @ codfw1devlabs/privatemaster+3 -1
hieradata: openstack: codfw1dev backups: add ldap_user_pass secretlabs/privatemaster+6 -0
site.pp: enable proper role for cloudbackup1001-devoperations/puppetproduction+3 -1
cloudbackup1001-dev: update DHCP configoperations/puppetproduction+5 -0
cloudbackup100X-dev: add insetup roleoperations/puppetproduction+5 -0
cloud: introduce role for cloudbackup-devoperations/puppetproduction+29 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero created this task.Nov 12 2021, 12:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2021, 12:02 PM
aborrero added a parent task: T292546: cloud NFS: figure out backups for cinder volumes.Nov 12 2021, 12:07 PM
aborrero updated the task description. (Show Details)Nov 12 2021, 12:10 PM
aborrero updated the task description. (Show Details)Nov 12 2021, 12:14 PM
aborrero updated the task description. (Show Details)Nov 12 2021, 12:41 PM
gerritbot added a comment.Nov 12 2021, 12:50 PM

Change 738376 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: introduce role for cloudbackup-dev

https://gerrit.wikimedia.org/r/738376

gerritbot added a project: Patch-For-Review.Nov 12 2021, 12:50 PM
aborrero mentioned this in T295592: deploy cinder-backups for eqiad1.Nov 12 2021, 12:58 PM
Dzahn added subscribers: MoritzMuehlenhoff, Dzahn.Nov 15 2021, 9:54 PM

sounds reasonable to me.

As @MoritzMuehlenhoff pointed out on previous requests we should not go under a certain size for the OS partition. Assuming you don't mind if it's 10G or more instead of 5.

Second disk can be added and mounted after VMs have been created (but caveat: this will likely stop the VMs from booting until we login via console and manually adjust renumbered devices in /etc/fstab).

aborrero added a comment.Nov 16 2021, 9:53 AM

OK! We don't really care about the OS drive size. What's important here is the extra drive for LVM, which should have at least 20G.

You create the VMs or I do? I never did it before, but the docs seem nice.

MoritzMuehlenhoff added a comment.Nov 16 2021, 10:26 AM
In T295584#7505985, @aborrero wrote:

OK! We don't really care about the OS drive size. What's important here is the extra drive for LVM, which should have at least 20G.

How long do you expect those to be around? Is this a short test or will they be around continously? If the latter, please use 20G for the OS partition, but regardless use at least 10G. 5G causes all kinds of needless disk space alerts (e.g. when new kernel updates are installed).

You create the VMs or I do? I never did it before, but the docs seem nice.

It's self-managed by anyone in SRE, but if you run into any issues, just ping the I channel on IRC :-)

The VM gets created with the sre.ganeti.makevm cookbook and you can add the second disk following https://wikitech.wikimedia.org/wiki/Ganeti#Adding_a_disk

aborrero claimed this task.Nov 16 2021, 11:20 AM
aborrero triaged this task as Medium priority.
aborrero added a project: cloud-services-team (Kanban).
In T295584#7506073, @MoritzMuehlenhoff wrote:
In T295584#7505985, @aborrero wrote:

OK! We don't really care about the OS drive size. What's important here is the extra drive for LVM, which should have at least 20G.

How long do you expect those to be around? Is this a short test or will they be around continously? If the latter, please use 20G for the OS partition, but regardless use at least 10G. 5G causes all kinds of needless disk space alerts (e.g. when new kernel updates are installed).

Ok, makes sense.

We expect the VMs to be around continuously for as long as we use the same setup in eqiad1.

You create the VMs or I do? I never did it before, but the docs seem nice.

It's self-managed by anyone in SRE, but if you run into any issues, just ping the I channel on IRC :-)

The VM gets created with the sre.ganeti.makevm cookbook and you can add the second disk following https://wikitech.wikimedia.org/wiki/Ganeti#Adding_a_disk

Thanks! Will do soon.

gerritbot added a comment.Nov 16 2021, 11:47 AM

Change 738376 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: introduce role for cloudbackup-dev

https://gerrit.wikimedia.org/r/738376

Maintenance_bot removed a project: Patch-For-Review.Nov 16 2021, 12:10 PM
gerritbot added a comment.Nov 17 2021, 12:59 PM

Change 739535 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup100X-dev: add insetup role

https://gerrit.wikimedia.org/r/739535

gerritbot added a project: Patch-For-Review.Nov 17 2021, 12:59 PM
gerritbot added a comment.Nov 17 2021, 1:11 PM

Change 739535 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup100X-dev: add insetup role

https://gerrit.wikimedia.org/r/739535

aborrero added a comment.Nov 17 2021, 1:21 PM

Created 1 VM with:

aborrero@cumin1001:~ $ sudo cookbook sre.ganeti.makevm eqiad_B cloudbackup1001-dev --vcpus 2 --memory 4

no errors in the run.

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2021, 2:10 PM
gerritbot added a comment.Nov 17 2021, 5:38 PM

Change 739581 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup1001-dev: update DHCP config

https://gerrit.wikimedia.org/r/739581

gerritbot added a project: Patch-For-Review.Nov 17 2021, 5:38 PM

Change 739581 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup1001-dev: update DHCP config

https://gerrit.wikimedia.org/r/739581

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2021, 6:10 PM
gerritbot added a comment.Nov 17 2021, 6:15 PM

Change 739591 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] site.pp: enable proper role for cloudbackup1001-dev

https://gerrit.wikimedia.org/r/739591

gerritbot added a project: Patch-For-Review.Nov 17 2021, 6:15 PM

Change 739591 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] site.pp: enable proper role for cloudbackup1001-dev

https://gerrit.wikimedia.org/r/739591

gerritbot added a comment.Nov 17 2021, 6:24 PM

Change 739595 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hieradata: openstack: codfw1dev backups: add ldap_user_pass secret

https://gerrit.wikimedia.org/r/739595

gerritbot added a comment.Nov 17 2021, 6:25 PM

Change 739595 merged by Arturo Borrero Gonzalez:

[labs/private@master] hieradata: openstack: codfw1dev backups: add ldap_user_pass secret

https://gerrit.wikimedia.org/r/739595

aborrero mentioned this in rLPRI9cc045d44513: hieradata: openstack: codfw1dev backups: add ldap_user_pass secret.Nov 17 2021, 6:26 PM
gerritbot added a comment.Nov 17 2021, 6:29 PM

Change 739599 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: codfw1dev: hiera update for new backup servers

https://gerrit.wikimedia.org/r/739599

gerritbot added a comment.Nov 18 2021, 12:00 PM

Change 739777 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: cloud: update secrets for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739777

gerritbot added a comment.Nov 18 2021, 12:00 PM

Change 739777 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: cloud: update secrets for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739777

aborrero mentioned this in rLPRI4d11f9911e0d: hiera: cloud: update secrets for cinder-backups @ codfw1dev.Nov 18 2021, 12:03 PM
gerritbot added a comment.Nov 18 2021, 12:06 PM

Change 739779 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: cloud: add more secrets for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739779

gerritbot added a comment.Nov 18 2021, 12:08 PM

Change 739779 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: cloud: add more secrets for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739779

aborrero mentioned this in rLPRI3cc9aa1eac8d: hiera: cloud: add more secrets for cinder-backups @ codfw1dev.Nov 18 2021, 12:09 PM
gerritbot added a comment.Nov 18 2021, 12:13 PM

Change 739782 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: cloud: fix ceph hiera key name for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739782

gerritbot added a comment.Nov 18 2021, 12:14 PM

Change 739782 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: cloud: fix ceph hiera key name for cinder-backups @ codfw1dev

https://gerrit.wikimedia.org/r/739782

aborrero mentioned this in rLPRIb73dca7bdb40: hiera: cloud: fix ceph hiera key name for cinder-backups @ codfw1dev.Nov 18 2021, 12:15 PM
aborrero mentioned this in rLPRIcde5d9d138ff: hiera: cloud: fix ceph hiera key entry for cinder-backups @ codfw1dev.Nov 18 2021, 12:19 PM
gerritbot added a comment.Nov 18 2021, 12:56 PM

Change 739792 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] hiera: cloudbackup1001-dev: relocate ceph auth config

https://gerrit.wikimedia.org/r/739792

gerritbot added a comment.Nov 18 2021, 12:57 PM

Change 739792 merged by Arturo Borrero Gonzalez:

[labs/private@master] hiera: cloudbackup1001-dev: relocate ceph auth config

https://gerrit.wikimedia.org/r/739792

aborrero mentioned this in rLPRI7320c553a324: hiera: cloudbackup1001-dev: relocate ceph auth config.Nov 18 2021, 12:59 PM
gerritbot added a comment.Nov 18 2021, 4:16 PM

Change 739599 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: codfw1dev: hiera update for new backup servers

https://gerrit.wikimedia.org/r/739599

gerritbot added a comment.Nov 18 2021, 4:59 PM

Change 739870 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup: introduce ceph rbd configuration

https://gerrit.wikimedia.org/r/739870

gerritbot added a comment.Nov 18 2021, 5:45 PM

Change 739870 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup: introduce ceph rbd configuration

https://gerrit.wikimedia.org/r/739870

gerritbot added a comment.Nov 18 2021, 5:50 PM

Change 739892 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup: introduce base profiles

https://gerrit.wikimedia.org/r/739892

gerritbot added a comment.Nov 18 2021, 5:57 PM

Change 739892 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup: introduce base profiles

https://gerrit.wikimedia.org/r/739892

gerritbot added a comment.Nov 18 2021, 6:00 PM

Change 739895 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup: specify administrative contact

https://gerrit.wikimedia.org/r/739895

gerritbot added a comment.Nov 18 2021, 6:13 PM

Change 739895 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup: specify administrative contact

https://gerrit.wikimedia.org/r/739895

gerritbot added a comment.Nov 18 2021, 6:18 PM

Change 739903 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup: require the lvm package

https://gerrit.wikimedia.org/r/739903

gerritbot added a comment.Nov 18 2021, 6:22 PM

Change 739903 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup: require the lvm package

https://gerrit.wikimedia.org/r/739903

gerritbot added a comment.Nov 18 2021, 6:26 PM

Change 739905 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudbackup: fix disk allocation

https://gerrit.wikimedia.org/r/739905

gerritbot added a comment.Nov 18 2021, 6:28 PM

Change 739905 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudbackup: fix disk allocation

https://gerrit.wikimedia.org/r/739905

gerritbot added a comment.Nov 19 2021, 10:14 AM

Change 740107 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: cinder-backups: refresh node references

https://gerrit.wikimedia.org/r/740107

gerritbot added a comment.Nov 19 2021, 10:35 AM

Change 740107 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: cinder-backups: refresh node references

https://gerrit.wikimedia.org/r/740107

gerritbot added a comment.Nov 19 2021, 11:45 AM

Change 740131 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: cinder-backup: refresh hiera config for eqiad1

https://gerrit.wikimedia.org/r/740131

gerritbot added a comment.Nov 19 2021, 11:54 AM

Change 740131 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: cinder-backup: refresh hiera defaults for eqiad1

https://gerrit.wikimedia.org/r/740131

gerritbot added a comment.Nov 22 2021, 10:16 AM

Change 740535 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: cinder-backups: fix configuration values

https://gerrit.wikimedia.org/r/740535

gerritbot added a comment.Nov 22 2021, 10:20 AM

Change 740535 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: cinder-backups: fix configuration values

https://gerrit.wikimedia.org/r/740535

gerritbot added a comment.Nov 22 2021, 11:26 AM

Change 740551 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: cinder-backups: use main ceph cinder keyring

https://gerrit.wikimedia.org/r/740551

gerritbot added a comment.Nov 23 2021, 12:46 PM

Change 740820 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: introduce cloudbackup1002-dev.eqiad.wmnet

https://gerrit.wikimedia.org/r/740820

gerritbot added a comment.Nov 23 2021, 12:54 PM

Change 740820 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: introduce cloudbackup1002-dev.eqiad.wmnet

https://gerrit.wikimedia.org/r/740820

aborrero added a comment.Nov 23 2021, 1:06 PM

created second VM:

aborrero@cumin1001:~ $ sudo cookbook sre.ganeti.makevm eqiad_B cloudbackup1002-dev --vcpus 2 --memory 4
[..]
aborrero@ganeti1009:~ $ sudo gnt-instance modify --disk add:size=20g cloudbackup1002-dev
[..]
aborrero@ganeti1009:~ $ sudo gnt-instance startup cloudbackup1002-dev
Waiting for job 1461518 for cloudbackup1002-dev.eqiad.wmnet ...

etc.

aborrero closed this task as Resolved.Nov 24 2021, 10:06 AM
gerritbot added a comment.Nov 29 2021, 5:44 PM

Change 740551 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: cinder-backups: use main ceph cinder keyring

Reason:

a different patch was merged, see ed5658a51946148376ec19a6474d5e972bb34167

https://gerrit.wikimedia.org/r/740551

gerritbot added a comment.Dec 9 2021, 7:38 PM

Change 745588 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudbackup1002-dev: mark as spare for now

https://gerrit.wikimedia.org/r/745588

gerritbot added a comment.Dec 9 2021, 7:39 PM

Change 745588 merged by Andrew Bogott:

[operations/puppet@production] cloudbackup1002-dev: mark as spare for now

https://gerrit.wikimedia.org/r/745588

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL