Page MenuHomePhabricator
Create Task
Maniphest T345610

cloudrabbit: connect them via cloudsw and cloud-private
Closed, ResolvedPublic
Actions

Description

Now that the work in T341060: openstack eqiad1: introduce cloud-private and cloudlb enabled the cloud-private in eqiad1, there is no longer any reason for cloudrabbit hosts to have a .wikimedia.org domain with a public IPv4 address.

As of today, this affects:

hostoriginal racknew rackstatus
cloudrabbit1001B2TBDready
cloudrabbit1002C4E4ready
cloudrabbit1003D4F4done

We may consider:

  • reracking each server into a cloudsw-enabled rack
  • reimage with a new IP a new domain .eqiad.wmnet
  • enable cloud-private on them
  • enable connectivity with rabbitmq via cloud-private

Details

SubjectRepoBranchLines +/-
wikimediacloud.org: Move Rabbit traffic back to all nodesoperations/dnsmaster+2 -2
Move cloudrabbit1001/2 to private vlanoperations/puppetproduction+8 -24
wikimediacloud.org: Move RabbitMQ traffic to cloudrabbit1003operations/dnsmaster+3 -3
cr-labs: Add temporary term for cloudrabbit hostsoperations/homer/publicmaster+6 -0
site: Add cloudrabbit1003 as insetupoperations/puppetproduction+6 -1
wikimediacloud.org: point rabbitmq03.eqiad1 to cloudrabbit1002operations/dnsmaster+1 -1
hieradata: fix eqiad1 rabbitmq firewall rulesoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero created this task.Sep 5 2023, 10:00 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2023, 10:00 AM
gerritbot added a comment.Sep 20 2023, 11:48 AM

Change 959198 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: fix eqiad1 rabbitmq firewall rules

https://gerrit.wikimedia.org/r/959198

gerritbot added a project: Patch-For-Review.Sep 20 2023, 11:48 AM
gerritbot added a comment.Sep 20 2023, 11:54 AM

Change 959198 merged by Majavah:

[operations/puppet@production] hieradata: fix eqiad1 rabbitmq firewall rules

https://gerrit.wikimedia.org/r/959198

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2023, 12:10 PM
taavi edited parent tasks, added: T341494: cloud @ eqiad: hardware re-racking plan; removed: T314522: Dedicated cloudrabbit nodes in eqiad1.Sep 22 2023, 1:28 PM
cmooney added a comment.Sep 22 2023, 1:34 PM

Thanks for the task! Yeah these actually kicked off the whole project to some extent, so good to move them.

I gather they don't need to go behind the cloudlb, or use a shared VIP? If so hopefully it'd be straightforward to re-provision with the new network setup, and make the Rabbit MQ service available on cloud-private.

We should move any other hosts that need them to the cloud-specific racks before proceeding.

taavi added a comment.Sep 22 2023, 2:30 PM

I'll need to double-check but I think it's enough that everything else can talk to the Rabbit ports on the cloud-private addresses. The remaining cloudcontrols and cloudvirt-wdqs hosts will indeed need to be moved first.

taavi claimed this task.Nov 10 2023, 2:01 PM
taavi edited projects, added Cloud-VPS; removed User-aborrero.
taavi added a project: ops-eqiad.Dec 15 2023, 5:19 PM

This can move forward now, although due to the nature of Rabbit this needs to be coordinated to avoid unnecessary downtime. It's fine if this goes to January, I'd much prefer that to a non-coordinated move next week. cloudrabbit1001 is the current cluster leader so it should be moved last.

These hosts will need a single connection with the rack-specific cloud-hosts VLAN as primary and cloud-private trunked. They need to be spread out so there's no more than one node per rack.

Maintenance_bot added a project: SRE.Dec 15 2023, 5:29 PM
ayounsi added a comment.Dec 18 2023, 8:01 AM

Nice ! So next step here is to decom the current ones and then sync up with DCops to move them to the proper racks. From there we can re-provision them with the proper network settings.

gerritbot added a comment.Jan 9 2024, 3:43 PM

Change 989196 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] wikimediacloud.org: point rabbitmq03.eqiad1 to cloudrabbit1002

https://gerrit.wikimedia.org/r/989196

gerritbot added a project: Patch-For-Review.Jan 9 2024, 3:43 PM
gerritbot added a comment.Jan 9 2024, 3:45 PM

Change 989196 merged by Majavah:

[operations/dns@master] wikimediacloud.org: point rabbitmq03.eqiad1 to cloudrabbit1002

https://gerrit.wikimedia.org/r/989196

ops-monitoring-bot added a comment.Jan 9 2024, 4:10 PM

cookbooks.sre.hosts.decommission executed by taavi@cumin1002 for hosts: cloudrabbit1003.wikimedia.org

  • cloudrabbit1003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
taavi added a comment.Jan 9 2024, 4:11 PM
In T345610#9409823, @taavi wrote:

This can move forward now, although due to the nature of Rabbit this needs to be coordinated to avoid unnecessary downtime. It's fine if this goes to January, I'd much prefer that to a non-coordinated move next week. cloudrabbit1001 is the current cluster leader so it should be moved last.

These hosts will need a single connection with the rack-specific cloud-hosts VLAN as primary and cloud-private trunked. They need to be spread out so there's no more than one node per rack.

cloudrabbit1003 is now ready to be moved.

taavi removed taavi as the assignee of this task.Jan 9 2024, 4:24 PM
taavi added a project: DC-Ops.
Maintenance_bot removed a project: Patch-For-Review.Jan 9 2024, 4:30 PM
taavi mentioned this in T341060: openstack eqiad1: introduce cloud-private and cloudlb.Jan 19 2024, 1:00 PM
taavi updated the task description. (Show Details)Jan 22 2024, 1:08 PM
gerritbot added a comment.Jan 22 2024, 8:38 PM

Change 992234 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] site: Add cloudrabbit1003 as insetup

https://gerrit.wikimedia.org/r/992234

gerritbot added a project: Patch-For-Review.Jan 22 2024, 8:38 PM
gerritbot added a comment.Jan 22 2024, 9:04 PM

Change 992234 merged by Majavah:

[operations/puppet@production] site: Add cloudrabbit1003 as insetup

https://gerrit.wikimedia.org/r/992234

VRiley-WMF subscribed.Jan 22 2024, 9:08 PM

Physically moved the server to F4, U18. Port 4 CableID 2M-20220019

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 9:30 PM
gerritbot added a comment.Jan 22 2024, 10:08 PM

Change 992245 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-labs: Add temporary term for cloudrabbit hosts

https://gerrit.wikimedia.org/r/992245

gerritbot added a project: Patch-For-Review.Jan 22 2024, 10:08 PM
gerritbot added a comment.Jan 23 2024, 8:28 AM

Change 992245 merged by jenkins-bot:

[operations/homer/public@master] cr-labs: Add temporary term for cloudrabbit hosts

https://gerrit.wikimedia.org/r/992245

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2024, 8:30 AM
taavi updated the task description. (Show Details)Jan 23 2024, 8:44 AM
gerritbot added a comment.Jan 25 2024, 10:22 AM

Change 992884 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] wikimediacloud.org: Move RabbitMQ traffic to cloudrabbit1003

https://gerrit.wikimedia.org/r/992884

gerritbot added a project: Patch-For-Review.Jan 25 2024, 10:22 AM
gerritbot added a comment.Jan 25 2024, 4:36 PM

Change 992884 merged by Majavah:

[operations/dns@master] wikimediacloud.org: Move RabbitMQ traffic to cloudrabbit1003

https://gerrit.wikimedia.org/r/992884

Stashbot added a comment.Jan 25 2024, 4:48 PM

Mentioned in SAL (#wikimedia-cloud) [2024-01-25T16:48:36Z] <andrewbogott> taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of T345610

ops-monitoring-bot added a comment.Jan 25 2024, 5:04 PM

cookbooks.sre.hosts.decommission executed by taavi@cumin1002 for hosts: cloudrabbit[1001-1002].wikimedia.org

  • cloudrabbit1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • cloudrabbit1002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
taavi updated the task description. (Show Details)Jan 25 2024, 5:06 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 25 2024, 5:30 PM
VRiley-WMF updated the task description. (Show Details)Jan 25 2024, 7:28 PM
VRiley-WMF updated the task description. (Show Details)

cloudrabbit1002 is now in

E4
U17
CableID 2M-20220016
Port 3

VRiley-WMF added a comment.Jan 25 2024, 7:56 PM

cloudrabbit1001 is now in

C8
U19
CableID 5336
Port 21

gerritbot added a comment.Jan 25 2024, 8:06 PM

Change 993026 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Move cloudrabbit1001/2 to private vlan

https://gerrit.wikimedia.org/r/993026

gerritbot added a project: Patch-For-Review.Jan 25 2024, 8:06 PM

Change 993026 merged by Majavah:

[operations/puppet@production] Move cloudrabbit1001/2 to private vlan

https://gerrit.wikimedia.org/r/993026

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2024, 8:30 PM
Stashbot added a comment.Jan 26 2024, 9:01 AM

Mentioned in SAL (#wikimedia-cloud) [2024-01-26T09:01:15Z] <taavi> joining cloudrabbit1001/2 to the cluster on 1003 T345610

taavi closed this task as Resolved.Jan 26 2024, 9:04 AM
taavi claimed this task.
gerritbot added a comment.Jan 26 2024, 9:07 AM

Change 993062 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] wikimediacloud.org: Move Rabbit traffic back to all nodes

https://gerrit.wikimedia.org/r/993062

gerritbot added a project: Patch-For-Review.Jan 26 2024, 9:07 AM
gerritbot added a comment.Jan 30 2024, 9:42 AM

Change 993062 merged by Majavah:

[operations/dns@master] wikimediacloud.org: Move Rabbit traffic back to all nodes

https://gerrit.wikimedia.org/r/993062

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2024, 10:30 AM
gerritbot added a comment.Feb 7 2024, 2:31 PM

Change 998419 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack: rabbitmq: cleanup rabbitmq firewall

https://gerrit.wikimedia.org/r/998419

gerritbot added a project: Patch-For-Review.Feb 7 2024, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL