Page MenuHomePhabricator
Create Task
Maniphest T346947

Move wiki replicas behind cloudlb
Closed, ResolvedPublic
Actions

Description

We have a new load balancer layer cloudlb that can be used to expose services to the WMCS realm from hardware in the production realm without having to waste prod-realm public IP addresses. We should investigate if we can move the wiki replicas behind that, or with some other mechanism, like having the current dbproxy hosts advertise VIPs to cloud-private.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi created this task.Sep 20 2023, 5:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2023, 5:01 PM
taavi mentioned this in T347148: Determine how to monitor services in cloud-private / cloudlb.Sep 22 2023, 1:05 PM
BTullis mentioned this in T300427: Automate maintain-views replica depooling.Nov 13 2023, 12:29 PM
gerritbot added a comment.Nov 13 2023, 12:46 PM

Change 973769 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-labs: permit cloudlb to wiki replicas

https://gerrit.wikimedia.org/r/973769

gerritbot added a project: Patch-For-Review.Nov 13 2023, 12:46 PM
gerritbot added a comment.Nov 13 2023, 1:42 PM

Change 973761 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Add wiki replicas to cloudlb

https://gerrit.wikimedia.org/r/973761

gerritbot added a comment.Nov 13 2023, 1:42 PM

Change 973777 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs: wikireplicas: allow access from cloudlb

https://gerrit.wikimedia.org/r/973777

taavi added a parent task: T300427: Automate maintain-views replica depooling.Nov 13 2023, 2:08 PM
taavi claimed this task.Nov 15 2023, 9:50 AM
taavi closed subtask T351087: Migrate cloudlb hosts to nftables as Resolved.
taavi closed subtask T351094: nftables ignores drange filter for IPv6 if drange only has IPv4 addresses as Resolved.Nov 15 2023, 9:53 AM
gerritbot added a comment.Nov 15 2023, 1:44 PM

Change 973769 merged by jenkins-bot:

[operations/homer/public@master] cr-labs: permit cloudlb to wiki replicas

https://gerrit.wikimedia.org/r/973769

gerritbot added a comment.Nov 15 2023, 1:54 PM

Change 974534 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Remove includes for subnets from cloud-support1-a-eqiad

https://gerrit.wikimedia.org/r/974534

gerritbot added a comment.Nov 15 2023, 2:19 PM

Change 974534 merged by Cathal Mooney:

[operations/dns@master] Remove includes for subnets from cloud-support1-a-eqiad

https://gerrit.wikimedia.org/r/974534

taavi moved this task from Backlog to Wiki replicas on the Data-Services board.Nov 20 2023, 12:36 PM
gerritbot added a comment.Nov 22 2023, 11:43 AM

Change 973777 merged by Majavah:

[operations/puppet@production] P:wmcs: wikireplicas: allow access from cloudlb

https://gerrit.wikimedia.org/r/973777

gerritbot added a comment.Nov 22 2023, 12:01 PM

Change 976688 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: update wiki replica DNS to cloudlb

https://gerrit.wikimedia.org/r/976688

gerritbot added a comment.Nov 22 2023, 12:11 PM

Change 973761 merged by Majavah:

[operations/puppet@production] Add wiki replicas to cloudlb

https://gerrit.wikimedia.org/r/973761

gerritbot added a comment.Nov 22 2023, 1:44 PM

Change 976735 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:etcd: generate wiki replica pool accounts

https://gerrit.wikimedia.org/r/976735

gerritbot added a comment.Nov 23 2023, 1:23 PM

Change 977081 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/alerts@master] team-wmcs: Adapt cloudlb alerts for wiki replicas

https://gerrit.wikimedia.org/r/977081

gerritbot added a comment.Nov 23 2023, 3:08 PM

Change 977081 merged by jenkins-bot:

[operations/alerts@master] team-wmcs: Adapt cloudlb alerts for wiki replicas

https://gerrit.wikimedia.org/r/977081

gerritbot added a comment.Nov 28 2023, 2:15 PM

Change 976688 merged by Majavah:

[operations/puppet@production] openstack: update wiki replica DNS to cloudlb

https://gerrit.wikimedia.org/r/976688

Stashbot added a comment.Nov 28 2023, 2:18 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-28T14:18:07Z] <taavi> moving wiki replica DNS to use cloudlbs instead of the old proxy VMs T346947

gerritbot added a comment.Nov 29 2023, 1:36 PM

Change 978539 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: unconfigure wiki replica LVS services

https://gerrit.wikimedia.org/r/978539

gerritbot added a comment.Nov 30 2023, 10:53 AM

Change 979045 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] cloudlb: wikireplicas: fix timeouts

https://gerrit.wikimedia.org/r/979045

gerritbot added a comment.Nov 30 2023, 10:59 AM

Change 979045 merged by Majavah:

[operations/puppet@production] cloudlb: wikireplicas: fix timeouts

https://gerrit.wikimedia.org/r/979045

taavi mentioned this in T352211: [bug] query/77794: "This query was stopped".Nov 30 2023, 12:41 PM
Stashbot added a comment.Jan 8 2024, 12:57 PM

Mentioned in SAL (#wikimedia-cloud) [2024-01-08T12:57:11Z] <taavi> shut off old wiki replica proxies in cloud vps T346947

gerritbot added a comment.Jan 8 2024, 1:00 PM

Change 988483 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: remove wikireplica service catalog entries

https://gerrit.wikimedia.org/r/988483

gerritbot added a comment.Jan 8 2024, 1:02 PM

Change 988484 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/dns@master] wmnet: remove aliases for dbproxy1018/9

https://gerrit.wikimedia.org/r/988484

gerritbot added a comment.Jan 8 2024, 2:49 PM

Change 988484 merged by Majavah:

[operations/dns@master] wmnet: remove aliases for dbproxy1018/9

https://gerrit.wikimedia.org/r/988484

gerritbot added a comment.Jan 8 2024, 4:02 PM

Change 978539 merged by Majavah:

[operations/puppet@production] hieradata: unconfigure wiki replica LVS services

https://gerrit.wikimedia.org/r/978539

Stashbot added a comment.Jan 8 2024, 4:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:09:13Z] <taavi> restart pybal on lvs1020 - T346947

Stashbot added a comment.Jan 8 2024, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:15:27Z] <taavi> restart pybal on lvs1018 - T346947

Stashbot added a comment.Jan 8 2024, 4:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:20:54Z] <taavi> lvs1020: sudo ipvsadm --delete-service --tcp-service 208.80.154.242:3311 (and all the way to :3318) - T346947

Stashbot added a comment.Jan 8 2024, 4:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:21:28Z] <taavi> lvs1020: sudo ipvsadm --delete-service --tcp-service 208.80.154.243:3311 (and all the way to :3318) - T346947

Stashbot added a comment.Jan 8 2024, 4:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:23:17Z] <taavi> lvs1018: sudo ipvsadm --delete-service --tcp-service 208.80.154.242:3311 (and all the way to :3318) - T346947

Stashbot added a comment.Jan 8 2024, 4:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T16:24:18Z] <taavi> lvs1018: sudo ipvsadm --delete-service --tcp-service 208.80.154.243:3311 (and all the way to :3318) - T346947

gerritbot added a comment.Jan 8 2024, 4:28 PM

Change 988483 merged by Majavah:

[operations/puppet@production] hieradata: remove wikireplica service catalog entries

https://gerrit.wikimedia.org/r/988483

gerritbot added a comment.Jan 8 2024, 4:29 PM

Change 988670 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] O:mariadb::proxy: remove LVS realserver profile

https://gerrit.wikimedia.org/r/988670

gerritbot added a comment.Jan 8 2024, 4:32 PM

Change 988670 merged by Majavah:

[operations/puppet@production] O:mariadb::proxy::replicas: remove LVS realserver profile

https://gerrit.wikimedia.org/r/988670

gerritbot added a comment.Jan 8 2024, 6:03 PM

Change 988681 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Move dbproxy1018/9 to insetup

https://gerrit.wikimedia.org/r/988681

gerritbot added a comment.Jan 9 2024, 8:54 AM

Change 989087 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] conftool-data: Remove wiki replica dbproxies

https://gerrit.wikimedia.org/r/989087

gerritbot added a comment.Jan 9 2024, 8:58 AM

Change 989088 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] mariadb: remove grants and firewall rules for dbproxy1018/9

https://gerrit.wikimedia.org/r/989088

gerritbot added a comment.Jan 9 2024, 10:47 AM

Change 989087 merged by Majavah:

[operations/puppet@production] conftool-data: Remove wiki replica dbproxies

https://gerrit.wikimedia.org/r/989087

taavi mentioned this in T322658: Improve LVS config for wikireplicas (dbproxy1018/dbproxy1019).Jan 9 2024, 11:08 AM
Stashbot added a comment.Jan 9 2024, 12:01 PM

Mentioned in SAL (#wikimedia-cloud) [2024-01-09T12:01:07Z] <taavi> delete old neutron ports used for old wiki replica VIPs T346947

gerritbot added a comment.Jan 10 2024, 2:49 PM

Change 988681 merged by Majavah:

[operations/puppet@production] Move dbproxy1018/9 to insetup

https://gerrit.wikimedia.org/r/988681

ops-monitoring-bot added a comment.Jan 10 2024, 3:06 PM

cookbooks.sre.hosts.decommission executed by taavi@cumin1002 for hosts: dbproxy[1018-1019].eqiad.wmnet

  • dbproxy1018.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • dbproxy1019.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jan 10 2024, 3:37 PM

Change 989541 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] site: remove dbproxy1018/9

https://gerrit.wikimedia.org/r/989541

gerritbot added a comment.Jan 10 2024, 3:39 PM

Change 989541 merged by Majavah:

[operations/puppet@production] site: remove dbproxy1018/9

https://gerrit.wikimedia.org/r/989541

gerritbot added a comment.Jan 10 2024, 3:41 PM

Change 989088 merged by Majavah:

[operations/puppet@production] mariadb: remove grants and firewall rules for dbproxy1018/9

https://gerrit.wikimedia.org/r/989088

Stashbot added a comment.Jan 16 2024, 9:41 AM

Mentioned in SAL (#wikimedia-cloud) [2024-01-16T09:41:22Z] <taavi> drop dbproxy1018/9 grants from all clouddb hosts T346947

gerritbot added a comment.Jan 16 2024, 9:45 AM

Change 990957 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software@master] report_users: drop dbproxy1018/9

https://gerrit.wikimedia.org/r/990957

taavi added a parent task: T355115: Remove cloud-support1-c-eqiad VLAN.Jan 16 2024, 9:49 AM
gerritbot added a comment.Jan 16 2024, 1:36 PM

Change 990957 merged by jenkins-bot:

[operations/software@master] report_users: drop dbproxy1018/9

https://gerrit.wikimedia.org/r/990957

taavi closed this task as Resolved.Jan 17 2024, 1:46 PM

This is all done, I think.

aborrero mentioned this in T357543: clouddb: evaluate moving them into cloud-private.Feb 23 2024, 12:16 PM
Novem_Linguae subscribed.Mar 8 2024, 8:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL