Page MenuHomePhabricator
Create Task
Maniphest T332153

cloudlb PoC: prepare backends
Closed, ResolvedPublic
Actions

Description

As the cloudlb PoC keeps evolving, we will eventually need to connect some backends servers to it, more specifically cloudcontrols servers.

This task is to track such work.

Details

SubjectRepoBranchLines +/-
wikimediacloud.org: move openstack.codfw1dev.wikimediacloud.org to new VIPoperations/dnsmaster+5 -1
wikimediacloud.org: adjust openstack.codfw1dev FQDNoperations/dnsmaster+3 -1
cloudcontrol: allow services to be contacted by all cloudlb HAproxyoperations/puppetproduction+3 -0
cloudcontrol2001-dev: introduce cloudlb supportoperations/puppetproduction+37 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336236 cloudcontrol2001-dev: make it a cloudlb backend
 ResolvedJhancock.wmT336564 cloudcontrol2005-dev: make it a cloudlb backend
 ResolvedaborreroT336723 cloudlb vs. password_safelist
 ResolvedaborreroT336808 Inconsistent connectivity between cloudservices200[45]-dev and codfw1dev cloudcontrols
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT338125 cloudvirt: connect them to cloud-private
 ResolvedaborreroT338314 cloudnet: connect them to cloud-private
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedaborreroT338433 openstack: figure out how to change /etc/resolv.conf in all VMs
 ResolvedaborreroT338778 cloudservices2004-dev: reimage into new network setup
 InvalidNoneT339869 codfw1dev: LDAP database content seems to have lost years of content
 ResolvedaborreroT338779 cloudservices2005-dev: reimage into new network setup
 ResolvedaborreroT340047 LDAP problems following cloudservices2005-dev reimage
 OpenNoneT340127 systemctl restart keystone doesn't actually restart keystone sometimes
 ResolvedaborreroT340488 codfw1dev: OpenStack services can only sort of talk to memacached on cloudcontrols
 InvalidaborreroT338938 codfw1dev: finish auth DNS server transition
 ResolvedaborreroT339905 codfw1dev: LDAP setup needs a refresh so TLS works
 ResolvedfgiunchediT341966 prometheus in codfw can't reach cloudservices' pdns to fetch metrics
 ResolvedaborreroT342621 eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP
 InvalidaborreroT346092 cloudcontrols can't reach ns-recursor.openstack.eqiad1.wikimediacloud.org
 OpenaborreroT346426 Some VPS instances still using ns-recursor0
 ResolvedcmooneyT347858 Manage WMCS codfw public service VIP DNS from Netbox
 ResolvedJhancock.wmT337828 cloudcontrol2004-dev: make it a cloudlb backend
 ResolvedaborreroT338132 cloudcontrol: review connectivity with backup system
 ResolvedAndrewT340791 galera sync failures on cloudcontrol2005-dev
 ResolvedaborreroT340881 proxy api crashing in codfw1dev

Event Timeline

aborrero created this task.Mar 15 2023, 12:25 PM
gerritbot added a comment.Mar 15 2023, 12:57 PM

Change 899614 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudcontrol: introduce cloudlb support

https://gerrit.wikimedia.org/r/899614

gerritbot added a project: Patch-For-Review.Mar 15 2023, 12:57 PM
aborrero added a comment.Mar 29 2023, 9:17 AM

I just noticed that all 3 cloudcontrols are racked in a way that would make it impossible to partake in the cloud-private setup. We need to re-rack them. Will create a subtask for that.

aborrero added a comment.Mar 29 2023, 9:28 AM
In T332153#8737552, @aborrero wrote:

I just noticed that all 3 cloudcontrols are racked in a way that would make it impossible to partake in the cloud-private setup. We need to re-rack them. Will create a subtask for that.

Not exactly true after all:

So only one server needs re-racking for the initial stage of the PoC.

aborrero triaged this task as Medium priority.Mar 29 2023, 11:07 AM
• nskaggs edited projects, added cloud-services-team (FY2022/2023-Q4); removed cloud-services-team (FY2022/2023-Q3).Apr 28 2023, 1:53 PM
aborrero added a subscriber: Papaul.May 8 2023, 2:46 PM

for cloudcontrol2004-dev we would need to (from https://phabricator.wikimedia.org/T327919#8742373):

aborrero added a subscriber: taavi.May 8 2023, 2:48 PM

In https://gerrit.wikimedia.org/r/c/operations/puppet/+/899614/ @taavi wrote:

I think you temporarily want to be running both load balancers during the migration period.

gerritbot added a comment.May 9 2023, 3:18 PM

Change 917904 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudcontrol: allow services to be contacted by all cloudlb HAproxy

https://gerrit.wikimedia.org/r/917904

gerritbot added a comment.May 9 2023, 3:23 PM

Change 899614 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudcontrol2001-dev: introduce cloudlb support

https://gerrit.wikimedia.org/r/899614

gerritbot added a comment.May 10 2023, 8:41 AM

Change 917904 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudcontrol: allow services to be contacted by all cloudlb HAproxy

https://gerrit.wikimedia.org/r/917904

Maintenance_bot removed a project: Patch-For-Review.May 10 2023, 9:10 AM
aborrero closed subtask T336236: cloudcontrol2001-dev: make it a cloudlb backend as Resolved.May 10 2023, 4:08 PM
gerritbot added a comment.May 10 2023, 4:20 PM

Change 918525 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: move openstack.codfw1dev.wikimediacloud.org to new VIP

https://gerrit.wikimedia.org/r/918525

gerritbot added a project: Patch-For-Review.May 10 2023, 4:20 PM
aborrero changed the status of subtask T336564: cloudcontrol2005-dev: make it a cloudlb backend from Open to In Progress.May 12 2023, 9:19 AM
Andrew closed subtask T336723: cloudlb vs. password_safelist as Resolved.May 16 2023, 7:13 PM
gerritbot added a comment.May 30 2023, 2:17 PM

Change 924526 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: adjust openstack.codfw1dev FQDN

https://gerrit.wikimedia.org/r/924526

gerritbot added a comment.May 30 2023, 2:26 PM

Change 924526 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: adjust openstack.codfw1dev FQDN

https://gerrit.wikimedia.org/r/924526

aborrero closed subtask T336564: cloudcontrol2005-dev: make it a cloudlb backend as Resolved.May 31 2023, 8:57 AM
gerritbot added a comment.Jun 1 2023, 2:43 PM

Change 918525 abandoned by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: move openstack.codfw1dev.wikimediacloud.org to new VIP

Reason:

merged https://gerrit.wikimedia.org/r/c/operations/dns/+/924526

https://gerrit.wikimedia.org/r/918525

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2023, 3:11 PM
aborrero closed subtask T337828: cloudcontrol2004-dev: make it a cloudlb backend as Resolved.Jun 1 2023, 4:59 PM
aborrero closed subtask T336808: Inconsistent connectivity between cloudservices200[45]-dev and codfw1dev cloudcontrols as Resolved.Jun 2 2023, 12:07 PM
aborrero closed subtask T336963: cloudcontrol2001-dev can't reach cloud-vps public IPs as Resolved.Jun 12 2023, 11:13 AM
aborrero closed this task as Resolved.Jun 13 2023, 10:26 AM
aborrero closed subtask T338132: cloudcontrol: review connectivity with backup system as Resolved.Jun 20 2023, 3:48 PM
taavi closed subtask T340881: proxy api crashing in codfw1dev as Resolved.Jul 3 2023, 4:11 PM
fnegri moved this task from Backlog to Done on the cloud-services-team (FY2022/2023-Q4) board.Jul 27 2023, 3:18 PM
Andrew closed subtask T340791: galera sync failures on cloudcontrol2005-dev as Resolved.Aug 7 2023, 6:19 PM
aborrero renamed this task from cloudlb: prepare backends to cloudlb PoC: prepare backends.Sep 4 2023, 2:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL