Maniphest T209460

CloudVPS: network architecture
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	aborrero
	Nov 14 2018, 9:54 AM

Description

This is the epic tracking task for the current efforts to re-evaluate and re-think the CloudVPS network architecture and the relationship between production and cloud realms.

We have several working documents that are related to this:

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T209460 CloudVPS: network architecture
Resolved	aborrero	T174596 dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour
Resolved	ayounsi	T207663 Renumber cloud-instance-transport1-b-eqiad to public IPs
		Restricted Task
		Restricted Task
		Restricted Task
Resolved	ayounsi	T211254 Free up 185.15.59.0/24
Resolved	None	T244727 CloudVPS: networking improvements
Declined	aborrero	T244851 Neutron: replace NAT customization with address scopes
Declined	• JHedden	T247633 New network request for CloudVPS CODFW instances transport
Open	None	T245495 CloudVPS: IPv6 early PoC
Open	None	T187929 Cloud IPv6 subnets
Declined	None	T245606 CloudVPS: enable BGP in the neutron transport network
Resolved	aborrero	T246887 CloudVPS: introduce filtering for neutron BGP addresses
Resolved	aborrero	T247505 CloudVPS: neutron: consider dropping routing_source_ip custom hack from the l3 agent
Resolved	ayounsi	T265288 Enable L3 routing on cloudsw nodes
Resolved	aborrero	T261724 cloudgw: evaluate / validate setup in codfw1dev
Resolved	ayounsi	T263622 codfw: more vlans setup changes in the cloudgw PoC
Resolved	aborrero	T270704 cloud: introduce new edge network architecture for eqiad1 and codfw1dev
		Unknown Object (Task)
Resolved	Papaul	T271590 (Need By: TBD) rack/setup/install cloudgw2002-dev.codfw.wmnet
		Unknown Object (Task)
Resolved	RobH	T272403 (Need By: 2021-03-31) rack/setup/install cloudgw100[12].eqiad.wmnet
Resolved	aborrero	T271519 codfw1dev: repurpose/rename labtestvirt2003.codfw.wmnet as cloudgw2001-dev.codfw.wmnet
		Unknown Object (Task)
Resolved	aborrero	T272963 cloudgw: develop HA setup
Resolved	MoritzMuehlenhoff	T275129 cloudgw: linux kernel >= 5.6 highly convenient
Resolved	ayounsi	T277020 cloudgw eqiad1: review & allocate subnets and VLANs
Resolved	aborrero	T277287 cloudgw: IPv6 issue in the control plane network
Resolved	aborrero	T281124 cloudgw: introduce grafana dashboard panels
		Unknown Object (Task)

Event Timeline

aborrero created this task.Nov 14 2018, 9:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2018, 9:54 AM

aborrero triaged this task as Medium priority.Nov 14 2018, 9:54 AM

aborrero added subtasks: T207536: Move various support services for Cloud VPS currently in prod into their own instances, T209011: Change routing to ensure that traffic originating from Cloud VPS is seen as non-private IPs by Wikimedia wikis.

aborrero moved this task from Inbox to Epics on the cloud-services-team (Kanban) board.

Krenair subscribed.Nov 14 2018, 9:55 AM

MoritzMuehlenhoff subscribed.Nov 14 2018, 10:04 AM

The document is: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron_ideal_model (edits welcome).

I put in some basic ones: https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Neutron_ideal_model&diff=1808555&oldid=1808492

From the page:

full IPv6 support in Cloud VPS, meaning that every single VM has his own IPv6 allocated (TODO: or even a range per project?)

I'd suggest that each project should get it's own IPv6 range and ideally it's own IPv4 range. Recently I've seen a couple of interesting cross-project traffic flows that flew under the radar due to generic 10/8 rules. The eqiad1-r migration showed some of them but it's not the solution to the problem as there's still a flat network which any project will have instances scattered across. I haven't edited this in but I think we should? I don't know how much space we're willing to give each project.

Edit: After some semi-related discussion with Arturo I've split this out into it's own entry on the page

aborrero added a subtask: T174596: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour.Nov 14 2018, 10:46 AM

aborrero added a subtask: T41785: Create a Cloud VPS SMTP smarthost.

herron closed subtask T41785: Create a Cloud VPS SMTP smarthost as Resolved.Nov 26 2018, 9:05 PM

aborrero added a subtask: T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs.Dec 3 2018, 1:44 PM

ayounsi added a subtask: Restricted Task.Dec 6 2018, 9:15 PM

Paladox subscribed.Dec 6 2018, 9:20 PM

aborrero added a subtask: T37947: Enable IPv6 on CloudVPS.Dec 11 2018, 10:54 AM

Krenair removed a subtask: T37947: Enable IPv6 on CloudVPS.Dec 11 2018, 11:14 AM

Krenair added a parent task: T37947: Enable IPv6 on CloudVPS.

Can someone point me to the current network layout? Vlans, ip space in use, what's used to route/filter traffic, etc.? Knowing the current situation is usually a good first step when designing a to be situation. Does Wikimedia have an overall architecture or architecture principles? That would be good input too.

In T209460#4815447, @Multichill wrote:

Can someone point me to the current network layout? Vlans, ip space in use, what's used to route/filter traffic, etc.? Knowing the current situation is usually a good first step when designing a to be situation. Does Wikimedia have an overall architecture or architecture principles? That would be good input too.

Regarding the Neutron side, you can check this https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron

aborrero mentioned this in T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.Dec 19 2018, 4:52 PM

aborrero closed subtask T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs as Resolved.Dec 20 2018, 5:40 PM

ayounsi reopened subtask T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs as Open.Dec 20 2018, 5:50 PM

ayounsi closed subtask T207663: Renumber cloud-instance-transport1-b-eqiad to public IPs as Resolved.Jan 3 2019, 5:05 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:40 PM

• GTirloni unsubscribed.Mar 21 2019, 9:10 PM

aborrero changed the status of subtask T209011: Change routing to ensure that traffic originating from Cloud VPS is seen as non-private IPs by Wikimedia wikis from Open to Stalled.Nov 22 2019, 10:21 AM

aborrero removed aborrero as the assignee of this task.Jan 16 2020, 2:35 PM

• Bstorm added a subtask: T249022: Track and list the services that Cloud Services that connect to internal network endpoints.Mar 31 2020, 6:01 PM

The natural followup of this is in the 2020 network refresh project (clousw/cloudgw):

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh