185.15.56.0/24 has been in https://github.com/wikimedia/puppet/blob/production/modules/network/data/data.yaml#L7 for historical reasons.
As this potentially trusted in some ferm filters, and included in the "production" list, it would be great to investigate the impact of removing it from there, and later on, remove it.
No problems on my side.
Probably the smart thing to do is to clearly define the semantics of that data file, so we can safely add/remove stuff from there without fears of breaking/compromising stuff.
Change 677872 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Remove 185.15.56.0/24 from network::external
PCC result: https://puppet-compiler.wmflabs.org/compiler1003/28963/
Impacted hosts:
authdns1001.wikimedia.org cp1079.eqiad.wmnet cp1082.eqiad.wmnet cp2031.codfw.wmnet cp2042.codfw.wmnet cp3052.esams.wmnet cp3059.esams.wmnet cp4025.ulsfo.wmnet cp4032.ulsfo.wmnet cp5002.eqsin.wmnet cp5012.eqsin.wmnet dns2001.wikimedia.org lists1001.wikimedia.org lists1002.wikimedia.org mx1001.wikimedia.org mx2001.wikimedia.org netflow2001.codfw.wmnet otrs1001.eqiad.wmnet phab1001.eqiad.wmnet
Now to figure out the actual service impact and if it's safe to merge.
Pinging @ema, @herron, @BBlack, @akosiaris @Dzahn
To re-iterate, 185.15.56.0/24 is dedicated to WMCS CloudVPS
On cache hosts, the result of that change would be removing the IP from wikimedia_nets and wikimedia_trust, two Varnish ACLs used to distinguish between "internal" and "external" clients. The former ACL controls some rate limiting knobs and whether or not the client can set X-Client-IP, while the latter is used to allow setting X-Forwarded-Proto and X-Request-Id. We might have to revisit this I think (allowing clients to set X-Forwarded-Proto means that they can skip TLS termination, and I am not sure if/why we should allow that), but as things stand today the change would have a functional impact in VCL terms.
The cache proxies currently see Cloud users as IPs in the 172.16.0.0/12 range (T209011), which is not in network::external. Cloud users should not be able to arbitrarily change the IP that MediaWiki thinks that they are using (if that is what X-Client-IP does).
Speaking for phab1001: I am not aware of anything that needs to connect from cloud to phab in production but adding @20after4 as well
Speaking for otrs1001: That has just been replaced by a new system. Not expecting there is anything there either that needs to connect from cloud but adding @akosiaris
Speaking for lists*: Same as above but adding @Legoktm and @Ladsgroup
@mmodell Yes, given that stashbot.toolforge.org has address 185.15.56.11 it seem it would break it. I downvoted on Gerrit with this comment.
Looking at https://puppet-compiler.wmflabs.org/compiler1003/28963/phab1001.eqiad.wmnet/index.html it only affects exim configuration. I don't think phab1001 should be treating mail from Cloud VPS any differently than from the open internet.
This will remove Cloud VPS from wikimedia_nets, which gets some benefits like higher connection limits and bypassing some spam checks (AIUI). So it might have some impact on bots that send mail to lists from Toolforge (I can think of a few), but it'll be manageable and a good idea IMO. I would like to announce it in Tech/News/cloud-l whenever we do this.
It would be nice if we could deploy this change for services individually rather than one bulk puppet change. Maybe remove it from network::external but add it back to individual roles/manifests that currently include it, and then we can individually deploy removing it.
Change 827964 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Squid: permit production networks instead of aggregate_networks
Change 828016 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Exclude cloud-eqiad prefix from lists trusted networks
Change 828019 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Exclude cloud-eqiad prefix from MXs trusted networks
Change 828025 had a related patch set uploaded (by Ayounsi; author: Ayounsi):
[operations/puppet@production] Exclude cloud-eqiad prefix from VRT trusted networks
Reviving this old task, I sent patches (see above) for an incremental change on some of the services (MX/VRT(even if Alex said it was ok)/Lists).
For the ones left (CP, DNS, Phab) I didn't want to make too intrusive changes Puppet-wise. I think it's fine to do them at the same time.
Change 827964 merged by Ayounsi:
[operations/puppet@production] Squid: permit production networks instead of aggregate_networks
Mentioned in SAL (#wikimedia-operations) [2022-09-05T09:17:55Z] <XioNoX> Squid: permit production networks instead of aggregate_networks - T265864
Change 828016 merged by Ayounsi:
[operations/puppet@production] Exclude cloud-eqiad prefix from lists trusted networks
Change 828019 merged by Ayounsi:
[operations/puppet@production] Exclude cloud-eqiad prefix from MXs trusted networks
Change 828025 merged by Dzahn:
[operations/puppet@production] Exclude cloud-eqiad prefix from VRT trusted networks
Change 677872 merged by Ayounsi:
[operations/puppet@production] Remove 185.15.56.0/24 from network::external