Page MenuHomePhabricator

Remove 185.15.56.0/24 from network::external
Open, MediumPublic

Description

185.15.56.0/24 has been in https://github.com/wikimedia/puppet/blob/production/modules/network/data/data.yaml#L7 for historical reasons.
As this potentially trusted in some ferm filters, and included in the "production" list, it would be great to investigate the impact of removing it from there, and later on, remove it.

  • DNS
  • CP
  • Lists
  • MX
  • Netflow
  • OTRS
  • Phab

Event Timeline

ayounsi triaged this task as Medium priority.Oct 19 2020, 7:22 AM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

No problems on my side.

Probably the smart thing to do is to clearly define the semantics of that data file, so we can safely add/remove stuff from there without fears of breaking/compromising stuff.

Change 677872 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove 185.15.56.0/24 from network::external

https://gerrit.wikimedia.org/r/677872

PCC result: https://puppet-compiler.wmflabs.org/compiler1003/28963/
Impacted hosts:

authdns1001.wikimedia.org
cp1079.eqiad.wmnet
cp1082.eqiad.wmnet
cp2031.codfw.wmnet
cp2042.codfw.wmnet
cp3052.esams.wmnet
cp3059.esams.wmnet
cp4025.ulsfo.wmnet
cp4032.ulsfo.wmnet
cp5002.eqsin.wmnet
cp5012.eqsin.wmnet
dns2001.wikimedia.org
lists1001.wikimedia.org
lists1002.wikimedia.org
mx1001.wikimedia.org
mx2001.wikimedia.org
netflow2001.codfw.wmnet
otrs1001.eqiad.wmnet
phab1001.eqiad.wmnet

Now to figure out the actual service impact and if it's safe to merge.

Pinging @ema, @herron, @BBlack, @akosiaris @Dzahn

To re-iterate, 185.15.56.0/24 is dedicated to WMCS CloudVPS

Now to figure out the actual service impact and if it's safe to merge.
To re-iterate, 185.15.56.0/24 is dedicated to WMCS CloudVPS

On cache hosts, the result of that change would be removing the IP from wikimedia_nets and wikimedia_trust, two Varnish ACLs used to distinguish between "internal" and "external" clients. The former ACL controls some rate limiting knobs and whether or not the client can set X-Client-IP, while the latter is used to allow setting X-Forwarded-Proto and X-Request-Id. We might have to revisit this I think (allowing clients to set X-Forwarded-Proto means that they can skip TLS termination, and I am not sure if/why we should allow that), but as things stand today the change would have a functional impact in VCL terms.

On cache hosts, the result of that change would be removing the IP from wikimedia_nets and wikimedia_trust, two Varnish ACLs used to distinguish between "internal" and "external" clients. The former ACL controls some rate limiting knobs and whether or not the client can set X-Client-IP, while the latter is used to allow setting X-Forwarded-Proto and X-Request-Id. We might have to revisit this I think (allowing clients to set X-Forwarded-Proto means that they can skip TLS termination, and I am not sure if/why we should allow that), but as things stand today the change would have a functional impact in VCL terms.

The cache proxies currently see Cloud users as IPs in the 172.16.0.0/12 range (T209011), which is not in network::external. Cloud users should not be able to arbitrarily change the IP that MediaWiki thinks that they are using (if that is what X-Client-IP does).

They're currently coming from the 172.16.0.0/12 space but T209011 is going to change it so they will come from the 185.15.56.0/24 space.
Does that mean this change should be a prerequisite to T209011? Or is there more investigation to make? Trying to figure out the next steps.

Speaking for phab1001: I am not aware of anything that needs to connect from cloud to phab in production but adding @20after4 as well

Speaking for otrs1001: That has just been replaced by a new system. Not expecting there is anything there either that needs to connect from cloud but adding @akosiaris

Speaking for lists*: Same as above but adding @Legoktm and @Ladsgroup

@Dzahn: would this block toolhub tools from connecting to phab? Things like Stashbot and Wikibugs come to mind.

@mmodell Yes, given that stashbot.toolforge.org has address 185.15.56.11 it seem it would break it. I downvoted on Gerrit with this comment.

@Dzahn: would this block toolhub tools from connecting to phab? Things like Stashbot and Wikibugs come to mind.

Looking at https://puppet-compiler.wmflabs.org/compiler1003/28963/phab1001.eqiad.wmnet/index.html it only affects exim configuration. I don't think phab1001 should be treating mail from Cloud VPS any differently than from the open internet.

Speaking for lists*: Same as above but adding @Legoktm and @Ladsgroup

This will remove Cloud VPS from wikimedia_nets, which gets some benefits like higher connection limits and bypassing some spam checks (AIUI). So it might have some impact on bots that send mail to lists from Toolforge (I can think of a few), but it'll be manageable and a good idea IMO. I would like to announce it in Tech/News/cloud-l whenever we do this.


It would be nice if we could deploy this change for services individually rather than one bulk puppet change. Maybe remove it from network::external but add it back to individual roles/manifests that currently include it, and then we can individually deploy removing it.