Page MenuHomePhabricator

Traffic project in labs cannot talk HTTP with deployment-prep any longer
Closed, ResolvedPublic

Description

traffic-ats-stretch.traffic.eqiad.wmflabs (172.16.2.180/21) cannot open TCP connections to deployment-mediawiki-09.deployment-prep.eqiad.wmflabs (10.68.17.159/21) on port 80.

The latter host has a default INPUT firewall policy of DROP, but explicitly allows TCP connections to port 80 for 172.16.0.0/21.

0     0 ACCEPT     tcp  --  *      *       172.16.0.0/21        0.0.0.0/0            tcp dpt:80

ICMP traffic flows regularly between the two hosts.

The problem was not reproducible up to at least 2018-10-11.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Was traffic-ats-stretch using 172.16.2.180 before this broke? Is it possible you got migrated across regions (and therefore to a different IP range) but didn't put in a request to get a deployment-prep security group which you were relying on updated? Why is the traffic project even talking to our mediawiki servers directly?
I note that deployment-mediawiki-09 only has the default deployment-prep security group which permits in 10/8 traffic on some ports and ICMP from anywhere (as well as some irrelevant Jenkins SSH stuff).

Krenair claimed this task.
Krenair added a subscriber: aborrero.

<Krenair> ema, this was a cross-project traffic flow that I was unaware of... what do you need to talk to deployment-mediawiki* directly for?
<ema> Krenair: that's how I've been testing new varnish versions and varnish changes for a while, now using the same approach for ats as well

I've given IPs in the new region access to port 80 on deployment-mediawiki-0[79] like they had before by virtue of being in 10/8.