In theory this file should be public everywhere:
https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_testlabs/testcontainer/firstfile
It isn't, which has me thinking that we need a firewall or routing change for 28080.
In theory this file should be public everywhere:
https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_testlabs/testcontainer/firstfile
It isn't, which has me thinking that we need a firewall or routing change for 28080.
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | aborrero | T296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet | |||
Resolved | aborrero | T297596 have cloud hardware servers in the cloud realm using a dedicated LB layer | |||
Resolved | aborrero | T324992 cloudlb: create PoC on codfw | |||
Resolved | aborrero | T338937 cloudlb: review swift/radosgw status | |||
Resolved | Andrew | T341380 Open swift port (28080) to the public internet |
Hmm, I think for swift to be useful we will need to expose it publicly so that for example web tools can embed data directly from Swift instead of proxying it. And for that (usage in browsers) I think we should be using the standard port 443. Unless the API and public file access are routed somehow differently?
Change 936657 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] codfw1dev: open radosgw API to the internet
The patch https://gerrit.wikimedia.org/r/936657 should open it.
I'll let you decide if we want it merged or not.
Being codfw1dev, I'm a bit reluctant to have stuff exposed to the internet, so my recommendation would be no! and run tests and stuff from inside the WMF network.
Agreed for eqiad1, probably not for codfw1dev
And for that (usage in browsers) I think we should be using the standard port 443. Unless the API and public file access are routed somehow differently?
This I'm not sure about. The swift UI suggests urls like the above (https://openstack.codfw1dev.wikimediacloud.org:28080/swift/v1/AUTH_testlabs/testcontainer/firstfile) for accessing objects, and that seems to work ok with e.g. wget. We could provide some kind of middleware that reroutes from port 443 on e.g. swift.openstack.<whatever> but would that necessarily be better?
but would that necessarily be better?
I think yes. This service will be directly accessed by end users who might have weird corporate firewalls or similar restricting outbound traffic to non-standard ports.
Change 963325 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] Move radosgw/swift API to port 443, the standard swift port
Change 963325 merged by Andrew Bogott:
[operations/puppet@production] Move radosgw/swift API to port 443, the standard swift port
Change 963330 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/dns@master] wikimediacloud: Add a dedicated CNAME for object storage
Looking on one of the cloudlb hosts in codfw it doesn't look like port 443 is open to the world:
cmooney@cloudlb2001-dev:~$ sudo iptables -L -v --line -n | grep "dpt:443" 1582 0 0 ACCEPT tcp -- * * 10.128.0.0/24 0.0.0.0/0 tcp dpt:443 1583 0 0 ACCEPT tcp -- * * 10.132.0.0/24 0.0.0.0/0 tcp dpt:443 1584 0 0 ACCEPT tcp -- * * 10.136.0.0/24 0.0.0.0/0 tcp dpt:443 1585 0 0 ACCEPT tcp -- * * 10.136.1.0/24 0.0.0.0/0 tcp dpt:443 1586 0 0 ACCEPT tcp -- * * 10.192.0.0/22 0.0.0.0/0 tcp dpt:443 1587 0 0 ACCEPT tcp -- * * 10.192.10.0/24 0.0.0.0/0 tcp dpt:443 1588 0 0 ACCEPT tcp -- * * 10.192.11.0/24 0.0.0.0/0 tcp dpt:443 1589 0 0 ACCEPT tcp -- * * 10.192.12.0/24 0.0.0.0/0 tcp dpt:443 1590 0 0 ACCEPT tcp -- * * 10.192.13.0/24 0.0.0.0/0 tcp dpt:443 1591 0 0 ACCEPT tcp -- * * 10.192.14.0/24 0.0.0.0/0 tcp dpt:443 1592 0 0 ACCEPT tcp -- * * 10.192.15.0/24 0.0.0.0/0 tcp dpt:443 1593 0 0 ACCEPT tcp -- * * 10.192.16.0/22 0.0.0.0/0 tcp dpt:443 1594 0 0 ACCEPT tcp -- * * 10.192.20.0/24 0.0.0.0/0 tcp dpt:443 1595 0 0 ACCEPT tcp -- * * 10.192.21.0/24 0.0.0.0/0 tcp dpt:443 1596 0 0 ACCEPT tcp -- * * 10.192.22.0/24 0.0.0.0/0 tcp dpt:443 1597 0 0 ACCEPT tcp -- * * 10.192.23.0/24 0.0.0.0/0 tcp dpt:443 1598 0 0 ACCEPT tcp -- * * 10.192.32.0/22 0.0.0.0/0 tcp dpt:443 1599 0 0 ACCEPT tcp -- * * 10.192.4.0/24 0.0.0.0/0 tcp dpt:443 1600 0 0 ACCEPT tcp -- * * 10.192.48.0/22 0.0.0.0/0 tcp dpt:443 1601 0 0 ACCEPT tcp -- * * 10.192.5.0/24 0.0.0.0/0 tcp dpt:443 1602 0 0 ACCEPT tcp -- * * 10.192.6.0/24 0.0.0.0/0 tcp dpt:443 1603 0 0 ACCEPT tcp -- * * 10.192.7.0/24 0.0.0.0/0 tcp dpt:443 1604 0 0 ACCEPT tcp -- * * 10.192.72.0/24 0.0.0.0/0 tcp dpt:443 1605 0 0 ACCEPT tcp -- * * 10.192.75.0/24 0.0.0.0/0 tcp dpt:443 1606 0 0 ACCEPT tcp -- * * 10.192.76.0/24 0.0.0.0/0 tcp dpt:443 1607 0 0 ACCEPT tcp -- * * 10.192.8.0/24 0.0.0.0/0 tcp dpt:443 1608 0 0 ACCEPT tcp -- * * 10.192.9.0/24 0.0.0.0/0 tcp dpt:443 1609 0 0 ACCEPT tcp -- * * 10.194.0.0/20 0.0.0.0/0 tcp dpt:443 1610 0 0 ACCEPT tcp -- * * 10.194.128.0/18 0.0.0.0/0 tcp dpt:443 1611 0 0 ACCEPT tcp -- * * 10.194.16.0/21 0.0.0.0/0 tcp dpt:443 1612 0 0 ACCEPT tcp -- * * 10.194.61.0/24 0.0.0.0/0 tcp dpt:443 1613 0 0 ACCEPT tcp -- * * 10.194.62.0/23 0.0.0.0/0 tcp dpt:443 1614 0 0 ACCEPT tcp -- * * 10.2.1.0/24 0.0.0.0/0 tcp dpt:443 1615 0 0 ACCEPT tcp -- * * 10.2.2.0/24 0.0.0.0/0 tcp dpt:443 1616 0 0 ACCEPT tcp -- * * 10.2.3.0/24 0.0.0.0/0 tcp dpt:443 1617 0 0 ACCEPT tcp -- * * 10.2.4.0/24 0.0.0.0/0 tcp dpt:443 1618 0 0 ACCEPT tcp -- * * 10.2.5.0/24 0.0.0.0/0 tcp dpt:443 1619 0 0 ACCEPT tcp -- * * 10.2.6.0/24 0.0.0.0/0 tcp dpt:443 1620 0 0 ACCEPT tcp -- * * 10.64.0.0/22 0.0.0.0/0 tcp dpt:443 1621 0 0 ACCEPT tcp -- * * 10.64.130.0/24 0.0.0.0/0 tcp dpt:443 1622 0 0 ACCEPT tcp -- * * 10.64.131.0/24 0.0.0.0/0 tcp dpt:443 1623 0 0 ACCEPT tcp -- * * 10.64.132.0/24 0.0.0.0/0 tcp dpt:443 1624 0 0 ACCEPT tcp -- * * 10.64.134.0/24 0.0.0.0/0 tcp dpt:443 1625 0 0 ACCEPT tcp -- * * 10.64.135.0/24 0.0.0.0/0 tcp dpt:443 1626 0 0 ACCEPT tcp -- * * 10.64.136.0/24 0.0.0.0/0 tcp dpt:443 1627 0 0 ACCEPT tcp -- * * 10.64.138.0/24 0.0.0.0/0 tcp dpt:443 1628 0 0 ACCEPT tcp -- * * 10.64.139.0/24 0.0.0.0/0 tcp dpt:443 1629 0 0 ACCEPT tcp -- * * 10.64.140.0/24 0.0.0.0/0 tcp dpt:443 1630 0 0 ACCEPT tcp -- * * 10.64.142.0/24 0.0.0.0/0 tcp dpt:443 1631 0 0 ACCEPT tcp -- * * 10.64.143.0/24 0.0.0.0/0 tcp dpt:443 1632 0 0 ACCEPT tcp -- * * 10.64.144.0/24 0.0.0.0/0 tcp dpt:443 1633 0 0 ACCEPT tcp -- * * 10.64.148.0/24 0.0.0.0/0 tcp dpt:443 1634 0 0 ACCEPT tcp -- * * 10.64.149.0/24 0.0.0.0/0 tcp dpt:443 1635 0 0 ACCEPT tcp -- * * 10.64.150.0/24 0.0.0.0/0 tcp dpt:443 1636 0 0 ACCEPT tcp -- * * 10.64.151.0/24 0.0.0.0/0 tcp dpt:443 1637 0 0 ACCEPT tcp -- * * 10.64.16.0/22 0.0.0.0/0 tcp dpt:443 1638 0 0 ACCEPT tcp -- * * 10.64.20.0/24 0.0.0.0/0 tcp dpt:443 1639 0 0 ACCEPT tcp -- * * 10.64.21.0/24 0.0.0.0/0 tcp dpt:443 1640 0 0 ACCEPT tcp -- * * 10.64.32.0/22 0.0.0.0/0 tcp dpt:443 1641 0 0 ACCEPT tcp -- * * 10.64.36.0/24 0.0.0.0/0 tcp dpt:443 1642 0 0 ACCEPT tcp -- * * 10.64.37.0/24 0.0.0.0/0 tcp dpt:443 1643 0 0 ACCEPT tcp -- * * 10.64.4.0/24 0.0.0.0/0 tcp dpt:443 1644 0 0 ACCEPT tcp -- * * 10.64.48.0/22 0.0.0.0/0 tcp dpt:443 1645 0 0 ACCEPT tcp -- * * 10.64.5.0/24 0.0.0.0/0 tcp dpt:443 1646 0 0 ACCEPT tcp -- * * 10.64.52.0/24 0.0.0.0/0 tcp dpt:443 1647 0 0 ACCEPT tcp -- * * 10.64.53.0/24 0.0.0.0/0 tcp dpt:443 1648 0 0 ACCEPT tcp -- * * 10.64.72.0/24 0.0.0.0/0 tcp dpt:443 1649 0 0 ACCEPT tcp -- * * 10.64.75.0/24 0.0.0.0/0 tcp dpt:443 1650 0 0 ACCEPT tcp -- * * 10.64.76.0/24 0.0.0.0/0 tcp dpt:443 1651 0 0 ACCEPT tcp -- * * 10.67.0.0/20 0.0.0.0/0 tcp dpt:443 1652 0 0 ACCEPT tcp -- * * 10.67.128.0/18 0.0.0.0/0 tcp dpt:443 1653 0 0 ACCEPT tcp -- * * 10.67.16.0/21 0.0.0.0/0 tcp dpt:443 1654 0 0 ACCEPT tcp -- * * 10.67.24.0/21 0.0.0.0/0 tcp dpt:443 1655 0 0 ACCEPT tcp -- * * 10.67.32.0/20 0.0.0.0/0 tcp dpt:443 1656 0 0 ACCEPT tcp -- * * 10.67.64.0/20 0.0.0.0/0 tcp dpt:443 1657 0 0 ACCEPT tcp -- * * 10.67.80.0/21 0.0.0.0/0 tcp dpt:443 1658 0 0 ACCEPT tcp -- * * 10.80.0.0/24 0.0.0.0/0 tcp dpt:443 1659 0 0 ACCEPT tcp -- * * 10.80.1.0/24 0.0.0.0/0 tcp dpt:443 1660 0 0 ACCEPT tcp -- * * 103.102.166.0/28 0.0.0.0/0 tcp dpt:443 1661 0 0 ACCEPT tcp -- * * 103.102.166.224/27 0.0.0.0/0 tcp dpt:443 1662 0 0 ACCEPT tcp -- * * 172.16.0.0/21 0.0.0.0/0 tcp dpt:443 1663 0 0 ACCEPT tcp -- * * 172.16.128.0/24 0.0.0.0/0 tcp dpt:443 1664 0 0 ACCEPT tcp -- * * 172.20.1.0/24 0.0.0.0/0 tcp dpt:443 1665 0 0 ACCEPT tcp -- * * 172.20.2.0/24 0.0.0.0/0 tcp dpt:443 1666 0 0 ACCEPT tcp -- * * 172.20.255.0/24 0.0.0.0/0 tcp dpt:443 1667 0 0 ACCEPT tcp -- * * 172.20.3.0/24 0.0.0.0/0 tcp dpt:443 1668 0 0 ACCEPT tcp -- * * 172.20.4.0/24 0.0.0.0/0 tcp dpt:443 1669 0 0 ACCEPT tcp -- * * 172.20.5.0/24 0.0.0.0/0 tcp dpt:443 1670 0 0 ACCEPT tcp -- * * 185.15.56.0/25 0.0.0.0/0 tcp dpt:443 1671 0 0 ACCEPT tcp -- * * 185.15.56.160/28 0.0.0.0/0 tcp dpt:443 1672 0 0 ACCEPT tcp -- * * 185.15.57.0/29 0.0.0.0/0 tcp dpt:443 1673 0 0 ACCEPT tcp -- * * 185.15.57.16/29 0.0.0.0/0 tcp dpt:443 1674 0 0 ACCEPT tcp -- * * 185.15.57.24/29 0.0.0.0/0 tcp dpt:443 1675 0 0 ACCEPT tcp -- * * 185.15.58.0/27 0.0.0.0/0 tcp dpt:443 1676 0 0 ACCEPT tcp -- * * 185.15.58.224/27 0.0.0.0/0 tcp dpt:443 1677 0 0 ACCEPT tcp -- * * 185.15.58.32/27 0.0.0.0/0 tcp dpt:443 1678 0 0 ACCEPT tcp -- * * 185.15.59.0/27 0.0.0.0/0 tcp dpt:443 1679 0 0 ACCEPT tcp -- * * 185.15.59.224/27 0.0.0.0/0 tcp dpt:443 1680 0 0 ACCEPT tcp -- * * 185.15.59.32/27 0.0.0.0/0 tcp dpt:443 1681 0 0 ACCEPT tcp -- * * 198.35.26.0/28 0.0.0.0/0 tcp dpt:443 1682 0 0 ACCEPT tcp -- * * 198.35.26.96/27 0.0.0.0/0 tcp dpt:443 1683 0 0 ACCEPT tcp -- * * 208.80.153.0/27 0.0.0.0/0 tcp dpt:443 1684 0 0 ACCEPT tcp -- * * 208.80.153.224/27 0.0.0.0/0 tcp dpt:443 1685 0 0 ACCEPT tcp -- * * 208.80.153.32/27 0.0.0.0/0 tcp dpt:443 1686 0 0 ACCEPT tcp -- * * 208.80.153.64/27 0.0.0.0/0 tcp dpt:443 1687 0 0 ACCEPT tcp -- * * 208.80.153.96/27 0.0.0.0/0 tcp dpt:443 1688 0 0 ACCEPT tcp -- * * 208.80.154.0/26 0.0.0.0/0 tcp dpt:443 1689 0 0 ACCEPT tcp -- * * 208.80.154.128/26 0.0.0.0/0 tcp dpt:443 1690 0 0 ACCEPT tcp -- * * 208.80.154.224/27 0.0.0.0/0 tcp dpt:443 1691 0 0 ACCEPT tcp -- * * 208.80.154.64/26 0.0.0.0/0 tcp dpt:443 1692 0 0 ACCEPT tcp -- * * 208.80.155.96/27 0.0.0.0/0 tcp dpt:443
The CR routers don't block any traffic from public IPs like 185.15.57.24 (openstack.codfw1dev.wikimediacloud.org) routed in the cloud vrf, so I think the only place that it needs to be allowed is on the cloud hosts themselves. A curl from my home machine doesn't get a response, but the packets hit cloudlb2002-dev:
cathal@officepc:~$ curl -v https://openstack.codfw1dev.wikimediacloud.org/swift/v1/AUTH_testlabs/testcontainer/firstfile * Trying 185.15.57.24:443... * connect to 185.15.57.24 port 443 failed: Connection timed out * Failed to connect to openstack.codfw1dev.wikimediacloud.org port 443 after 129334 ms: Connection timed out
cmooney@cloudlb2002-dev:~$ sudo tcpdump -l -p -nn -i vlan2151 host 185.15.57.24 and tcp port 443 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on vlan2151, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:33:41.875842 IP 176.61.34.2.56374 > 185.15.57.24.443: Flags [S], seq 1305605804, win 64240, options [mss 1460,sackOK,TS val 381362388 ecr 0,nop,wscale 7], length 0 17:33:42.904491 IP 176.61.34.2.56374 > 185.15.57.24.443: Flags [S], seq 1305605804, win 64240, options [mss 1460,sackOK,TS val 381363417 ecr 0,nop,wscale 7], length 0 17:33:44.921946 IP 176.61.34.2.56374 > 185.15.57.24.443: Flags [S], seq 1305605804, win 64240, options [mss 1460,sackOK,TS val 381365434 ecr 0,nop,wscale 7], length 0
Change 963330 merged by Majavah:
[operations/dns@master] wikimediacloud: Add a dedicated CNAME for object storage
That's on purpose. There's an 'open_to_internet' flag in hiera that determines if a given service is firewalled or not; for codfw1dev we've been leaving the apis closed off. It's easy to switch if we need it later.
Change 936657 abandoned by Andrew Bogott:
[operations/puppet@production] codfw1dev: open radosgw API to the internet
Reason:
As a rule we don't have codfw1dev apis open to the public.