Page MenuHomePhabricator
Create Task
Maniphest T238720

Deprecate and disable port 80 for one-off sites under canonical domains
Closed, ResolvedPublic
Actions

Description

Leveraging the fact that WMF canonical domains are on the HSTS preload list, we don't need to listen on port 80, as every request even http:// ones should reach the servers via https thanks to HSTS.

The first step will be to replace the current redirect from http to https with a simple 403, on a second stage we will disable port 80 completely.

Sites rejecting traffic on port 80 with a 403 response:

  • apt.wikimedia.org
  • archiva.wikimedia.org
  • icinga.wikimedia.org
  • dumps.wikimedia.org
  • gerrit.wikimedia.org
  • gitlab.wikimedia.org
  • librenms.wikimedia.org
  • lists.wikimedia.org (per https://phabricator.wikimedia.org/T238720#8752022)
  • mirrors.wikimedia.org
  • orchestrator.wikimedia.org

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -12lists: Disable access on port 80
operations/puppetproduction+1 -9gitlab: Disable listening on port 80
operations/puppetproduction+3 -2gerrit: Reject non-tls requests with a 403
operations/puppetproduction+1 -1archiva: Reject non-tls requests with a 403
operations/puppetproduction+3 -3icinga: Reject non-tls requests with a 403
operations/puppetproduction+1 -1dumps: fix http alert to check the new status
operations/puppetproduction+1 -1dumps: Reject non-tls requests with a 403
operations/puppetproduction+3 -4orchestrator: Reject non-tls requests with a 403
operations/puppetproduction+3 -2librenms: Reject plain text requests with a 403
Customize query in gerrit

Related Objects

Event Timeline

Vgutierrez created this task.Nov 20 2019, 4:57 AM
Restricted Application added a project: SRE. · View Herald TranscriptNov 20 2019, 4:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a comment.Nov 20 2019, 5:05 AM

Change 551950 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] librenms: Reject plain text requests with a 403

https://gerrit.wikimedia.org/r/551950

gerritbot added a project: Patch-For-Review.Nov 20 2019, 5:05 AM
Bugreporter added a subscriber: Bugreporter.Nov 20 2019, 5:09 AM

Note very old browsers may not support HSTS preload list or even HSTS itself; probably we want to configure a specific 403 message (or still redirect them when user agent is these browsers).

Vgutierrez added a comment.Nov 20 2019, 5:20 AM

we are targeting here the one-off sites, some of them are already configured to support TLSv1.2 only, that's usually a stricter requirement for UAs than HSTS support itself.

Comparing https://caniuse.com/#feat=tls1-2 VS https://caniuse.com/#feat=stricttransportsecurity it looks like all the UAs that support TLSv1.2 also support HSTS wit h the exception of Opera Mini, I believe that's not a blocker for the one-off sites.

Vgutierrez added a comment.Nov 20 2019, 5:21 AM

Just to be clear, wikipedia and the rest of the canonical sites are out of scope for this task :)

Vgutierrez triaged this task as Medium priority.Nov 20 2019, 5:22 AM
Vgutierrez moved this task from Backlog to TLS on the Traffic board.
gerritbot added a comment.Nov 20 2019, 10:38 AM

Change 551950 merged by Vgutierrez:
[operations/puppet@production] librenms: Reject plain text requests with a 403

https://gerrit.wikimedia.org/r/551950

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2019, 11:10 AM
Seb35 added a subscriber: Seb35.Nov 28 2019, 11:25 AM

For information, due to this bug in Firefox, when the user type the URL without the "https://" prefix Firefox does not even try to connect to https when port 80 is closed. This means Firefox displays a timeout error - I just tried with Firefox Nightly. Hence some users could report "Wikipedia is unreachable".

Vgutierrez added a comment.Nov 28 2019, 11:30 AM

Hmm with HSTS the browser shouldn't even try port 80.

Seb35 added a comment.Nov 28 2019, 2:54 PM

Yes, indeed, I have to precise my test was with a non-HSTS site, and it seems there is no issue with HSTS-preloaded sites according to this comment. I tested connecting to Wikipedia as an HSTS-preloaded site and blocking locally my port 80 in output with iptables, there is no issue.

So this Firefox bug is not an issue here.

BBlack edited projects, added Traffic-Icebox; removed Traffic.Sep 1 2021, 11:21 PM
BBlack added a subscriber: BBlack.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Vgutierrez mentioned this in T323557: Let HAProxy handle port 80.Nov 22 2022, 10:11 AM
gerritbot added a comment.Nov 22 2022, 10:31 AM

Change 859449 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] orchestrator: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859449

gerritbot added a project: Patch-For-Review.Nov 22 2022, 10:31 AM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Nov 22 2022, 10:39 AM
Vgutierrez edited projects, added Traffic; removed Traffic-Icebox.Nov 22 2022, 10:46 AM
gerritbot added a comment.Nov 22 2022, 10:47 AM

Change 859449 merged by Vgutierrez:

[operations/puppet@production] orchestrator: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859449

jbond added a subscriber: jbond.Nov 22 2022, 10:49 AM
gerritbot added a comment.Nov 22 2022, 10:55 AM

Change 859457 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] icinga: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859457

Vgutierrez updated the task description. (Show Details)Nov 22 2022, 11:18 AM
Vgutierrez updated the task description. (Show Details)Nov 22 2022, 11:20 AM
gerritbot added a comment.Nov 22 2022, 11:26 AM

Change 859467 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] dumps: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859467

gerritbot added a comment.Nov 22 2022, 11:58 AM

Change 859467 merged by Vgutierrez:

[operations/puppet@production] dumps: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859467

Vgutierrez updated the task description. (Show Details)Nov 22 2022, 12:04 PM
gerritbot added a comment.Nov 22 2022, 1:28 PM

Change 859498 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] dumps: fix http alert to check the new status

https://gerrit.wikimedia.org/r/859498

gerritbot added a comment.Nov 22 2022, 1:35 PM

Change 859498 merged by David Caro:

[operations/puppet@production] dumps: fix http alert to check the new status

https://gerrit.wikimedia.org/r/859498

gerritbot added a comment.Nov 22 2022, 1:56 PM

Change 859457 merged by Vgutierrez:

[operations/puppet@production] icinga: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859457

Stashbot added a comment.Nov 22 2022, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-22T13:57:46Z] <vgutierrez> block plain text requests on icinga.wm.o - T238720

Vgutierrez updated the task description. (Show Details)Nov 22 2022, 2:15 PM
KOfori added a subscriber: KOfori.Nov 22 2022, 2:25 PM
gerritbot added a comment.Nov 23 2022, 10:03 AM

Change 859983 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] archiva: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859983

gerritbot added a comment.Nov 23 2022, 10:06 AM

Change 859986 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] gerrit: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859986

gerritbot added a comment.Nov 23 2022, 2:44 PM

Change 859983 merged by Vgutierrez:

[operations/puppet@production] archiva: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859983

Vgutierrez updated the task description. (Show Details)Nov 23 2022, 3:17 PM
gerritbot added a comment.Nov 28 2022, 9:41 AM

Change 859986 merged by Vgutierrez:

[operations/puppet@production] gerrit: Reject non-tls requests with a 403

https://gerrit.wikimedia.org/r/859986

Vgutierrez updated the task description. (Show Details)Nov 28 2022, 9:45 AM
Vgutierrez updated the task description. (Show Details)Nov 28 2022, 9:47 AM
Vgutierrez updated the task description. (Show Details)
hashar added a subscriber: hashar.Nov 28 2022, 9:52 AM

For Gerrit, I have made the announcement on wikitech-l: Gerrit now forbids plain HTTP.

BCornwall changed the task status from Open to Stalled.Mar 24 2023, 10:42 PM
BCornwall moved this task from TLS to Ready for work on the Traffic board.
BCornwall lowered the priority of this task from Medium to Low.Mar 24 2023, 10:49 PM
BCornwall added a subscriber: BCornwall.
Maintenance_bot removed a project: Patch-For-Review.Mar 24 2023, 11:10 PM
hashar removed a subscriber: hashar.Mar 26 2023, 8:16 AM
BCornwall moved this task from Ready for work to Backlog on the Traffic board.Mar 30 2023, 8:40 PM
gerritbot added a comment.Mar 31 2023, 7:48 PM

Change 904843 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] gitlab: Disable listening on port 80

https://gerrit.wikimedia.org/r/904843

gerritbot added a project: Patch-For-Review.Mar 31 2023, 7:48 PM
gerritbot added a comment.Mar 31 2023, 9:04 PM

Change 904854 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] lists: Disable access on port 80

https://gerrit.wikimedia.org/r/904854

BCornwall changed the task status from Stalled to In Progress.Mar 31 2023, 9:16 PM
BCornwall claimed this task.
BCornwall moved this task from Backlog to Traffic team actively servicing on the Traffic board.
BCornwall added a subscriber: Ladsgroup.Apr 3 2023, 5:11 PM

@Ladsgroup You expressed opposition to removing :80 from lists. Is that to say that you don't see a way forward and that lists should be removed from the sites in the ticket description?

Ladsgroup added a comment.EditedApr 3 2023, 7:48 PM

Well, I don't oppose it, it's just that this port is used internally and it'll break, you might be able to see the traffic flowing if you do tcpdump on the host, the archiver service (hyperkitty daemon) calls the webservice on port 80 in localhost and when I was doing the upgrade of mm2 to mm3 I couldn't find a workaround for it back then but this kind of work is not usual to me so I might have missed some easy way to fix it. If there is not an easy way or you don't want to/have resources, I don't oppose leaving it as-is and removing the site from this ticket's description.

Hope that clears it a bit.

RhinosF1 added a subscriber: RhinosF1.Apr 3 2023, 7:52 PM
Jelto added a project: collaboration-services.Apr 4 2023, 7:42 AM
Jelto moved this task from Incoming to Consultation on the collaboration-services board.
Jelto added a project: GitLab (Infrastructure).
Jelto added subscribers: brennen, hashar, Jelto.Apr 4 2023, 8:00 AM

@brennen @hashar there is a open change to disable port 80 on GitLab, similar to Gerrit some time ago.

Any concerns with that? I would create a similar announcement on wikitech-l like we've done for gerrit.

hashar added a comment.Apr 4 2023, 8:57 AM

Gitlab being fairly recent, I don't think it ever got advertised with http rather than https so it must be fine to drop port 80 entirely. For Gerrit, that is pretty much the same (it always had https).

Maybe we have some statistics somewhere to lookup whether people try to reach Gitlab over http / port 80, but I don't think it matters.

I have +1 ed the Puppet change.

Jelto added a comment.Apr 4 2023, 11:58 AM
In T238720#8753483, @hashar wrote:

Gitlab being fairly recent, I don't think it ever got advertised with http rather than https so it must be fine to drop port 80 entirely. For Gerrit, that is pretty much the same (it always had https).

Maybe we have some statistics somewhere to lookup whether people try to reach Gitlab over http / port 80, but I don't think it matters.

I have +1 ed the Puppet change.

Thanks for the feedback and review.

Regarding statistics we have nginx access logs. Logstash/access logs do not contain the port so it's a bit tricky to find http requests. We can look for http 301 like the following logstash query:
https://logstash.wikimedia.org/goto/c2db07f5cf2163763138820d35062d33

But I think the above search contains also https traffic.

hashar added a comment.Apr 4 2023, 12:31 PM

That does not provide much information :) I say go for it, I don't think anything accesses Gitlab from http.

gerritbot added a comment.Apr 4 2023, 5:10 PM

Change 904843 merged by BCornwall:

[operations/puppet@production] gitlab: Disable listening on port 80

https://gerrit.wikimedia.org/r/904843

BCornwall updated the task description. (Show Details)Apr 4 2023, 5:21 PM
Dzahn awarded a token.Apr 4 2023, 5:37 PM
Jelto added a comment.Apr 4 2023, 5:50 PM

I send a short message on wikitech-l, in case something breaks on GitLab so users are aware of the change.
Thanks for merging!

BCornwall closed this task as Resolved.Apr 7 2023, 3:51 PM
BCornwall updated the task description. (Show Details)

I went ahead and struck lists off of the.... list since it seems there's a reason for port 80 to be exposed. If removing port 80 is something we want to do for that service, I think creating a new ticket would be beneficial. Thanks!

gerritbot added a comment.Apr 28 2023, 4:32 PM

Change 904854 abandoned by BCornwall:

[operations/puppet@production] lists: Disable access on port 80

Reason:

The service still uses :80 for various tasks

https://gerrit.wikimedia.org/r/904854

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2023, 5:10 PM
hashar removed a subscriber: hashar.May 3 2023, 2:51 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL