Page MenuHomePhabricator

lighttpd redirects URLs of directories without a trailing slash from https to http
Closed, ResolvedPublic

Description

+++ This bug was initially created as a clone of Bug #59926 +++

lighttpd redirects URLs of directories without a trailing slash from https to http, i. e.:

[tim@passepartout ~]$ curl -I https://tools.wmflabs.org/matthewrbowker/cnrd
HTTP/1.1 301 Moved Permanently
Server: nginx/1.5.0
Date: Tue, 29 Apr 2014 22:07:20 GMT
Connection: keep-alive
Location: http://tools.wmflabs.org/matthewrbowker/cnrd/
[tim@passepartout ~]$

So we'll need to teach lighttpd about https (or rewrite redirects in nginx?). From bug #59926, comment #8:

[...] The proxy provides an X-FORWARDED-PROTO header which can
be used by the application to construct URIs if it really must redirect
between protocols.


Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=53689

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:17 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz64627.

(In reply to Liangent from comment #1)

dupe of https://bugzilla.wikimedia.org/show_bug.cgi?id=53689 ?

seems this bug was accidentially fixed.

(In reply to Liangent from comment #2)

dupe of https://bugzilla.wikimedia.org/show_bug.cgi?id=53689 ?

seems this bug was accidentially fixed.

Just to be clear: Now you don't consider /this/ bug #64627 a duplicate of bug #53689, and this bug #64627 is "still" not fixed?

(In reply to Tim Landscheidt from comment #3)

(In reply to Liangent from comment #2)

dupe of https://bugzilla.wikimedia.org/show_bug.cgi?id=53689 ?

seems this bug was accidentially fixed.

Just to be clear: Now you don't consider /this/ bug #64627 a duplicate of
bug #53689, and this bug #64627 is "still" not fixed?

right

metatron wrote:

If trailing / is missing, proxy adds it but switches then from https to http

Example:
(all but the last entry in top menu have trailing / ) I left the last without it, to have some guinea pig.

  1. https://tools.wmflabs.org/xtools/ec/
  2. Select menu SC

metatron wrote:

And it is webproxy who does this, not lighty. So, the caption of this bz may be misleading.

(In reply to metatron from comment #6)

And it is webproxy who does this, not lighty. So, the caption of this bz may
be misleading.

No, it doesn't. The webproxy passes the request to lighttpd who then -- if needed -- generates a redirect which will always be to the http server. That gets passed unchanged to the client. Isn't that what I already wrote in comment #0?

metatron wrote:

Hmm, this need further knowledge on the subject.

  1. Lighttpd is configured without ssl engine, and makes rewrite only for tool's root

ssl.engine = "disable"
alias.url = ( "/$tool" => "$home/public_html/" )

  1. the requested url /is/ https and should be served as https. This is indsiputabled the job of webproxy, as lighty doesn't know anything about ssl.
  1. the debug output below shows 2 requests, https with and without trauling slash for static index.html in /newwebtest/test folder - they are both identical (as expected on lighty's side)

A. lighttp request handling , debug /without/ slash

https://tools.wmflabs.org/newwebtest/test

2014-05-31 19:37:21: (response.c.300) -- splitting Request-URI
2014-05-31 19:37:21: (response.c.301) Request-URI : /newwebtest/test/
2014-05-31 19:37:21: (response.c.302) URI-scheme : http
2014-05-31 19:37:21: (response.c.303) URI-authority: tools.wmflabs.org
2014-05-31 19:37:21: (response.c.304) URI-path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.305) URI-query :
2014-05-31 19:37:21: (response.c.349) -- sanatising URI
2014-05-31 19:37:21: (response.c.350) URI-path : /newwebtest/test/
2014-05-31 19:37:21: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:37:21: (response.c.470) -- before doc_root
2014-05-31 19:37:21: (response.c.471) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.472) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.473) Path :
2014-05-31 19:37:21: (response.c.521) -- after doc_root
2014-05-31 19:37:21: (response.c.522) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.523) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.524) Path : /data/project/newwebtest/public_html/newwebtest/test/
2014-05-31 19:37:21: (response.c.541) -- logical -> physical
2014-05-31 19:37:21: (response.c.542) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.543) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.544) Path : /data/project/newwebtest/public_htmltest/
2014-05-31 19:37:21: (response.c.561) -- handling physical path
2014-05-31 19:37:21: (response.c.562) Path : /data/project/newwebtest/public_html
test/
2014-05-31 19:37:21: (response.c.569) -- file found
2014-05-31 19:37:21: (response.c.570) Path : /data/project/newwebtest/public_htmltest/
2014-05-31 19:37:21: (response.c.719) -- handling subrequest
2014-05-31 19:37:21: (response.c.720) Path : /data/project/newwebtest/public_html
test/
2014-05-31 19:37:21: (mod_indexfile.c.150) -- handling the request as Indexfile
2014-05-31 19:37:21: (mod_indexfile.c.151) URI : /newwebtest/test/
2014-05-31 19:37:21: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:37:21: (mod_compress.c.683) -- handling file as static file
2014-05-31 19:37:21: (mod_staticfile.c.397) -- handling file as static file
2014-05-31 19:37:21: (response.c.731) -- subrequest finished

B. lighttp request handling , debug /with/ slash

https://tools.wmflabs.org/newwebtest/test/

2014-05-31 19:40:57: (response.c.300) -- splitting Request-URI
2014-05-31 19:40:57: (response.c.301) Request-URI : /newwebtest/test/
2014-05-31 19:40:57: (response.c.302) URI-scheme : http
2014-05-31 19:40:57: (response.c.303) URI-authority: tools.wmflabs.org
2014-05-31 19:40:57: (response.c.304) URI-path : /newwebtest/test/
2014-05-31 19:40:57: (response.c.305) URI-query :
2014-05-31 19:40:57: (response.c.349) -- sanatising URI
2014-05-31 19:40:57: (response.c.350) URI-path : /newwebtest/test/
2014-05-31 19:40:57: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:40:57: (response.c.470) -- before doc_root
2014-05-31 19:40:57: (response.c.471) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:40:57: (response.c.472) Rel-Path : /newwebtest/test/
2014-05-31 19:40:57: (response.c.473) Path :
2014-05-31 19:40:57: (response.c.521) -- after doc_root
2014-05-31 19:40:57: (response.c.522) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:40:57: (response.c.523) Rel-Path : /newwebtest/test/
2014-05-31 19:40:57: (response.c.524) Path : /data/project/newwebtest/public_html/newwebtest/test/
2014-05-31 19:40:57: (response.c.541) -- logical -> physical
2014-05-31 19:40:57: (response.c.542) Doc-Root : /data/project/newwebtest/public_html
2014-05-31 19:40:57: (response.c.543) Rel-Path : /newwebtest/test/
2014-05-31 19:40:57: (response.c.544) Path : /data/project/newwebtest/public_htmltest/
2014-05-31 19:40:57: (response.c.561) -- handling physical path
2014-05-31 19:40:57: (response.c.562) Path : /data/project/newwebtest/public_html
test/
2014-05-31 19:40:57: (response.c.569) -- file found
2014-05-31 19:40:57: (response.c.570) Path : /data/project/newwebtest/public_htmltest/
2014-05-31 19:40:57: (response.c.719) -- handling subrequest
2014-05-31 19:40:57: (response.c.720) Path : /data/project/newwebtest/public_html
test/
2014-05-31 19:40:57: (mod_indexfile.c.150) -- handling the request as Indexfile
2014-05-31 19:40:57: (mod_indexfile.c.151) URI : /newwebtest/test/
2014-05-31 19:40:57: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:40:57: (mod_compress.c.683) -- handling file as static file
2014-05-31 19:40:57: (mod_staticfile.c.397) -- handling file as static file
2014-05-31 19:40:57: (response.c.731) -- subrequest finished

(In reply to metatron from comment #8)

Hmm, this need further knowledge on the subject.

  1. Lighttpd is configured without ssl engine, and makes rewrite only for

tool's root

ssl.engine = "disable"
alias.url = ( "/$tool" => "$home/public_html/" )

[...]

No, it does not rewrite only tools' roots:

scfc@tools-login:~$ curl -H 'Host: tools.wmflabs.org' -I http://tools-webgrid-02:4113/drtrigonbot/docs
HTTP/1.1 301 Moved Permanently
Location: http://tools.wmflabs.org/drtrigonbot/docs/
Date: Mon, 02 Jun 2014 02:52:20 GMT
Server: lighttpd/1.4.28
scfc@tools-login:~$

As I have written a month ago. And reiterated the day before yesterday.

A. lighttp request handling , debug /without/ slash

  1. https://tools.wmflabs.org/newwebtest/test

2014-05-31 19:37:21: (response.c.300) -- splitting Request-URI
2014-05-31 19:37:21: (response.c.301) Request-URI : /newwebtest/test/
2014-05-31 19:37:21: (response.c.302) URI-scheme : http
2014-05-31 19:37:21: (response.c.303) URI-authority: tools.wmflabs.org
2014-05-31 19:37:21: (response.c.304) URI-path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.305) URI-query :
2014-05-31 19:37:21: (response.c.349) -- sanatising URI
2014-05-31 19:37:21: (response.c.350) URI-path : /newwebtest/test/
2014-05-31 19:37:21: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:37:21: (response.c.470) -- before doc_root
2014-05-31 19:37:21: (response.c.471) Doc-Root :
/data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.472) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.473) Path :
2014-05-31 19:37:21: (response.c.521) -- after doc_root
2014-05-31 19:37:21: (response.c.522) Doc-Root :
/data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.523) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.524) Path :
/data/project/newwebtest/public_html/newwebtest/test/
2014-05-31 19:37:21: (response.c.541) -- logical -> physical
2014-05-31 19:37:21: (response.c.542) Doc-Root :
/data/project/newwebtest/public_html
2014-05-31 19:37:21: (response.c.543) Rel-Path : /newwebtest/test/
2014-05-31 19:37:21: (response.c.544) Path :
/data/project/newwebtest/public_htmltest/
2014-05-31 19:37:21: (response.c.561) -- handling physical path
2014-05-31 19:37:21: (response.c.562) Path :
/data/project/newwebtest/public_html
test/
2014-05-31 19:37:21: (response.c.569) -- file found
2014-05-31 19:37:21: (response.c.570) Path :
/data/project/newwebtest/public_htmltest/
2014-05-31 19:37:21: (response.c.719) -- handling subrequest
2014-05-31 19:37:21: (response.c.720) Path :
/data/project/newwebtest/public_html
test/
2014-05-31 19:37:21: (mod_indexfile.c.150) -- handling the request as
Indexfile
2014-05-31 19:37:21: (mod_indexfile.c.151) URI : /newwebtest/test/
2014-05-31 19:37:21: (mod_access.c.135) -- mod_access_uri_handler called
2014-05-31 19:37:21: (mod_compress.c.683) -- handling file as static file
2014-05-31 19:37:21: (mod_staticfile.c.397) -- handling file as static file
2014-05-31 19:37:21: (response.c.731) -- subrequest finished

[...]

And this is just a lie. The request log for https://tools.wmflabs.org/newwebtest/test looks like:

2014-06-02 03:00:13: (response.c.300) -- splitting Request-URI
2014-06-02 03:00:13: (response.c.301) Request-URI : /newwebtest/test
2014-06-02 03:00:13: (response.c.302) URI-scheme : http
2014-06-02 03:00:13: (response.c.303) URI-authority: tools.wmflabs.org
2014-06-02 03:00:13: (response.c.304) URI-path : /newwebtest/test
2014-06-02 03:00:13: (response.c.305) URI-query :
2014-06-02 03:00:13: (response.c.349) -- sanatising URI
2014-06-02 03:00:13: (response.c.350) URI-path : /newwebtest/test
2014-06-02 03:00:13: (mod_access.c.135) -- mod_access_uri_handler called
2014-06-02 03:00:13: (response.c.470) -- before doc_root
2014-06-02 03:00:13: (response.c.471) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.472) Rel-Path : /newwebtest/test
2014-06-02 03:00:13: (response.c.473) Path :
2014-06-02 03:00:13: (response.c.521) -- after doc_root
2014-06-02 03:00:13: (response.c.522) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.523) Rel-Path : /newwebtest/test
2014-06-02 03:00:13: (response.c.524) Path : /data/project/newwebtest/public_html/newwebtest/test
2014-06-02 03:00:13: (response.c.541) -- logical -> physical
2014-06-02 03:00:13: (response.c.542) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.543) Rel-Path : /newwebtest/test
2014-06-02 03:00:13: (response.c.544) Path : /data/project/newwebtest/public_html//test
2014-06-02 03:00:13: (response.c.561) -- handling physical path
2014-06-02 03:00:13: (response.c.562) Path : /data/project/newwebtest/public_html//test
2014-06-02 03:00:13: (response.c.569) -- file found
2014-06-02 03:00:13: (response.c.570) Path : /data/project/newwebtest/public_html//test
2014-06-02 03:00:13: (response.c.300) -- splitting Request-URI
2014-06-02 03:00:13: (response.c.301) Request-URI : /newwebtest/test/
2014-06-02 03:00:13: (response.c.302) URI-scheme : http
2014-06-02 03:00:13: (response.c.303) URI-authority: tools.wmflabs.org
2014-06-02 03:00:13: (response.c.304) URI-path : /newwebtest/test/
2014-06-02 03:00:13: (response.c.305) URI-query :
2014-06-02 03:00:13: (response.c.349) -- sanatising URI
2014-06-02 03:00:13: (response.c.350) URI-path : /newwebtest/test/
2014-06-02 03:00:13: (mod_access.c.135) -- mod_access_uri_handler called
2014-06-02 03:00:13: (response.c.470) -- before doc_root
2014-06-02 03:00:13: (response.c.471) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.472) Rel-Path : /newwebtest/test/
2014-06-02 03:00:13: (response.c.473) Path :
2014-06-02 03:00:13: (response.c.521) -- after doc_root
2014-06-02 03:00:13: (response.c.522) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.523) Rel-Path : /newwebtest/test/
2014-06-02 03:00:13: (response.c.524) Path : /data/project/newwebtest/public_html/newwebtest/test/
2014-06-02 03:00:13: (response.c.541) -- logical -> physical
2014-06-02 03:00:13: (response.c.542) Doc-Root : /data/project/newwebtest/public_html
2014-06-02 03:00:13: (response.c.543) Rel-Path : /newwebtest/test/
2014-06-02 03:00:13: (response.c.544) Path : /data/project/newwebtest/public_html//test/
2014-06-02 03:00:13: (response.c.561) -- handling physical path
2014-06-02 03:00:13: (response.c.562) Path : /data/project/newwebtest/public_html//test/
2014-06-02 03:00:13: (response.c.569) -- file found
2014-06-02 03:00:13: (response.c.570) Path : /data/project/newwebtest/public_html//test/
2014-06-02 03:00:13: (response.c.719) -- handling subrequest
2014-06-02 03:00:13: (response.c.720) Path : /data/project/newwebtest/public_html//test/
2014-06-02 03:00:13: (mod_indexfile.c.150) -- handling the request as Indexfile
2014-06-02 03:00:13: (mod_indexfile.c.151) URI : /newwebtest/test/
2014-06-02 03:00:13: (mod_access.c.135) -- mod_access_uri_handler called
2014-06-02 03:00:13: (mod_compress.c.683) -- handling file as static file
2014-06-02 03:00:13: (mod_staticfile.c.397) -- handling file as static file
2014-06-02 03:00:13: (response.c.731) -- subrequest finished

That clearly shows that there is an initial request that causes a redirect (by lighttpd) to be returned to the browser followed by another request with the correct URL.

I haven't receive any answer to http://serverfault.com/questions/602386/how-to-make-lighttpd-respect-x-forwarded-proto-when-constructing-redirects-for-d in the past two weeks; I have posted on the lighttpd support forum now as well at http://redmine.lighttpd.net/boards/2/topics/6028.

Solving this in nginx looks doable, but I'd prefer if we can fix it in lighttpd.

This is no longer an issue since the redirection is handled by the proxy, now.

No, it is not:

[tim@passepartout ~]$ curl -I https://tools.wmflabs.org/typoscan
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Wed, 25 Mar 2015 19:26:45 GMT
Connection: keep-alive
Location: http://tools.wmflabs.org/typoscan/
X-Clacks-Overhead: GNU Terry Pratchett

[tim@passepartout ~]$ curl -I https://tools.wmflabs.org/typoscan/test-for-T66627
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Wed, 25 Mar 2015 19:26:49 GMT
Connection: keep-alive
Location: http://tools.wmflabs.org/typoscan/test-for-T66627/
X-Clacks-Overhead: GNU Terry Pratchett

[tim@passepartout ~]$

Ah, hm. My understanding from the config was that this was (supposed to) be done by the proxy. @yuvipanda: is that a bug or expected behaviour?

The proxy should do it, but it doesn't atm and I haven't had time to do it properly yet.

Although, note that the new uwsgi services don't have this problem...

Scratch that, uwsgi also has this problem. It 'worked for me' because of httpseverywhere...

coren removed coren as the assignee of this task.Mar 25 2015, 7:37 PM
coren edited projects, added Cloud-Services; removed Toolforge.
coren set Security to None.

I had intended to fix this in nginx; as I'll be diving into that for T88216 anyhow, licking the cookie.

Change 206488 had a related patch set uploaded (by Yuvipanda):
tools: Faux enable https for lighttpd by default

https://gerrit.wikimedia.org/r/206488

Change 206488 merged by Yuvipanda:
tools: Faux enable https for lighttpd by default

https://gerrit.wikimedia.org/r/206488

After some false starts, looks like http://redmine.lighttpd.net/projects/1/wiki/Docs_ModMagnet might be the way to fix this. but that embeds lua in lighttpd as well, and I"m feeling a bit 'eugh' about that.

Looks like the only solution is to write a small lighttpd module.

Change 206504 had a related patch set uploaded (by Yuvipanda):
tools: Redirect tools.wmflabs.org/toolname appropriately

https://gerrit.wikimedia.org/r/206504

Change 206504 merged by Yuvipanda:
tools: Redirect tools.wmflabs.org/toolname appropriately

https://gerrit.wikimedia.org/r/206504

ok, so ^ does it at the nginx level, which only handles the tool level redirect.

Nope, it doesn't. I give up. Fuck lighttpd.

Change 206519 had a related patch set uploaded (by Tim Landscheidt):
Tools: Fix redirects from https to http

https://gerrit.wikimedia.org/r/206519

Ah, sorry, I forgot to jot down how I intended it to fix.

scfc triaged this task as Medium priority.Apr 25 2015, 4:26 PM
scfc edited projects, added Toolforge; removed Cloud-Services.
scfc moved this task from Ready to be worked on to Waiting for code review on the Toolforge board.

Change 206519 merged by Yuvipanda:
Tools: Fix redirects from https to http

https://gerrit.wikimedia.org/r/206519

@scfc <3 I did not know about proxy_redirect and did not look in that direection at all. Thank you very much :D Seems to work fine now both for redirects, non-redirects and root elements.