Page MenuHomePhabricator
Create Task
Maniphest T213509

doc.wikimedia.org urls without trailing slash respond with HTTP 403 Forbidden
Closed, ResolvedPublic
Actions

Description

Urls like https://doc.wikimedia.org/cover/visualeditor/ fail if navigated to without a trailing slash, like https://doc.wikimedia.org/cover/visualeditor.

This looks like it might be a configuration error on the doc.wm.o Apache, or missing a common default we normally enable on static servers.

This is particularly common to see because our publish jobs produce a clickable link without trailing slash.

E.g. at https://integration.wikimedia.org/ci/job/publish-to-doc1001/1810/console

+ rsync --archive --compress --delete-after . rsync://doc1001.eqiad.wmnet/doc/cover/visualeditor
Published at https://doc.wikimedia.org/cover/visualeditor
Done.

That could be fixed separately, but it'd be nice for general address-bar usability if things worked as expected when typing /cover or /cover/something

Related Objects

Event Timeline

Krinkle created this task.Jan 11 2019, 2:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2019, 2:15 AM
Krinkle updated the task description. (Show Details)Jan 11 2019, 2:15 AM
hashar added a comment.Jan 11 2019, 7:53 AM

My fault, previously we normalized the IP address with a trailing slash:

doc-publish macro:

echo "Publishing to https://doc.wikimedia.org/{docdest}/"

publish-on-contint1001 job:

echo "Published to https://$LOCAL_VHOST/$WMF_CI_PUB_DEST/"

The new job does not add the trailing slash:

DOC_URL="https://doc.wikimedia.org/${WMF_CI_PUB_DEST}"
echo -e "\nPublished at ${DOC_URL}\nDone."
gerritbot added a subscriber: gerritbot.Jan 11 2019, 7:53 AM

Change 483682 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Adding trailling slash to doc publishing URL

https://gerrit.wikimedia.org/r/483682

hashar added a comment.Jan 11 2019, 7:54 AM

The previous and current Apache configs for doc.wikimedia.org are similar and both have:

# T95164
DirectorySlash Off
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.+[^/])$ https://doc.wikimedia.org/$1/ [R=301,QSA]

T95164: when reaching /cover, Apace would forge a redirect with HTTP instead of HTTPS. The rewrite rule should be taking care of it.

$ curl https://doc.wikimedia.org/cover/visualeditor
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /cover/visualeditor
on this server.<br />
</p>
</body></html>

Gotta find out what is wrong in the rewrite condition / rule :(

hashar claimed this task.Jan 11 2019, 7:54 AM
hashar added a comment.Jan 11 2019, 8:29 AM

Testing on contint1001 which has Apache 2.4:

curl -H 'Host: doc.wikimedia.org' http://208.80.154.17/cover/visualeditor

AH01276: Cannot serve directory /srv/org/wikimedia/doc/cover/visualeditor:
    No matching DirectoryIndex (none) found, and server-generated directory index forbidden by Options directive

I would guess mod_dir kicks in before mod_rewrite had a chance to rewrite the request :-/ A couple potential solutions to investigate:

DirectoryCheckHandler Off to disable mod_dir if another module handles the URL.

RewriteOptions AllowNoSlash

By default, mod_rewrite will ignore URLs that map to a directory on disk but lack a trailing slash, in the expectation that the mod_dir module will issue the client with a redirect to the canonical URL with a trailing slash.
When the DirectorySlash directive is set to off, the AllowNoSlash option can be enabled to ensure that rewrite rules are no longer ignored. This option makes it possible to apply rewrite rules within .htaccess files that match the directory without a trailing slash, if so desired.

Since we have DirectorySlash Off, I guess we need to add RewriteOptions AllowNoSlash so that mod_rewrite stop ignoring those requests.

hashar added a comment.Jan 11 2019, 3:04 PM

So this one took me a while to figure out. I tested it on contint1001 which still has the configuration:

$ curl -H 'Host: doc.wikimedia.org' http://208.80.154.17/cover/visualeditor
<title>403 Forbidden</title>
<p>You don't have permission to access /cover/visualeditor on this server.<br />

Thus it was broken before we migrated to doc1001.eqiad.wmnet so that is not a regression.

The rewrite of requests lacking a trialling slash is handled by mod_dir DirectorySlashes On (the default). However on our setup that causes the redirect to use a http URL (that is T95164).

07c7f447ad3bb60940d812219411640ae696e95b disables DirectorySlashes and reimplements it with a rewrite rule. Unfortunately its condition was wrong, it tests the existence of a file that is literally /cover/visualeditor which does not exist. Hence the rewrite and redirection is never done.

The reason is that at this stage of the processing REQUEST_FILENAME is set to the URI path. We have to force the filesystem resolution which then fix the rewrite condition.

gerritbot added a comment.Jan 11 2019, 3:05 PM

Change 483775 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: fix redirect of dir lacking a trailing slash

https://gerrit.wikimedia.org/r/483775

hashar added a comment.Jan 11 2019, 3:24 PM

And in the end the TLDR is to use:

ServerName https://doc.wikimedia.org
DirectorySlash On

:)

gerritbot added a comment.Jan 14 2019, 11:35 PM

Change 483775 merged by Dzahn:
[operations/puppet@production] doc: fix Apache redirects to use https

https://gerrit.wikimedia.org/r/483775

Dzahn added a subscriber: Dzahn.Jan 14 2019, 11:51 PM

Urls like https://doc.wikimedia.org/cover/visualeditor/ fail if navigated to without a trailing slash, like https://doc.wikimedia.org/cover/visualeditor.

https://doc.wikimedia.org/cover/visualeditor works now just like https://doc.wikimedia.org/cover/visualeditor/

[deploy1001:~] $ apache-fast-test doc.url doc1001.eqiad.wmnet
testing 6 urls on 1 servers, totalling 6 requests
spawning threads..

http://doc.wikimedia.org
 * 200 OK 7336
http://doc.wikimedia.org/cover/visualeditor
 * 403 Forbidden
http://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836
https://doc.wikimedia.org
 * 200 OK 7336
https://doc.wikimedia.org/cover/visualeditor
 * 403 Forbidden
https://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836


after:

[deploy1001:~] $ apache-fast-test doc.url doc1001.eqiad.wmnet
testing 6 urls on 1 servers, totalling 6 requests
spawning threads..

http://doc.wikimedia.org
 * 200 OK 7336
http://doc.wikimedia.org/cover/visualeditor
 * 301 Moved Permanently https://doc.wikimedia.org/cover/visualeditor/
http://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836
https://doc.wikimedia.org
 * 200 OK 7336
https://doc.wikimedia.org/cover/visualeditor
 * 301 Moved Permanently https://doc.wikimedia.org/cover/visualeditor/
https://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836
Dzahn closed this task as Resolved.Jan 14 2019, 11:52 PM
hashar added a comment.Jan 15 2019, 8:30 AM

Amazing, thank you very much for the apache-fast-test test!

gerritbot added a comment.May 6 2019, 10:46 AM

Change 483682 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Adding trailling slash to doc publishing URL

https://gerrit.wikimedia.org/r/483682

gerritbot added a comment.May 6 2019, 10:49 AM

Change 483682 merged by jenkins-bot:
[integration/config@master] Adding trailling slash to doc publishing URL

https://gerrit.wikimedia.org/r/483682

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL