doc.wikimedia.org urls without trailing slash respond with HTTP 403 Forbidden
Closed, ResolvedPublic
Actions

Description

Urls like https://doc.wikimedia.org/cover/visualeditor/ fail if navigated to without a trailing slash, like https://doc.wikimedia.org/cover/visualeditor.

This looks like it might be a configuration error on the doc.wm.o Apache, or missing a common default we normally enable on static servers.

This is particularly common to see because our publish jobs produce a clickable link without trailing slash.

E.g. at https://integration.wikimedia.org/ci/job/publish-to-doc1001/1810/console

+ rsync --archive --compress --delete-after . rsync://doc1001.eqiad.wmnet/doc/cover/visualeditor

Published at https://doc.wikimedia.org/cover/visualeditor
Done.

That could be fixed separately, but it'd be nice for general address-bar usability if things worked as expected when typing /cover or /cover/something

Related Objects

Mentioned Here: rOPUP07c7f447ad3b: doc.wikimedia.org: fix DirectorySlash https->http
T95164: Apache slash expansion should not redirect from HTTPS to HTTP

Event Timeline

Krinkle created this task.Jan 11 2019, 2:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2019, 2:15 AM

Krinkle updated the task description. (Show Details)Jan 11 2019, 2:15 AM

My fault, previously we normalized the IP address with a trailing slash:

doc-publish macro:

echo "Publishing to https://doc.wikimedia.org/{docdest}/"

publish-on-contint1001 job:

echo "Published to https://$LOCAL_VHOST/$WMF_CI_PUB_DEST/"

The new job does not add the trailing slash:

DOC_URL="https://doc.wikimedia.org/${WMF_CI_PUB_DEST}"
echo -e "\nPublished at ${DOC_URL}\nDone."

Change 483682 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Adding trailling slash to doc publishing URL

https://gerrit.wikimedia.org/r/483682

gerritbot added a project: Patch-For-Review.Jan 11 2019, 7:53 AM

The previous and current Apache configs for doc.wikimedia.org are similar and both have:

# T95164
DirectorySlash Off
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.+[^/])$ https://doc.wikimedia.org/$1/ [R=301,QSA]

T95164: when reaching /cover, Apace would forge a redirect with HTTP instead of HTTPS. The rewrite rule should be taking care of it.

$ curl https://doc.wikimedia.org/cover/visualeditor
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /cover/visualeditor
on this server.<br />
</p>
</body></html>

Gotta find out what is wrong in the rewrite condition / rule :(

hashar claimed this task.Jan 11 2019, 7:54 AM

Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team. · View Herald TranscriptJan 11 2019, 7:54 AM

Testing on contint1001 which has Apache 2.4:

curl -H 'Host: doc.wikimedia.org' http://208.80.154.17/cover/visualeditor

AH01276: Cannot serve directory /srv/org/wikimedia/doc/cover/visualeditor:
    No matching DirectoryIndex (none) found, and server-generated directory index forbidden by Options directive

I would guess mod_dir kicks in before mod_rewrite had a chance to rewrite the request :-/ A couple potential solutions to investigate:

DirectoryCheckHandler Off to disable mod_dir if another module handles the URL.

RewriteOptions AllowNoSlash

By default, mod_rewrite will ignore URLs that map to a directory on disk but lack a trailing slash, in the expectation that the mod_dir module will issue the client with a redirect to the canonical URL with a trailing slash.
When the DirectorySlash directive is set to off, the AllowNoSlash option can be enabled to ensure that rewrite rules are no longer ignored. This option makes it possible to apply rewrite rules within .htaccess files that match the directory without a trailing slash, if so desired.

Since we have DirectorySlash Off, I guess we need to add RewriteOptions AllowNoSlash so that mod_rewrite stop ignoring those requests.

So this one took me a while to figure out. I tested it on contint1001 which still has the configuration:

$ curl -H 'Host: doc.wikimedia.org' http://208.80.154.17/cover/visualeditor
<title>403 Forbidden</title>
<p>You don't have permission to access /cover/visualeditor on this server.<br />

Thus it was broken before we migrated to doc1001.eqiad.wmnet so that is not a regression.

The rewrite of requests lacking a trialling slash is handled by mod_dir DirectorySlashes On (the default). However on our setup that causes the redirect to use a http URL (that is T95164).

07c7f447ad3bb60940d812219411640ae696e95b disables DirectorySlashes and reimplements it with a rewrite rule. Unfortunately its condition was wrong, it tests the existence of a file that is literally /cover/visualeditor which does not exist. Hence the rewrite and redirection is never done.

The reason is that at this stage of the processing REQUEST_FILENAME is set to the URI path. We have to force the filesystem resolution which then fix the rewrite condition.

Change 483775 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] doc: fix redirect of dir lacking a trailing slash

https://gerrit.wikimedia.org/r/483775

And in the end the TLDR is to use:

ServerName https://doc.wikimedia.org
DirectorySlash On

Change 483775 merged by Dzahn:
[operations/puppet@production] doc: fix Apache redirects to use https

https://gerrit.wikimedia.org/r/483775

Urls like https://doc.wikimedia.org/cover/visualeditor/ fail if navigated to without a trailing slash, like https://doc.wikimedia.org/cover/visualeditor.

https://doc.wikimedia.org/cover/visualeditor works now just like https://doc.wikimedia.org/cover/visualeditor/

Added firewall hole to allow testing with apache-fast-test from deployment servers. (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/484317/)
Manually applied Hashar's change on doc1001.eqiad.wmnet
Tested with apache-fast-test:
Re-enabled puppet, deployed on both doc1001 and contint1001
created patch to delete unused config from contint1001 (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/484321/)

[deploy1001:~] $ apache-fast-test doc.url doc1001.eqiad.wmnet
testing 6 urls on 1 servers, totalling 6 requests
spawning threads..

http://doc.wikimedia.org
 * 200 OK 7336
http://doc.wikimedia.org/cover/visualeditor
 * 403 Forbidden
http://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836
https://doc.wikimedia.org
 * 200 OK 7336
https://doc.wikimedia.org/cover/visualeditor
 * 403 Forbidden
https://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836


after:

[deploy1001:~] $ apache-fast-test doc.url doc1001.eqiad.wmnet
testing 6 urls on 1 servers, totalling 6 requests
spawning threads..

http://doc.wikimedia.org
 * 200 OK 7336
http://doc.wikimedia.org/cover/visualeditor
 * 301 Moved Permanently https://doc.wikimedia.org/cover/visualeditor/
http://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836
https://doc.wikimedia.org
 * 200 OK 7336
https://doc.wikimedia.org/cover/visualeditor
 * 301 Moved Permanently https://doc.wikimedia.org/cover/visualeditor/
https://doc.wikimedia.org/cover/visualeditor/
 * 200 OK 22836

Dzahn closed this task as Resolved.Jan 14 2019, 11:52 PM

Amazing, thank you very much for the apache-fast-test test!

doc.wikimedia.org urls without trailing slash respond with HTTP 403 ForbiddenClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

doc.wikimedia.org urls without trailing slash respond with HTTP 403 Forbidden
Closed, ResolvedPublic
Actions