Page MenuHomePhabricator

If-Modified-Since handling is broken
Closed, ResolvedPublic

Description

We (crawling team in Google) found that Wikipedia's If-Modified-Since handling is broken, at least when it comes to (Wikipedia-style) redirects/symlinks.

As a short-term work-around (in order to serve up-to-date content), we temporarily stopped sending out If-Modified-Since header in the crawl request.

Hope you take a look at this issue, and let us know when it is resolved.

Below is what we did to reproduce the issue (Actually, we have observed many times that we cannot fetch the latest content of an articles before)

  1. Pick a Wikipedia page to vandalize it (apologies for being a vandal; we promise it is for the greater good...), and wait for it is reverted.

http://en.wikipedia.org/wiki/SSh (note the capitalization). Its history (http://en.wikipedia.org/w/index.php?title=SSh&action=history) will contains the vandalization at 01:47:42 (GMT) and its rollback at 01:49:06 (GMT)

  1. Fetch the url (at 01:47:54 GMT, right after the vandalized revision is submitted) using telnet.

It seems the "Last-Modified" is using the last redirect destination's latest revision: 08 Jul 2014 11:49:08 GMT (http://en.wikipedia.org/w/index.php?title=SSL&action=history)

$ telnet en.wikipedia.org 80
Trying 2620:0:861:ed1a::1...
Connected to text-lb.eqiad.wikimedia.org.
Escape character is '^]'.
GET /wiki/SSh HTTP/1.1
Host: en.wikipedia.org

HTTP/1.1 200 OK
Server: Apache
X-Content-Type-Options: nosniff
Content-language: en
X-UA-Compatible: IE=Edge
Vary: Accept-Encoding,Cookie
Last-Modified: Tue, 08 Jul 2014 11:49:08 GMT
Content-Type: text/html; charset=UTF-8
X-Varnish: 4282068392, 1036646122
Via: 1.1 varnish, 1.1 varnish
Transfer-Encoding: chunked
Date: Thu, 10 Jul 2014 01:47:54 GMT ← This one's accurate; this is when the crawl happened.
Age: 0
Connection: keep-alive
X-Cache: cp1065 miss (0), cp1053 frontend miss (0)
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Set-Cookie: GeoIP=::::v6; Path=/; Domain=.wikipedia.org

…Page for SSL follows…

  1. After it's reverted, re-crawl (at 01:50:57 GMT) the article with a If-Modified-Since header, setting to the Last-Modified value in step2 (we are simulating the production crawling works).

It's strange that we got a 304 response this time. And the "Last-Modified" value is weird too, neither SSh nor SSH has update around 08 Jul 2014 11:49:08 GMT.

$ telnet en.wikipedia.org 80
Trying 2620:0:861:ed1a::1...
Connected to text-lb.eqiad.wikimedia.org.
Escape character is '^]'.
GET /wiki/SSh HTTP/1.1
Host: en.wikipedia.org
If-Modified-Since: Tue, 08 Jul 2014 11:49:08 GMT

HTTP/1.1 304 Not Modified
Server: Apache
X-Content-Type-Options: nosniff
Content-language: en
X-UA-Compatible: IE=Edge
Vary: Accept-Encoding,Cookie
Last-Modified: Thu, 26 Jun 2014 11:22:21 GMT ← A mysterious timestamp.
Content-Type: text/html; charset=UTF-8
X-Varnish: 4282315466 4282211494, 2309645493
Via: 1.1 varnish, 1.1 varnish
Date: Thu, 10 Jul 2014 01:50:57 GMT
Age: 74
Connection: keep-alive
X-Cache: cp1065 hit (1), cp1066 frontend miss (0)
Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
Set-Cookie: GeoIP=::::v6; Path=/; Domain=.wikipedia.org

Connection closed by foreign host.


Version: unspecified
Severity: major

Details

Reference
bz67849

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:38 AM
bzimport added a project: MediaWiki-General.
bzimport set Reference to bz67849.
bzimport added a subscriber: Unknown Object (MLST).

Thanks for reporting this!
Wondering who could investigate here. Greg?

The cause of the problem is simple enough: when MediaWiki follows a redirect, it uses the page_touched timestamp of the target page.[1]

A simple change to fix this bug would be to adjust Article::view() to use the maximum of the target page's page_touched or the redirect's page_touched when calling OutputPage::checkLastModified().

[1]: @Bianjiang: MediaWiki stores a "touched" timestamp for each page, which is updated when the page is edited, when a template transcluded on the page is edited, and in other cases too. The current page_touched timestamp for the SSH article is 2014-06-26T11:22:21Z,[2] which matches what you're seeing.
[2]: https://en.wikipedia.org/w/api.php?format=jsonfm&action=query&prop=info&titles=SSH

Change 145568 had a related patch set uploaded by Anomie:
Take redirect modifications into account for If-Modified-Since

https://gerrit.wikimedia.org/r/145568

Change 145568 merged by jenkins-bot:
Take redirect modifications into account for If-Modified-Since

https://gerrit.wikimedia.org/r/145568

@Bianjiang: You should be able to test this change in a few minutes on Beta Labs, see http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page for details on that site.

If nothing else happens, this change will go out to the non-Wikipedia production wikis on July 22 and to the Wikipedias on July 24. We could push it out on Monday, though, especially if testing on Beta Labs over the weekend indicates that all's well.

@Brad, it seems I cannot test the it on deployment.wikimedia.beta.wmflabs.org, because i have no way to "revert" a revision (I just registered a new account there: bianjiang, but has no privilege to revert - undo seems always generate a new revision instead of "reverting" to an old one.

(In reply to bianjiang from comment #6)

undo seems always generate a new revision instead of
"reverting" to an old one.

That's how undo and other methods of reverting work in MediaWiki. The only way to actually "revert" as you're thinking is to delete the page entirely and then restore all but the most recent revision, which is not particularly encouraged.