Page MenuHomePhabricator

Multiple users reporting content pages displaying "NULL" compared to desired content
Closed, ResolvedPublic

Description

<ankry> Hmm, what does it mean:
<ankry> https://pl.wikisource.org/wiki/Dyskusja_wikiskryby:Nawider
<ankry> I got NULL instead of page content
<nadando> Does anyone know why this page just says NULL https://en.wiktionary.org/wiki/Wiktionary:Tea_room/2016/March
<p858snake> nadando / ankry: are you currently logged in?
<nadando> p858snake, yes
<Waggie> p858snake, we just had another user report the same thing on https://en.wikipedia.org/wiki/Australian_rules_football
<Waggie> (over on -en-help)
<p858snake> try doing a force refreshed (most browsers: Ctrl+F5
<Waggie> They've left, so I can't inquire further.
<p858snake> *forced refresh
<nadando> That page says NULL too, same if I log out and force refresh
<p858snake> Can you take a screenshot please?
<Waggie> I get NULL also, Chrome, logged in, on Linux. Will take screenshot momentarily.
<nadando> http://i.imgur.com/G4HwR1O.png
<Waggie> http://imgur.com/mfjq6sr
<Waggie> Also, page source consists only of "NULL". No html, or anything.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2016, 6:10 AM
Peachey88 updated the task description. (Show Details)Mar 22 2016, 6:13 AM

Somehow "NULL" got cached in varnish.

legoktm@terbium:~$ echo "https://commons.wikimedia.org/wiki/Special:RecentChanges" | mwscript purgeList.php --wiki=commonswiki
Purging 1 urls
Done!

And the url works now.

If you ?action=purge on the affected articles, they should be fixed. We'll (probably not me though) figure out some way to purge everything with the content of "NULL".

Vort added a subscriber: Vort.Mar 22 2016, 6:49 AM

In ru-wiki, this problem hit the Special:Watchlist page.
So I think this bug needs a critical priority.

Mentioned in SAL [2016-03-22T06:54:27Z] <_joe_> banning all pages with content-length of 25 from the caches, T130575

Joe triaged this task as Unbreak Now! priority.Mar 22 2016, 6:56 AM
Joe added a subscriber: Joe.Mar 22 2016, 6:59 AM

My hackish ban should've removed all the current pages with NULL content (at least if they're gzipped).

The fact remains we have no idea what can have caused this.

Joe added a comment.Mar 22 2016, 7:00 AM

Please let me know if you see this again, I'm not confident at all this was a "permanent" fix (which would mean the error we've seen was transient)

Joe added a comment.Mar 22 2016, 7:04 AM

@John_of_Reading I think my ban of the cached content should've stopped this from happening now, so let's see if more reports come in from now on.

Joe lowered the priority of this task from Unbreak Now! to High.Mar 22 2016, 7:45 AM

I tested one of the affected urls against all appservers, and they're all responding correctly now.

Change 278849 had a related patch set uploaded (by Giuseppe Lavagetto):
conftoool: remove the debug appservers from the pool

https://gerrit.wikimedia.org/r/278849

Joe added a comment.Mar 22 2016, 9:43 AM

I just found out that three out of four of the application servers dedicated to debugging were still pooled to serve traffic. If someone was performing some tests on those, the assumption that tests would not impact production would be voided.

This hypothesis came to my mind as @Legoktm suggested on irc

< legoktm> NULL in all uppercase suggests a "var_dump(null)" or something

earlier this morning. It's completely unverified, but I'm fixing the potential issue anyways as this could be a real-world scenario.

Change 278849 merged by Giuseppe Lavagetto:
conftoool: remove the debug appservers from the pool

https://gerrit.wikimedia.org/r/278849

Joe added a comment.Mar 22 2016, 9:47 AM

I am also resolving the ticket as it seems like no more incidents have been reported

Joe closed this task as Resolved.Mar 22 2016, 9:48 AM
Joe claimed this task.