Page MenuHomePhabricator

Purge pages cached with mobile editlinks
Closed, ResolvedPublic

Description

Pages cached with mobile editlinks should be purged so it would be possible to find out if new are popping up again (aka the parent problem (T124356: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() )) persist) or not.

Event Timeline

Anomie removed a subscriber: Anomie.Feb 4 2016, 5:21 PM
BBlack added a comment.Feb 5 2016, 4:08 AM

Repeating from the other ticket: Can we get some more-specific details about the unique attributes of these pages (please tell me it's something other than the HTML contents)? Even possibly an exact-ish timestamp range of when the pages could've been output by MW? I have to have some parameters to purge by...

Repeating from the other ticket: Can we get some more-specific details about the unique attributes of these pages (please tell me it's something other than the HTML contents)? Even possibly an exact-ish timestamp range of when the pages could've been output by MW? I have to have some parameters to purge by...

I can't think about any. Last edit dates vary in the range of several years. There is no specific mobile web/mobile edit tag (or any other) on last edit.
Maybe the date of first wmf10 deployment to such wiki, as it seems that the cause was there? (But then you would have to purge all pages since then, which I guess is more expensive than search by text.)
Maybe others will find any other criteria, but I'm afraid this is it...

@BBlack: What's the status, please?

@Danny_B - we can't purge caches on response body content matching, that's why I'm asking about at least a Date range if we have no other metadata to go on. By now, I'd suspect this is affecting only a very small fraction of responses due to natural rollover of caches (from edit traffic and natural LRU fallout), and we could without much impact wipe everything emitted before the Date that the code/parsercache was known to be emitting good content again.

(To be clear in reference to earlier comments - last edit date doesn't matter, just the date range in which the MediaWiki servers were emitting bad content, due to the code that was running and/or the still-corrupted parsercache).

Probably should've put this here instead of in the parent ticket:

I've executed bans for date matching anything up through (and including) Feb 5th now, with ban expression `obj.http.Date ~ " (([1-5] Feb| Jan) 2016|2015) "`.  Since this cutoff is now nearly two weeks in the past, the impact should be minimal.
BBlack closed this task as Resolved.Feb 18 2016, 12:57 AM
BBlack claimed this task.
Danny_B reopened this task as Open.Feb 19 2016, 9:53 AM

I've just hit another page: https://sk.wiktionary.org/wiki/duplikova%C5%A5 thus reopening.

I've just tried to reproduce this on all text caches and can't, so I don't see how anyone else could either...

salt -v -t 15 -b 1000 --out=raw -G cluster:cache_text cmd.run 'curl -s -H "Host: sk.wiktionary.org" -H "X-Forwarded-Proto: https" "http://localhost/wiki/duplikova%C5%A5"|egrep "title>|/editor/"'

Executing run on ['cp3030.esams.wmnet', 'cp1054.eqiad.wmnet', 'cp4009.ulsfo.wmnet', 'cp3041.esams.wmnet', 'cp3013.esams.wmnet', 'cp1052.eqiad.wmnet', 'cp2004.codfw.wmnet', 'cp2001.codfw.wmnet', 'cp3010.esams.wmnet', 'cp3006.esams.wmnet', 'cp3012.esams.wmnet', 'cp2023.codfw.wmnet', 'cp1067.eqiad.wmnet', 'cp3008.esams.wmnet', 'cp1055.eqiad.wmnet', 'cp4016.ulsfo.wmnet', 'cp3005.esams.wmnet', 'cp4018.ulsfo.wmnet', 'cp1066.eqiad.wmnet', 'cp4010.ulsfo.wmnet', 'cp3007.esams.wmnet', 'cp1008.wikimedia.org', 'cp2019.codfw.wmnet', 'cp4017.ulsfo.wmnet', 'cp4008.ulsfo.wmnet', 'cp3004.esams.wmnet', 'cp2007.codfw.wmnet', 'cp1068.eqiad.wmnet', 'cp3031.esams.wmnet', 'cp2010.codfw.wmnet', 'cp3040.esams.wmnet', 'cp3003.esams.wmnet', 'cp1053.eqiad.wmnet', 'cp1065.eqiad.wmnet', 'cp2013.codfw.wmnet', 'cp3009.esams.wmnet', 'cp2016.codfw.wmnet', 'cp3014.esams.wmnet']

{'cp4017.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2019.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1008.wikimedia.org': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3007.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp4010.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1066.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3003.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp4018.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3005.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp4016.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3040.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1055.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2016.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3009.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3008.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2013.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1067.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2023.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1065.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1053.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3006.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3012.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3010.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2001.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2004.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2010.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3031.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1068.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp2007.codfw.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3004.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3014.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1052.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp4008.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3013.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3041.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp4009.ulsfo.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp1054.eqiad.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}
{'cp3030.esams.wmnet': '<title>duplikova\xc5\xa5 - Wikislovn\xc3\xadk</title>'}

@BBlack When you open the link I provided (don't purge it!), can you see those wrong editlinks or not?

"wgHostname":"mw1246" if that helps

@Danny_B - I don't think I see "wrong" editlinks, although I can't say I know for sure what we're looking for visually. In any case, you and I are likely to hit two entirely different sets of caches for our browser-based requests, so when something bad might be lingering in a subset of caches, it's never going to reliably reproduce between two different users' browsers.

But what I can do, which I've done above, is run a commandline query that pulls the content for that article from all of the relevant caches and greps their outputs. If any user is affected by a bad cache entry, it would show up for at least one of the caches I'm querying.

What I'm basing my test on is the technical description at the top of T124356 (where it shows examples of the bad output and the expected output for the headline), and also the parsercache reject hook which seems to key on the same distinction here: https://gerrit.wikimedia.org/r/#/c/266399/2/includes/MobileFrontend.hooks.php .

What my earlier command shows is that none of the caches are emitting /editor/ anywhere in their HTML output on requests for this article. To double-check, I just re-ran the command from earlier but grepping for mw-headline instead, and they again all had identical output in that regard, which was 4x different mw-headline lines that are all of the correct form with links like: <a href="/w/index.php?title=duplikova%C5%A5&amp;action=edit&amp;section=1" ... and not containing /editor/.

Assuming I haven't missed something dumb above, the next question is: are you logged in when you make your request? If you're logged into a wiki account, you never get Varnish-cached article HTML. Therefore (a) you seeing bad editlinks means MW is still emitting them, at least sometimes from some servers, (which sucks, that means my purges were pointless, and someone needs to validate that T124356 is actually fixed and close it before coming back here about Varnish purges) and (b) my tests are clean because the shared anonymous caches don't happen to have bad copies when I run my tests (by lucky chance of which MW server they've hit).

It would be helpful to get a full dump of the request/response headers and at least confirm that it's not a cache hit (be sure to wipe out personal information like session cookie values and your IP, etc).

@BBlack: regardless logged in or anonymous, both are


Other replies will follow...

Jay8g added a subscriber: Jay8g.Feb 20 2016, 6:33 AM

I see it looking wrong, but acting correct.
Source:

<h2><span class="mw-headline" id="Extern.C3.A9_odkazy">Externé odkazy</span><span><a href="/w/index.php?title=duplikova%C5%A5&amp;action=edit&amp;section=4" title="Upraviť sekciu: Externé odkazy" data-section="4" class="mw-ui-icon mw-ui-icon-element mw-ui-icon-edit-enabled edit-page">Upraviť</a></span></h2>
BBlack added a comment.EditedFeb 20 2016, 3:05 PM

@Danny_B and I compared notes on IRC a bit, and yeah there's obviously some confusion here. The #/editor/ stuff is gone (at least, in this particular case we're looking at), he contends that the mw-headline currently shown (what @Jay8g quotes above) is still "wrong", and should look more like:

<h2><span class="mw-headline" id="Sloven.C4.8Dina">Slovenčina</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a
                href="/w/index.php?title=golier&amp;action=edit&amp;section=1" title="Upraviť sekciu: Slovenčina">upraviť</a><span class="mw-editsection-bracket">]</span></span></h2>

(note the added square brackets and span class mw-editsection-bracket)

Either this is an entirely separate problem/regression, or the description of the problem in T124356 and the parsercache hook to fix it in https://gerrit.wikimedia.org/r/#/c/266399/2/includes/MobileFrontend.hooks.php are incomplete/wrong. Either way, I also get the "bad" results (no #/editor/, but no editsection brackets either) when querying MediaWiki servers directly (no varnish caches involved at all), so this isn't (yet) a "varnish has stale broken objects that MW doesn't serve anymore" sort of problem...

In a sample of 100 pages visited via random I found only one page with the issue: https://en.wikipedia.org/wiki/List_of_neighbourhoods_in_Vancouver

My concern around re-reverting the ParserCache hook is that it was either not working or the number of pages was so small it looked like it wasn't working.

I'm not convinced the edit-page class would appear in the ParserCache like this - edit-page has always been in SkinMinerva generated after the parser cache, so it's possible looking for that class may be a red herring. A better test would be to check for the in-block class but that only gets added when there are headings within a section...

For the current code, I think a good enough check is to match pages that contain <span class="mw-headline" and do not contain <span class="mw-editsection-bracket".

Jdlrobson added a subscriber: ori.Mar 16 2016, 5:45 PM

@ori @Legoktm @tstarling I'm sure you understand this stuff far greater than I can. Could you let us know why the parser cache hook was reverted and whether this will help?

With the parent resolved, do we need to look at wiping out any lingering corner-cases that are still cached in varnish?

BBlack closed this task as Resolved.Apr 6 2016, 1:30 PM

I'm guessing by now they're all naturally expiring out anyways since there's no further feedback.