Page MenuHomePhabricator

REST API is not invalidating caches after template and/or module changes
Closed, ResolvedPublicBUG REPORT

Description

A now globally-banned user vandalised en.wikivoyage's Coordinates module (edit deleted) on 28 April 2023 with the phrase "Anorexia is fun". This continues to show up on some pages in the mobile REST API as Lua error in package.lua at line 80: module 'Anorexia is fun' not found. For example:

$ curl https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178 | grep 'Anorexia is fun'
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html><html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="https://en.wikivoyage.org/wiki/Special:Redirect/revision/4502178" class="no-editing"><head prefix="mwr: https://en.wikivoyage.org/wiki/Special:Redirect/"><meta property="mw:TimeUuid" content="f2c83640-e55d-11ed-b1ea-a51106ac9ee2"><meta charset="utf-8"><meta property="mw:pageId" content="10438"><meta property="mw:pageNamespace" content="0"><meta property="mw:revisionSHA1" content="01c4129de48233f105f84c3a1cb6e3410621da17"><meta property="dc:modified" content="2022-08-16T12:29:20.000Z"><meta property="mw:htmlVersion" content="2.7.0"><meta property="mw:html:version" content="2.7.0"><link rel="dc:isVersionOf" href="//en.wikivoyage.org/wiki/East_Frisia"><base href="//en.wikivoyage.org/wiki/"><title>East Frisia</title><meta property="mw:jsConfigVars" content="{&quot;wgKartographerLiveData&quot;:{&quot;mask&quot;:[],&quot;around&quot;:[],&quot;buy&quot;:[],&quot;city&quot;:[],&quot;do&quot;:[],&quot;event&quot;:[],&quot;drink&quot;:[],&quot;eat&quot;:[],&quot;go&quot;:[],&quot;listing&quot;:[],&quot;other&quot;:[],&quot;see&quot;:[],&quot;sleep&quot;:[],&quot;vicinity&quot;:[],&quot;view&quot;:[],&quot;black&quot;:[],&quot;blue&quot;:[],&quot;brown&quot;:[],&quot;chocolate&quot;:[],&quot;forestgreen&quot;:[],&quot;gold&quot;:[],&quot;gray&quot;:[],&quot;grey&quot;:[],&quot;lime&quot;:[],&quot;magenta&quot;:[],&quot;maroon&quot;:[],&quot;mediumaquamarine&quot;:[],&quot;navy&quot;:[],&quot;red&quot;:[],&quot;royalblue&quot;:[],&quot;orange&quot;:[],&quot;silver&quot;:[],&quot;steelblue&quot;:[],&quot;teal&quot;:[],&quot;fuchsia&quot;:[],&quot;route1&quot;:[],&quot;route2&quot;:[],&quot;route3&quot;:[],&quot;route4&quot;:[],&quot;route5&quot;:[]},&quot;ScribuntoErrors&quot;:[&quot;<p>Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</p><p>Backtrace:</p><ol class=\&quot;scribunto-trace\&quot;><li><strong>[C]</strong>: in function \&quot;error\&quot;</li><li><strong>package.lua:80</strong>: in function \&quot;load\&quot;</li><li><strong>package.lua:99</strong>: in function \&quot;require\&quot;</li><li><strong><a href=\&quot;//en.wikivoyage.org/w/index.php?title=Module:Coordinates&amp;amp;action=edit#mw-ce-l21\&quot;>Module:Coordinates:21</a></strong>: in function \&quot;init\&quot;</li><li><strong>package.lua:103</strong>: in function \&quot;require\&quot;</li><li><strong><a href=\&quot;//en.wikivoyage.org/w/index.php?title=Module:Marker&amp;amp;action=edit#mw-ce-l8\&quot;>Module:Marker:8</a></strong>: in function \&quot;chunk\&quot;</li><li><strong>mw.lua:496</strong>:&amp;#160;?</li><li><strong>[C]</strong>:&amp;#160;?</li></ol>&quot;]}"><meta property="mw:generalModules" content="ext.kartographer.frame|ext.scribunto.errors"><meta property="mw:moduleStyles" content="ext.kartographer.style"><meta http-equiv="content-language" content="en"><meta http-equiv="vary" content="Accept"><link rel="stylesheet" href="//meta.wikimedia.org/api/rest_v1/data/css/mobile/base"><link rel="stylesheet" href="//en.wikivoyage.org/api/rest_v1/data/css/mobile/site"><link rel="stylesheet" href="//meta.wikimedia.org/api/rest_v1/data/css/mobile/pcs"><meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1, shrink-to-fit=no"><script src="//meta.wikimedia.org/api/rest_v1/data/javascript/mobile/pcs"></script><link rel="icon" href="data:,"><meta property="pcs:locale" content="en"><meta property="mw:leadImage" content="https://upload.wikimedia.org/wikipedia/commons/e/e7/Neuharlingersiel_-_Hafen_-_16.07.07.JPG" data-file-width="1200" data-file-height="900"></head><body lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output content skin-minerva" dir="ltr"><div id="pcs" class="mw-parser-output"><script>pcs.c1.Page.onBodyStart();</script><header><div class="pcs-edit-section-header v2"><div class="pcs-header-inner-left"><h1 data-id="0" class="pcs-edit-section-title"><span class="mw-page-title-main">East Frisia</span></h1><p data-description-source="central" data-wikdata-entity-id="Q165269" id="pcs-edit-section-title-description">historic and cultural region in Lower Saxony, Germany</p><hr id="pcs-edit-section-divider"></div><div class="pcs-header-inner-right"></div></div></header><section data-mw-section-id="0"><span class="pcs-edit-section-link-container"><a href="/w/index.php?title=East_Frisia&amp;action=edit&amp;section=0" data-id="0" data-action="edit_section" aria-labelledby="pcs-edit-section-aria-normal" class="pcs-edit-section-link"></a></span><p><style data-mw-deduplicate="TemplateStyles:r4253834" typeof="mw:Extension/templatestyles mw:Transclusion">.mw-parser-output .oo-ui-icon-unesco{background-image:url("https://upload.wikimedia.org/wikipedia/commons/b/b1/WorldHeritageBlanc.svg")}.mw-parser-output .oo-ui-icon-star{background-image:url("https://upload.wikimedia.org/wikipedia/commons/e/e7/Cscr-featured.svg")}.mw-parser-output .oo-ui-icon-ftt{background-image:url("https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Writing_Circle.svg/25px-Writing_Circle.svg.png")}.mw-parser-output .oo-ui-icon-dotm{background-image:url("https://upload.wikimedia.org/wikipedia/commons/thumb/5/50/Yes_Check_Circle.svg/25px-Yes_Check_Circle.svg.png")}.mw-parser-output .oo-ui-icon-otbp{background-image:url("https://upload.wikimedia.org/wikipedia/commons/thumb/6/6e/Right_turn_icon_blue.svg/25px-Right_turn_icon_blue.svg.png")}</style><span class="countryData" data-country-calling-code="+49" data-local-dialing-code="" data-currency="EUR, €, c" style="display:none;"></span><span>   </span>
<ul><li><strong class="error" about="#mwt9" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Aurich]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.471389&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.483611&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q14839&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong></li>
<li><strong class="error" about="#mwt10" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Emden]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.366944&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.206111&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q4174&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong></li>
<li><strong class="error" about="#mwt11" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Esens]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.646944&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.612778&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q509338&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> — a regional transportation hub with buses departing for the ferryports of several <a href="./East_Frisian_islands" title="East Frisian islands" class="mw-redirect">East Frisian islands</a></li>
<li><strong class="error" about="#mwt12" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Leer]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.230833&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.452778&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q15984&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong></li>
<li><strong class="error" about="#mwt13" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Norden Norddeich]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.596667&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.205556&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q492366&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> — ferry to Norderney and Juist</li>
<li><strong class="error" about="#mwt14" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Varel]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.396944&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;8.136111&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q496056&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> —</li>
<li><strong class="error" about="#mwt15" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Wilhelmshaven]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.516667&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;8.133333&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q3857&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> — named for the Prussian ruler, this city used to be Germany's most important military harbour</li>
<li><strong class="error" about="#mwt16" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;city&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Wittmund]]&quot;},&quot;lat&quot;:{&quot;wt&quot;:&quot;53.574722&quot;},&quot;long&quot;:{&quot;wt&quot;:&quot;7.780833&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q509353&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> —</li></ul>
]&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q27769&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong></li>
<li><strong class="error" about="#mwt18" typeof="mw:Transclusion" data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;marker&quot;,&quot;href&quot;:&quot;./Template:Marker&quot;},&quot;params&quot;:{&quot;type&quot;:{&quot;wt&quot;:&quot;vicinity&quot;},&quot;name&quot;:{&quot;wt&quot;:&quot;[[Lower Saxon Wadden Sea National Park]]&quot;},&quot;wikidata&quot;:{&quot;wt&quot;:&quot;Q661217&quot;}},&quot;i&quot;:0}}]}"><span class="scribunto-error" id="mw-scribunto-error-0">Lua error in package.lua at line 80: module 'Anorexia is fun' not found.</span></strong> — the large national park area stretches along the North Sea coast of East Frisia and <a href="./Elbe-Weser_Triangle" title="Elbe-Weser Triangle">Elbe-Weser Triangle</a> between Ems and Elbe river estuaries.</li></ul>

Null editing the page, the template and the module does not clear the caching.

Event Timeline

Was just thinking... if either @MSantos or @akosiaris might try invalidating the CDN cache for East Frisia, that might remove the (pretty deeply) offensive content and then this could be backlogged for a bit pending further investigation, I suppose.

I just null edited the article and that updated the cache...

@TheDJ I'm still seeing it (from Belgium): curl https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178 | grep 'Anorexia is fun' or just take a look at https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178 ... so I'm thinking a CDN cache invalidation might help.

@TheDJ I'm still seeing it (from Belgium): curl https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178 | grep 'Anorexia is fun' or just take a look at https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178 ... so I'm thinking a CDN cache invalidation might help.

Can't say that I see that. In any part of the global CDN. $ for i in eqsin eqiad codfw esams ulsfo drmrs ; do curl -s --resolve en.wikivoyage.org:443:$(host text-lb.${i}.wikimedia.org | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}') https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178; done |grep -i Anorexia returns nothing.

@akosiaris if I curl it from San Francisco (VPN) I don't see it. If I curl it from Belgium (no VPN) I see it. It's being seen by at least one other experienced editor at Wikivoyage as well. If it's not CDN... what could possibly be up with the geographical discrepancy? s-maxage=1209600, max-age=0??

So, wild guess, but an ISP might interpret s-maxage=1209600 themselves and do their own caching based on that. If true, then... ugh. If this is true, then we could just do the standard "blame Huawei" and move on :/

If an ISP (an internet service provider) is interfering with your HTTPS traffic, that's a very very bad thing. I have serious doubts that would happen in Belgium. So, if you do this tests from a home connection and you see these, we need to dive in deeper.

However, if you do these from the infrastructure of some company or likewise entity, there is a higher probability that indeed some caching forward proxy exists, and systems are configured to use it and trust it. That would be the caching forward proxy that indeed would cache and honor that s-maxage setting.

For what is worth, retried my test from above today as well and all parts of the CDN report having the latest version of page (that is revision https://en.wikivoyage.org/w/index.php?title=East_Frisia&diff=prev&oldid=4656297)

Home network (Airbnb). Get this: restarting the home wifi router fixed it.

I also got clean results (before restarting the router) when using any VPN connection (personal one and random ones via Proton – USA, Japan, Netherlands) and also my Belgian mobile hotspot.

I did get "Anorexia is fun" on an new, clean machine on the home network (before router restart), so that told me it wasn't a local machine issue.

I guess the headers wouldn't matter because any forward proxy (even a router, I guess) would need the SSL cert to read them.

So, a few things:

  1. I have at least one report of a user still seeing "Anorexia is fun" from the Netherlands, but only via the REST API (as was the same with me).
  2. Other users (@SHB2000, @Bawolff) reported seeing "Anorexia is fun" two days ago when I reported it, although SHB2000 now reports that it's clean. (The vandalism occurred on 28 April; I reported on 2 May)
  3. Let's assume some wifi routers do some sort of caching... why would this only affect the REST API and not the standard web/mobile pages?

For what it's worth, the router here appears to be a combined router/DSL modem, so that makes it even more fun. I'd be tempted to close this ticket and just say that home wifi routers suck (they do), but question (3) on why this might only affect the REST API and the fact that other users are still seeing it really makes me wonder...

One more thing, before hitting submit here I tried the curl one more time and "Anorexia is fun" came back! Then I tried it immediately again several times and it was gone :/

@akosiaris update: the user who reported seeing the issue from the Netherlands also reported that restarting their wifi router fixed the problem. This is really... something.

Other users (@SHB2000, @Bawolff) reported seeing "Anorexia is fun" two days ago when I reported it, although SHB2000 now reports that it's clean. (The vandalism occurred on 28 April; I reported on 2 May)

To clarify. I did not see the vandalism itself (or actually look), I just checked RC to try and find the potential source.

akosiaris added a subscriber: ssingh.

Home network (Airbnb). Get this: restarting the home wifi router fixed it.

I also got clean results (before restarting the router) when using any VPN connection (personal one and random ones via Proton – USA, Japan, Netherlands) and also my Belgian mobile hotspot.

I did get "Anorexia is fun" on an new, clean machine on the home network (before router restart), so that told me it wasn't a local machine issue.

I guess the headers wouldn't matter because any forward proxy (even a router, I guess) would need the SSL cert to read them.

So, a few things:

  1. I have at least one report of a user still seeing "Anorexia is fun" from the Netherlands, but only via the REST API (as was the same with me).
  2. Other users (@SHB2000, @Bawolff) reported seeing "Anorexia is fun" two days ago when I reported it, although SHB2000 now reports that it's clean. (The vandalism occurred on 28 April; I reported on 2 May)
  3. Let's assume some wifi routers do some sort of caching... why would this only affect the REST API and not the standard web/mobile pages?

For what it's worth, the router here appears to be a combined router/DSL modem, so that makes it even more fun. I'd be tempted to close this ticket and just say that home wifi routers suck (they do), but question (3) on why this might only affect the REST API and the fact that other users are still seeing it really makes me wonder...

One more thing, before hitting submit here I tried the curl one more time and "Anorexia is fun" came back! Then I tried it immediately again several times and it was gone :/

I am gonna cue in Traffic on this one, specifically @ssingh. They are probably interested on how this can happen given your data points and might have more concrete debugging steps than yours truly.

@akosiaris if I curl it from San Francisco (VPN) I don't see it. If I curl it from Belgium (no VPN) I see it. It's being seen by at least one other experienced editor at Wikivoyage as well. If it's not CDN... what could possibly be up with the geographical discrepancy? s-maxage=1209600, max-age=0??

What certificate does curl get when requesting the resource? (This could verify whether or not the theory of a caching external proxy is at fault). I think you can get that from curl with the -vvi option.

If this happens again to you, it would probably be useful to include all http headers, in case anything there is relavant. (E.g. via and age header could be informative)

@Bawolff We may be stuck for now, since I seem to have resolved the problem at my Belgian address but resetting my router, and I am in currently in Lisbon until next week (where the error does not seem to occur). But, let me give it another shot next week when I'm back in Belg and if I see it again I'll make sure to include the headers.

I can also probe again with the Wikivoyage folks to see if anyone else is still seeing it and perhaps grab the headers from them, if I ask nicely.

I also hope it's not something doing MITM into the https connection without you being aware. It seems unlikely, though, and listing the https certificate you see will let us discard that idea.

note: wmf uses two sets of certificates, so don't panic: it is expected that you will get a different certificate when curling from US (Let's Encrypt) than from Belgium (Digicert).
 You can copy the full certificate in PEM format (base64) with openssl s_client -connect en.wikivoyage.org:443

As for the reason restarting the router fixed it, I suspect restarting the router might have changed your IP address and, in turn, the IP address could be used for routing to one endpoint/server or other, with only one of them holding the bad cached page.

As for the reason restarting the router fixed it, I suspect restarting the router might have changed your IP address and, in turn, the IP address could be used for routing to one endpoint/server or other, with only one of them holding the bad cached page.

That's a very valid idea, but I did already account for it (I realized I wasn't clear on what command I used, so here goes). The command for that can only be run inside the infrastructure and it's

cumin1001:~$ sudo cumin 'cp*' 'curl -s --resolve en.wikivoyage.org:443:$(facter -p -j | jq -r .ipaddress) https://en.wikivoyage.org/api/rest_v1/page/mobile-html/East_Frisia/4502178|grep -i Anorexia'

It asks every single node in the CDN to return the page it has locally and search in the output for "Anorexia". Needless to say, it returns nothing.

If this is reproduced somehow, indeed curl -vv output might help pinpoint this down.

Re-reading my answer, I realized that I have left a window open to suggest that a Belgian ISP might be doing a MITM attack. Let me be very clear. I very much doubt that's the case. If anything, this is almost certainly a cache-invalidation issue. Most probably on the WMF infra side, even though my replies above seem to rule that out. I might have missed something.

akosiaris claimed this task.

I am gonna tentatively resolve this in the interest of not leaving a ticket lingering open long-term. @Brycehughes please do reopen if you experience this again and paste some curl -vvv output so we can pinpoint the reason. Thanks!

@akosiaris Fair enough. I also highly doubt it's anything MITM related. There is something weird going on, but it's spurious, so I suppose we'll just have to wait and see, because I can't think of a good reproduction case (apart from writing my own module and template and dummy page and then vandalising it/reverting it while curling it via the REST API... oh but who has the time...). Thanks all who participated here. Sorry about the lack of -vvi output... quite ironically, i was initially curling it with -vvi (or similar) but didn't insert it in this bug report for the sake of simplicity (and I don't log my terminal output).

Ok, reopening this, because I think I've been able to re-create the issue. I'm in Tbilisi, Georgia, I updated a wikivoyage lede. The change does not show up in the "extract" field when I'm on my local router here. The change does show when I jump on my San Francisco-based VPN.

Here are the two results, for comparison. First, local router (located in Georgia, the country):

curl -vvv https://en.wikivoyage.org/api/rest_v1/page/summary/Čičmany
*   Trying 91.198.174.192:443...
* Connected to en.wikivoyage.org (91.198.174.192) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Unknown (8):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
*  start date: Oct 27 00:00:00 2022 GMT
*  expire date: Nov 17 23:59:59 2023 GMT
*  subjectAltName: host "en.wikivoyage.org" matched cert's "*.wikivoyage.org"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /api/rest_v1/page/summary/%c4%8ci%c4%8dmany]
* h2h3 [:scheme: https]
* h2h3 [:authority: en.wikivoyage.org]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x12e80a800)
> GET /api/rest_v1/page/summary/%c4%8ci%c4%8dmany HTTP/2
> Host: en.wikivoyage.org
> user-agent: curl/7.87.0
> accept: */*
>
< HTTP/2 200
< vary: Accept-Encoding
< cache-control: s-maxage=1209600, max-age=300
< content-language: en
< content-type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/Summary/1.5.0"
< x-restbase-sunset: true
< content-location: https://en.wikivoyage.org/api/rest_v1/page/summary/%C4%8Ci%C4%8Dmany
< access-control-allow-origin: *
< access-control-allow-methods: GET,HEAD
< access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
< access-control-expose-headers: etag
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< referrer-policy: origin-when-cross-origin
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-webkit-csp: default-src 'none'; frame-ancestors 'none'
< server: restbase1016
< date: Sun, 02 Jul 2023 14:30:27 GMT
< etag: W/"3500520/fd5329c0-18e4-11ee-ae3f-5746a96d0b3b"
< age: 665
< x-cache: cp3056 miss, cp3056 hit/30
< x-cache-status: hit-front
< server-timing: cache;desc="hit-front", host;desc="cp3056"
< strict-transport-security: max-age=106384710; includeSubDomains; preload
< report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
< nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
< set-cookie: WMF-Last-Access=02-Jul-2023;Path=/;HttpOnly;secure;Expires=Thu, 03 Aug 2023 12:00:00 GMT
< set-cookie: WMF-Last-Access-Global=02-Jul-2023;Path=/;Domain=.wikivoyage.org;HttpOnly;secure;Expires=Thu, 03 Aug 2023 12:00:00 GMT
< x-client-ip: 92.54.248.160
< set-cookie: GeoIP=GE:TB:Tbilisi:41.70:44.83:v4; Path=/; secure; Domain=.wikivoyage.org
< set-cookie: NetworkProbeLimit=0.001;Path=/;Secure;Max-Age=3600
< accept-ranges: bytes
< content-length: 1532
<
* Connection #0 to host en.wikivoyage.org left intact
{"type":"standard","title":"Čičmany","displaytitle":"<span class=\"mw-page-title-main\">Čičmany</span>","namespace":{"id":0,"text":""},"wikibase_item":"Q341594","titles":{"canonical":"Čičmany","normalized":"Čičmany","display":"<span class=\"mw-page-title-main\">Čičmany</span>"},"pageid":5359,"thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/f/f0/Cicmany-drevenice.jpg","width":300,"height":400},"originalimage":{"source":"https://upload.wikimedia.org/wikipedia/commons/f/f0/Cicmany-drevenice.jpg","width":300,"height":400},"lang":"en","dir":"ltr","revision":"3500520","tid":"09e26c10-025a-11ee-b438-173fb3d684c8","timestamp":"2018-05-13T17:06:51Z","description":"municipality of Slovakia","description_source":"central","coordinates":{"lat":48.9553,"lon":18.5167},"content_urls":{"desktop":{"page":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany","revisions":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=history","edit":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=edit","talk":"https://en.wikivoyage.org/wiki/Talk:%C4%8Ci%C4%8Dmany"},"mobile":{"page":"https://en.m.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany","revisions":"https://en.m.wikivoyage.org/wiki/Special:History/%C4%8Ci%C4%8Dmany","edit":"https://en.m.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=edit","talk":"https://en.m.wikivoyage.org/wiki/Talk:%C4%8Ci%C4%8Dmany"}},"extract":" \nČičmany is a folks village in Central Slovakia.","extract_html":"<p> \n<i>Čičmany</i> is a folks village in Central Slovakia.</p>"}%

Now from my VPN (San Francisco):

curl -vvv https://en.wikivoyage.org/api/rest_v1/page/summary/Čičmany
*   Trying [2620:0:863:ed1a::1]:443...
*   Trying 198.35.26.96:443...
* Connected to en.wikivoyage.org (2620:0:863:ed1a::1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Unknown (8):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.wikipedia.org
*  start date: Jun 23 07:22:49 2023 GMT
*  expire date: Sep 21 07:22:48 2023 GMT
*  subjectAltName: host "en.wikivoyage.org" matched cert's "*.wikivoyage.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /api/rest_v1/page/summary/%c4%8ci%c4%8dmany]
* h2h3 [:scheme: https]
* h2h3 [:authority: en.wikivoyage.org]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15600a800)
> GET /api/rest_v1/page/summary/%c4%8ci%c4%8dmany HTTP/2
> Host: en.wikivoyage.org
> user-agent: curl/7.87.0
> accept: */*
>
< HTTP/2 200
< vary: Accept-Encoding
< cache-control: s-maxage=1209600, max-age=300
< content-language: en
< content-type: application/json; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/Summary/1.5.0"
< x-restbase-sunset: true
< content-location: https://en.wikivoyage.org/api/rest_v1/page/summary/%C4%8Ci%C4%8Dmany
< access-control-allow-origin: *
< access-control-allow-methods: GET,HEAD
< access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
< access-control-expose-headers: etag
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< referrer-policy: origin-when-cross-origin
< x-xss-protection: 1; mode=block
< content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-webkit-csp: default-src 'none'; frame-ancestors 'none'
< server: restbase2024
< date: Sun, 02 Jul 2023 14:38:03 GMT
< etag: W/"4688664/0d18ae10-18e6-11ee-83b7-50b0ec03d4ce"
< age: 227
< x-cache: cp4037 miss, cp4037 hit/1
< x-cache-status: hit-front
< server-timing: cache;desc="hit-front", host;desc="cp4037"
< strict-transport-security: max-age=106384710; includeSubDomains; preload
< report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
< nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
< set-cookie: WMF-Last-Access=02-Jul-2023;Path=/;HttpOnly;secure;Expires=Thu, 03 Aug 2023 12:00:00 GMT
< set-cookie: WMF-Last-Access-Global=02-Jul-2023;Path=/;Domain=.wikivoyage.org;HttpOnly;secure;Expires=Thu, 03 Aug 2023 12:00:00 GMT
< x-client-ip: 2604:a880:4:1d0::363:8000
< set-cookie: GeoIP=US:CA:Santa_Clara:37.39:-121.96:v4; Path=/; secure; Domain=.wikivoyage.org
< set-cookie: NetworkProbeLimit=0.001;Path=/;Secure;Max-Age=3600
< accept-ranges: bytes
< content-length: 1520
<
* Connection #0 to host en.wikivoyage.org left intact
{"type":"standard","title":"Čičmany","displaytitle":"<span class=\"mw-page-title-main\">Čičmany</span>","namespace":{"id":0,"text":""},"wikibase_item":"Q341594","titles":{"canonical":"Čičmany","normalized":"Čičmany","display":"<span class=\"mw-page-title-main\">Čičmany</span>"},"pageid":5359,"thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/f/f0/Cicmany-drevenice.jpg","width":300,"height":400},"originalimage":{"source":"https://upload.wikimedia.org/wikipedia/commons/f/f0/Cicmany-drevenice.jpg","width":300,"height":400},"lang":"en","dir":"ltr","revision":"4688664","tid":"33a65420-18e5-11ee-a7db-45123709aa45","timestamp":"2023-07-02T14:31:58Z","description":"municipality of Slovakia","description_source":"central","coordinates":{"lat":48.9553,"lon":18.5167},"content_urls":{"desktop":{"page":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany","revisions":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=history","edit":"https://en.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=edit","talk":"https://en.wikivoyage.org/wiki/Talk:%C4%8Ci%C4%8Dmany"},"mobile":{"page":"https://en.m.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany","revisions":"https://en.m.wikivoyage.org/wiki/Special:History/%C4%8Ci%C4%8Dmany","edit":"https://en.m.wikivoyage.org/wiki/%C4%8Ci%C4%8Dmany?action=edit","talk":"https://en.m.wikivoyage.org/wiki/Talk:%C4%8Ci%C4%8Dmany"}},"extract":" \nČičmany is a village in Central Slovakia.","extract_html":"<p> \n<i>Čičmany</i> is a village in Central Slovakia.</p>"}%

I deleted the word "folks" from the lede section. Notice it shows up in the local router curl, but not in the VPN call. @akosiaris is this information at all useful? I know it isn't directly related to module changes like I mentioned in the bug report, but it does feel like a similar (if not the same) caching issue. Many thanks.

@Brycehughes it does, albeit I think you shouldn't be able to reproduce now. This was possibly the result of cache purging taking some time to happen.

From the curl output it's clear that for the case that you witnessed the old content, you were hitting cp3056 and it was replying with content 1532 bytes in size. The response was generated at Sun, 02 Jul 2023 14:30:27 GMT which is 1 minute before your change (https://en.wikivoyage.org/w/index.php?title=%C4%8Ci%C4%8Dmany&diff=prev&oldid=4688664).

The response from San Francisco is clearly shorter in length, 1520 bytes, generated 7 minutes after your change date: Sun, 02 Jul 2023 14:38:03 GMT.

It is not unheard of for CDN cache purges to take up to a few minutes in some extreme cases.

I 've tried to reproduce just right now and I see no caching proxy returning the stale content, which matches with my theory that this was just a late purge event delivery.

Can you please test on your side?

@akosiaris Yep all clear now from Georgia (the country). However, this lasted much more than "several minutes". What do you think is the maximum expected time on the CDN cache purge? (FWIW I am impressed that you can even purge your CDN [individually?] so quickly... cool tbh)

@akosiaris Yep all clear now from Georgia (the country). However, this lasted much more than "several minutes". What do you think is the maximum expected time on the CDN cache purge? (FWIW I am impressed that you can even purge your CDN [individually?] so quickly... cool tbh)

It's not easy to answer that for all cases, as there might be some race conditions for some internal workloads, but for most intents and purposes, you can think the upper bound as 24H. That the amount of time after which the CDN will try to revalidate the content, no matter what. Purges haven't taken a large amount of time to happen in a very long time, but it is possible that in some cases due to race conditions between components we might end up with an older revision in some internal caches.

@akosiaris Fair enough. Ah, the joys of caching. Thanks.