Updated TemplateData from transcluded items take a long time to show up
OpenPublic

Description

TemplateData doesn't appear to be loading for https://fr.wikipedia.org/wiki/Mod%C3%A8le:Date or https://en.wikipedia.org/wiki/Template:Tracked - any idea what's going on? This is very high-priority since I'm about to send out notices asking high-contribution template editors to add template data to things.


Version: unspecified
Severity: normal

bzimport added a subscriber: wikibugs-l.
bzimport set Reference to bz50372.
Ironholds created this task.Via LegacyJun 28 2013, 12:14 PM
Ironholds added a comment.Via ConduitJun 28 2013, 12:44 PM

Belay that, {{tracked}} works.

Ironholds added a comment.Via ConduitJun 28 2013, 12:44 PM

As does the fr implementation. I guess the question here is "how long does it take for TemplateData implementations to show up?"

Jdforrester-WMF added a comment.Via ConduitJun 28 2013, 8:34 PM

(In reply to comment #2)

As does the fr implementation. I guess the question here is "how long does it
take for TemplateData implementations to show up?"

Do you mean,

  • "how long does it take newly-added TemplateData hinting to become available in the API for use in VisualEditor?",
  • "how long does it take the VisualEditor transclusions dialog to show with the TemplateData hinting when adding a template in?", or
  • "how long does it take the VisualEditor transclusions dialog to show with the TemplateData hinting on load?"

… or something else?

Ironholds added a comment.Via ConduitJun 28 2013, 8:35 PM

I'd go with A and C, Chris. If I add TemplateData to a template, go to an implementation of it that already exists and click the jigsaw piece, how long do I need between those actions to see the TemplateData?

Jdforrester-WMF added a comment.Via ConduitJun 28 2013, 9:13 PM

(In reply to comment #4)

I'd go with A and C, Chris. If I add TemplateData to a template, go to an
implementation of it that already exists and click the jigsaw piece, how long
do I need between those actions to see the TemplateData?

A: This should be instantaneous - if not, action=purge on the template should do it; if not, this is an issue we need to follow-up in general with how MW works.
B: This is not yet done (bug 49778 which is being worked on right now).
C: This should be near-instantaneous (but we're about to change how this works to make it faster).

For A & C in particular as you've asked, seconds or even less. Was this during a time of serious server strain, maybe? Otherwise it sounds worrying.

Ironholds added a comment.Via ConduitJun 28 2013, 9:44 PM

Nope; remember that time I added it to the bugzilla tracked template, and nothing happened? Well, it started working finally some time today. I'm going to try implementing it in a couple of other minor places and look at what happens.

Jdforrester-WMF added a comment.Via ConduitJun 28 2013, 9:46 PM

(In reply to comment #6)

Nope; remember that time I added it to the bugzilla tracked template, and
nothing happened?

No.

Well, it started working finally some time today.

That's not good. Did you purge the template, or did it get purged somehow (e.g. it was edited)?

I'm going to try implementing it in a couple of other minor places and look
at what happens.

OK.

Jdforrester-WMF added a comment.Via ConduitJun 28 2013, 10:03 PM

Argh, that's not good - https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Talkback is not updated even though it appears in transclusion, even after purge. Timo, thoughts?

gpaumier added a comment.Via ConduitJun 29 2013, 1:45 PM

The behavior seems inconsistent. Today, TemplateData was available in transclusions immediately after adding it to the template's documentation subpage. (tested with https://fr.wikipedia.org/wiki/Mod%C3%A8le:Unit%C3%A9 )

Krinkle added a comment.Via ConduitJun 30 2013, 2:37 AM

Though it is possible that VisualEditor and/or TemplateData do something wrong, in general there's nothing relevant in either VisualEditor or TemplateData that can delay the updating of the data.

The template data is changes by users by editing the wikitext of a Template page (or a page transcluded into a Template page). When they click "Save page", the MediaWiki parser detects the <templatedata> tag, tells the TemplateData extension to make sense of it, which then adds (or replaces) the parsed data in the database.

There is no deferred update here, it is replaced directly.

The API request to fetch templatedata grabs it directly from the database. This isn't specifically cached in anyway.

The only caching layer is in your browser cache and the general cache in the Wikimedia cluster for HTTP requests to content domain names (e.g. "en.wikipedia.org") which is Squid.

Ironholds added a comment.Via ConduitJul 1 2013, 4:09 PM

https://en.wikipedia.org/wiki/Template:Mileposts was updated 48 hours ago and still hasn't shown. That seems like a time period even the squids should blink at.

Mattflaschen added a comment.Via ConduitJul 2 2013, 6:55 AM

The extension page says:

"The TemplateData extension works through having a <templatedata> tag defined in the wikitext of a template page (optionally it can be transcluded into the template page as well). This is designed to be compatible with the common layout many wikis have of transcluding template documentation from a separate page (but is not required)."

And this seems to work partly, e.g. for https://en.wikipedia.org/wiki/Template:Italic_title. TemplateData is at https://en.wikipedia.org/wiki/Template:Italic_title/doc and it comes up at https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Italic%20title&format=jsonfm , but it doesn't seem to actually work in VE (e.g. no description).

Then, https://en.wikipedia.org/wiki/Template:Dead_link appears set up similar, with TemplateData at https://en.wikipedia.org/wiki/Template:Dead_link/doc . But it does not come up at https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Dead%20link&format=jsonfm . It does come up at https://en.wikipedia.org/w/api.php?action=templatedata&titles=Template:Dead%20link/doc&format=jsonfm (explicitly asking for the /doc page).

He7d3r added a comment.Via ConduitJul 2 2013, 2:12 PM

Wouldn't we avoid this delay if the template data were in a specific namespace, instead of being saved in one page, which is transcluded in another, which on its turn may be transcluded another, and so on, until the data finally gets to the template? See bug 50512.

Krinkle added a comment.Via ConduitJul 2 2013, 8:50 PM

I added[1] templatedata to en.wiki [[Template:Redirect]] yesterday (through the /doc transclusion) and also noticed that the main template page wasn't updating (I waited for about 10 minutes).

Then I did a null-edit on Template:Redirect and it was instantly there.

Roan mentioned something earlier (IRC or IRL, don't remember) about this perhaps being caused by the JobQueue (which we use to invalidate pages transcluding other pages, in this case the transclusion of /doc into Template:. *NOT* the transclusion of Template into articles, of course, that'd be insane).

CC-ing Roan, Tim and Aaron who may have more ideas about the cause of this.

As for TemplateData itself, it is triggered as a extension parser tag (<templatedata>) and during the parsing it adds page properties which MediaWiki ends up storing in the page_props db table. The API does a read query from that same table. Pretty basic.

So what could cause the read query to the page_props table to be delayed (or perhaps even not happen at all?) from the page properties added by a page that is transcluded (Template:Foo/doc) into a template page (Template:Foo).

JobQueue? Database replication? Something else?

Note that in my testing yesterday on Template:Redirect I ruled out HTTP caching (API cluster Squid) by making unique urls each time.

[1] https://en.wikipedia.org/w/index.php?title=Template%3ARedirect%2Fdoc&diff=562472124&oldid=562471783

Rschen7754 added a comment.Via ConduitJul 2 2013, 10:00 PM

Thought I'd drop a note and say that when I tried a null edit on https://en.wikipedia.org/wiki/Template:Mileposts, the overall template description loaded right away on https://en.wikipedia.org/wiki/British_Columbia_Highway_7. However, the parameter description (km=) did not load until I deleted the template from the article and reinserted the template anew.

Jdforrester-WMF added a comment.Via ConduitJul 2 2013, 10:09 PM

Timo has narrowed down the issue to bug 5382 - we're working now to see whether we can fix that bug and so resolve this, or whether that will put too much weight on the cluster.

Krinkle added a comment.Via ConduitJul 2 2013, 11:26 PM

Observation:

  • Make a null edit to "Template:Pagetype".

So what is causing this to not be auto updated? I think bug 5382. But having trouble figuring out how come bug 5382 exists since we do have LinkUpdate and RefreshLinks2 which are triggered on edit from the job queue and update all pages linking to it, and the code in LinkUpdate includes all link tables and page properties.

tstarling added a comment.Via ConduitJul 3 2013, 2:04 AM

You appear to just be seeing the effect of weeks of job queue lag, i.e. a system problem, not a MediaWiki problem.

Eloquence added a comment.Via ConduitJul 9 2013, 2:49 AM

Tim's change https://gerrit.wikimedia.org/r/#/c/72679/ (deployed) seems to have made it possible to force templatedata updates by just performing a null edit on the affected template, which is now safe to do on highly-used templates. Note that a null edit is an edit made by just pressing "Save page" without making _any_ changes to the template code itself.

SalixAlba added a comment.Via ConduitAug 9 2013, 7:31 AM

What would be nice is if action=purge would force templatedata updates, like null edits do. This is necessary as many templates are protected so only admins can do null edits. Any editor can do an action=purge (I believe).

Jdforrester-WMF added a comment.Via ConduitAug 10 2013, 8:07 AM

Moving this to be a site issue, and re-grading. Thanks to Tim's fix this isn't critical any more (users can 'just' null-edit touch the relevant template to cause TemplateData content to come through), but it's very irritating and we need to fix it.

gpaumier added a comment.Via ConduitAug 13 2013, 1:38 PM

(In reply to comment #23)

Thanks to Tim's fix this isn't
critical any more (users can 'just' null-edit touch the relevant template to
cause TemplateData content to come through)

Note: This workaround only works for templates that aren't widely used.

Widely-used templates are protected, and users can't null-edit them unless they are an admin (which drastically reduces the population of people who can add and edit TemplateData on the exact templates which would benefit from TemplateData most).

Mvolz added a comment.Via ConduitAug 22 2014, 6:38 AM
  • Bug 69647 has been marked as a duplicate of this bug. ***
Ironholds removed a subscriber: Ironholds.Via WebDec 5 2014, 7:46 PM
Elitre added a subscriber: Elitre.Via WebDec 11 2014, 12:42 PM
SamB changed the title from "Updated page properties from transcluded items take a long time to show up" to "Updated TemplateData from transcluded items take a long time to show up".Via WebApr 4 2015, 4:31 AM
SamB set Security to None.
SamB added a subscriber: SamB.Via WebApr 4 2015, 4:39 AM

What would be nice is if action=purge would force templatedata updates, like null edits do. This is necessary as many templates are protected so only admins can do null edits. Any editor can do an action=purge (I believe).

Hear, hear! And yes, I certainly seem to be able to purge pages I can't edit.

Jdforrester-WMF moved this task to Backlog on the VisualEditor workboard.Via WebTue, Apr 7, 4:18 PM
Mattflaschen removed a subscriber: Mattflaschen.Via WebTue, Apr 14, 2:47 AM

Add Comment