| Architecture meetings | ||
|---|---|---|
| 13:00 PT ArchCom Planning Meetings | upcoming | all since 2016-03-30 |
| 14:00 PT ArchCom-RFC Meetings | upcoming | all since 2015-09-09 |
TechCom discussed this in E170. @daniel and @GWicke agreed that T102476 would be a good topic to discuss (with no objections). That RFC lacked a shepherd, so I agreed to shepherd it.
I'm not entirely clear as to what meeting type we should strive for in this meeting. What should we strive to accomplish in this meeting?
Last week, I wrote:
I'm not entirely clear as to what meeting type we should strive for in this meeting. What should we strive to accomplish in this meeting?
It seems that on track for a "Problem definition" conversation in the upcoming meeting. I haven't had a chance to coordinate with @GWicke on this yet, but I think the next step is to move the prose of the RFC over to MediaWiki.org, as I suggested in T102476#2296335 (and even have the prose copied over; just need to RFC-ize it)
3:03 PM <wm-labs-meetbot> Meeting ended Wed May 18 22:03:45 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
3:03 PM <wm-labs-meetbot> Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.html
3:03 PM <wm-labs-meetbot> Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.txt
3:03 PM <wm-labs-meetbot> Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.wiki
3:03 PM <wm-labs-meetbot> Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.log.html
Meeting started by TimStarling at 21:00:02 UTC. The full logs are
available at
https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.log.html
.
Meeting ended at 22:03:45 UTC.
Log:
| 1 | 21:00:02 <TimStarling> #startmeeting RFC meeting |
|---|---|
| 2 | 21:00:02 <wm-labs-meetbot> Meeting started Wed May 18 21:00:02 2016 UTC and is due to finish in 60 minutes. The chair is TimStarling. Information about MeetBot at http://wiki.debian.org/MeetBot. |
| 3 | 21:00:02 <wm-labs-meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. |
| 4 | 21:00:02 <wm-labs-meetbot> The meeting name has been set to 'rfc_meeting' |
| 5 | 21:00:14 <TimStarling> #topic RFC: Requirements for change propagation | Wikimedia meeting channel | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ |
| 6 | 21:00:44 <gwicke> hi |
| 7 | 21:00:58 <mobrovac> hello |
| 8 | 21:01:03 <gwicke> #link https://phabricator.wikimedia.org/T102476 |
| 9 | 21:02:11 <gwicke> DanielK_WMDE & myself thought that it would be useful to catch up on what has been going on in eventbus & change propagation land, and talk about needs and open questions |
| 10 | 21:02:52 <Scott_WUaS> Great, Gabriel |
| 11 | 21:03:39 <gwicke> while most of the RFC is older & aimed at providing high-level background to the general issue, the section "Current status" is new & has information on current work, as well as some notes on next steps & open questions |
| 12 | 21:05:02 <gwicke> the biggest of those open questions is probably cross-project dependency tracking & change propagation |
| 13 | 21:05:51 <robla> gwicke: what are your thoughts about my last comment? https://phabricator.wikimedia.org/T102476#2296335 I think it'd be easier to keep track of the prose for long RFCs (like this one) using our software. |
| 14 | 21:06:20 <DanielK_WMDE> so, for the sake of stating the obvious: the general idea is to track dependencies between "things" (identified by URIs or some such) as a DAG, so we know what to purgfe or re-generate when. at least, that seems to be the core of it. |
| 15 | 21:06:21 <gwicke> I know DanielK_WMDE is involved in that area, so I hope that he can tell us a bit more about the situation & needs for wikidata |
| 16 | 21:06:52 <DanielK_WMDE> gwicke: yea. so, one important issue is thæt the artifact we track may be created on the fly, during a page view (GET request) |
| 17 | 21:07:19 <DanielK_WMDE> if we track that in an SQL db, that means a master write during a get request, possible a cross-DC operation. |
| 18 | 21:07:25 <gwicke> robla: I'm not opposed to moving the general portion especially to mw.org, lets just make sure we don't end up with multiple copies |
| 19 | 21:07:25 <DanielK_WMDE> aaron and ori don't like that... |
| 20 | 21:07:42 <matt_flaschen> DanielK_WMDE, there are workarounds for it, as long as it can be queued. |
| 21 | 21:07:55 <matt_flaschen> DanielK_WMDE, what is an example of triggering on a GET |
| 22 | 21:07:55 <mobrovac> DanielK_WMDE: and the artefact dies with the req or? |
| 23 | 21:08:09 <bd808> cross-dc master writes on GET certainly aren't optimal |
| 24 | 21:08:25 <DanielK_WMDE> matt_flaschen: yes, we currently use the job queue for that - but the JQ doesn't seem to kope well with the load. |
| 25 | 21:08:41 <gwicke> DanielK_WMDE: is this about queries? |
| 26 | 21:08:49 <DanielK_WMDE> mobrovac: no, the arztifact should be persistet. think parser cache. |
| 27 | 21:09:10 <robla> DanielK_WMDE: "aaron and ori don't like that...", I'm assuming you mean AaronSchulz, right? |
| 28 | 21:09:30 <DanielK_WMDE> gwicke: no, just tracking which rendering of a page (think parser cache key) uses which bits of which wikidata entity |
| 29 | 21:09:36 <DanielK_WMDE> robla: indeed |
| 30 | 21:10:12 <jzerebecki> DanielK_WMDE: the job queue doesn't cope well with cycles |
| 31 | 21:10:18 <DanielK_WMDE> gwicke: in case of the parser cache, we currently always purge all renderings if any of them needs purging. we havn't found a good meachnism to avoid that |
| 32 | 21:10:18 <subbu> DanielK_WMDE, what kind of stuff might get generated during GET reqs? that sounds non-ideal. |
| 33 | 21:10:21 <DanielK_WMDE> that's another open issue |
| 34 | 21:10:28 <mobrovac> ok, so it's not critical that the write happens during the life of the request, it just needs to be recorded somehow |
| 35 | 21:10:46 <robla> which question are we trying to answer here? |
| 36 | 21:10:48 <DanielK_WMDE> subbu: rendering in any of the possible user languages. multilingual wikis can't pre-generate all possible renderings on save. |
| 37 | 21:10:50 <matt_flaschen> DanielK_WMDE, why does that need to be updated on a GET request? I would think that dependencies only change on POST (either update to Wikidata or update to Lua module or update to Wikipedia article)? |
| 38 | 21:11:19 <DanielK_WMDE> matt_flaschen: we are tracking *renderings* of a page. they are generated on demand. |
| 39 | 21:11:46 <bd808> "tracking"? |
| 40 | 21:12:02 <bd808> like a log event or something else entirely? |
| 41 | 21:12:06 <DanielK_WMDE> matt_flaschen: if page Foo uses template X, the wikitext of Foo doesn't depends on X (it does reference X, but that's another issue). the HTML rendering depends on X (and on the wikitext of Foo) |
| 42 | 21:12:25 <DanielK_WMDE> bd808: like link tables. |
| 43 | 21:12:28 <matt_flaschen> DanielK_WMDE, right, i understand that, but the dependencies in that case only change on POST. |
| 44 | 21:12:29 <mobrovac> exaxtly the issue we have in RESTBase |
| 45 | 21:12:34 <matt_flaschen> This part from the RFC seems relevant: |
| 46 | 21:12:36 <matt_flaschen> "Our current approach of re-rendering all seven million articles can easily result in large backlogs of template updates. It might be useful to consider pull based or hybrid solutions (where only a timestamp is propagated and polled) as an alternative to pure push." |
| 47 | 21:12:37 <robla> #info robla asks for response on T102476#2296335. gwicke answers "I'm not opposed to moving the general portion especially to mw.org, lets just make sure we don't end up with multiple copies" |
| 48 | 21:12:37 <stashbot> T102476: RFC: Requirements for change propagation - https://phabricator.wikimedia.org/T102476 |
| 49 | 21:12:37 <DanielK_WMDE> but also for generated artifacts. possible stuff that is generated on demand |
| 50 | 21:12:42 <mobrovac> in that we care more about renders than wikitext |
| 51 | 21:13:09 <matt_flaschen> So maybe not all the HTML renderings should be pre-generated on save, but all the *dependencies* should be tracked on POST, and when to actually re-render is a different question, just as with templates. |
| 52 | 21:13:17 <DanielK_WMDE> matt_flaschen: the dependencies of the de rendering of page Foo are unknown until it is actually rendered for de output, and then cached. that happens on demand, during a get request. |
| 53 | 21:13:32 <DanielK_WMDE> matt_flaschen: image description pages have conditionals that depend on the user language. |
| 54 | 21:13:41 <DanielK_WMDE> not to speak of wikidata, where all of the output depends on the user language |
| 55 | 21:14:13 <DanielK_WMDE> matt_flaschen: we can't know the dependencies fro mthe wikitext. not even after resolvoing templates. |
| 56 | 21:14:16 <TimStarling> currently we have a canonical ParserOptions which is used to determine dependencies, and then those dependencies are used to purge all renderings |
| 57 | 21:14:25 <gwicke> DanielK_WMDE: I'm trying to understand how this is different from links, templates & media; is the main issue that dependencies differ between languages? |
| 58 | 21:14:34 <DanielK_WMDE> gwicke: yes. |
| 59 | 21:14:38 <TimStarling> even though renderings may have different dependencies to the canonical one |
| 60 | 21:14:44 <DanielK_WMDE> gwicke: which is already broken for links, templates, and media. |
| 61 | 21:14:57 <DanielK_WMDE> we don't track dependencies that only occurr in non-canonical renderings |
| 62 | 21:15:04 <gwicke> yeah, I was just going to say.. language variants are happily messing with that too |
| 63 | 21:15:08 <DanielK_WMDE> so we sometimes fail to purge |
| 64 | 21:15:50 <DanielK_WMDE> TimStarling: yea, currently, we just oignore the nastiness, because it's not very visible. but with better support for multilingual content, we need a better solution |
| 65 | 21:16:15 <TimStarling> there is a proposal in here to migrate all link tables to cassandra, is that correct? T105766 |
| 66 | 21:16:15 <stashbot> T105766: RFC: Dependency graph storage; sketch: adjacency list in DB - https://phabricator.wikimedia.org/T105766 |
| 67 | 21:16:24 <DanielK_WMDE> gwicke: i want to point to another open question: usage tracking gor 3rd parties. think InstantCommons. and perhaps InstantWikidata in the future. |
| 68 | 21:16:59 <gwicke> TimStarling: as I said on the task, it's largely theoretical & premature at this point |
| 69 | 21:18:24 <DanielK_WMDE> how abpout redis? |
| 70 | 21:18:35 <gwicke> DanielK_WMDE: we have structured some of the recent change propagation work around URLs, with a view to possibly supporting outside resources |
| 71 | 21:18:37 <DanielK_WMDE> i think this raises the question of how reliable our tracking needs to be |
| 72 | 21:19:01 <DanielK_WMDE> is it ok to have this transient? or should we be sure that it's persistent? |
| 73 | 21:19:02 <gwicke> but we have not tackled any dependency tracking so far |
| 74 | 21:19:29 <DanielK_WMDE> gwicke: 3rd party dependency tracking would basically require a pubsub service. but with very high granularity |
| 75 | 21:19:37 <DanielK_WMDE> it's not hard to do, but hard to scale |
| 76 | 21:19:38 <subbu> DanielK_WMDE, i am confused by your response to matt_flaschen's qn .. " the dependencies of the de rendering of page Foo are unknown until it is actually rendered for de output" .. that seems independent of when a page is rendered .. on a POST or on a GET, right? |
| 77 | 21:20:06 <subbu> how do you re-render all affected pages on a POST then? |
| 78 | 21:20:15 <subbu> or is that what you mean it is currently broken. |
| 79 | 21:20:18 <DanielK_WMDE> subbu: currect. so in some cases, we will only have this information in a GET request - and need to somehow store it |
| 80 | 21:20:26 <gwicke> DanielK_WMDE: to me, the other big question is how to structure interfaces for dependency updates, so that they are both usable & efficient |
| 81 | 21:20:55 <matt_flaschen> subbu, I think DanielK_WMDE's point is that it's computationally infeasible to determine the dependencies for all languages ahead of time. Since you need to parse it for each language just to determine what the dependencies of e.g. the Spanish language version are. |
| 82 | 21:21:12 <matt_flaschen> With e.g. Commons and {{int:}}, etc. |
| 83 | 21:21:38 <gwicke> the XKey Varnish work is heading in this direction as well |
| 84 | 21:21:50 <DanielK_WMDE> gwicke: well, basically, you need add(x,y), remove(x,y), lget(x), rget(y), lpurge(x), rpurge(y). Plus perhaps a batch interface. |
| 85 | 21:22:27 <gwicke> DanielK_WMDE: alternatively, you could post the dependencies each time & let the API figure out diffs |
| 86 | 21:22:47 <DanielK_WMDE> matt_flaschen: also, we only need that info if we actually have a rendering cached. if the page was never rendered in hebrew, i don't care what dependenceis the hebrew version might have |
| 87 | 21:23:19 <DanielK_WMDE> gwicke: yes. that would be part of the batch interface. |
| 88 | 21:23:35 <gwicke> DanielK_WMDE: could you describe an example for such language variance? |
| 89 | 21:23:45 <gwicke> which kind of content would be pulled in conditionally? |
| 90 | 21:24:11 <DanielK_WMDE> gwicke: all renderings of any item page on wikidata. all of it is language dependent. |
| 91 | 21:24:26 <DanielK_WMDE> gwicke: license templates on file description pages on commons |
| 92 | 21:24:36 <DanielK_WMDE> image descriptions on commons |
| 93 | 21:24:44 <DanielK_WMDE> (blame the translate extension) |
| 94 | 21:24:59 <matt_flaschen> https://commons.wikimedia.org/wiki/Template:LangSwitch |
| 95 | 21:25:05 <DanielK_WMDE> anything that uses {{int}} |
| 96 | 21:25:10 <DanielK_WMDE> matt_flaschen: indeed |
| 97 | 21:25:29 <matt_flaschen> DanielK_WMDE, for Wikidata, how are the dependencies language-dependent? Doesn't one page always depend on the same thing regardless of language. It's just the actual output would be different, but how is the dependency graph different? |
| 98 | 21:25:32 <gwicke> fun ;/ |
| 99 | 21:25:36 <matt_flaschen> For Q pages? |
| 100 | 21:25:50 <gwicke> for item pages, is there a concern beyond CDN purging? |
| 101 | 21:25:50 <matt_flaschen> Since Q pages don't have LangSwitch AFAIK |
| 102 | 21:26:11 <matt_flaschen> gwicke, it's also the ParserOutput itself, right? |
| 103 | 21:26:48 <DanielK_WMDE> matt_flaschen: in case of wikidata, the dependency graph is probably the same, that is true. that's not true for anything that uses {{int}} or <translate> (or variants) |
| 104 | 21:26:50 <gwicke> yeah, but afaik that's still keyed on the Q page & it's possible to purge all at once |
| 105 | 21:27:19 <gwicke> at first sight, item pages sound like a somewhat simpler case to me |
| 106 | 21:27:49 <DanielK_WMDE> matt_flaschen: but it depends on the granualitty. If Q1 references Q7, that will be the same for all languages. But the de-ch rendering of Q1 would epends on Q7.label.de, and the en rendering would depend on Q7.label.en |
| 107 | 21:28:10 <gwicke> conditional dependencies will always be tricky, and languages are not the only source of complexity there |
| 108 | 21:28:13 <TimStarling> gwicke: is work on this underway? which parts of the RFC need comment most urgently? |
| 109 | 21:28:13 <DanielK_WMDE> I'd like to have a system that can at least in theory handle this kind of thing. |
| 110 | 21:28:21 <gwicke> lua code can transclude things based on the phase of the moon |
| 111 | 21:28:37 <matt_flaschen> gwicke +1 on reference |
| 112 | 21:28:47 <DanielK_WMDE> heh :) |
| 113 | 21:29:14 <DanielK_WMDE> gwicke: they are not conditional. different artifacts depend on different artifacts. the dependency tracking doesn't care how or why. |
| 114 | 21:29:35 <gwicke> TimStarling: work is underway on eventbus & changeprop, but for dependency tracking we are mostly trying to better understand the issues at this point |
| 115 | 21:29:37 <DanielK_WMDE> if you have a uri for "phase of the moon" and you touch it every day, the dependency management should be able to handle this |
| 116 | 21:29:58 <DanielK_WMDE> uri = artifact, here |
| 117 | 21:30:00 <gwicke> TimStarling: this discussion is very helpful in that regard |
| 118 | 21:30:23 <matt_flaschen> DanielK_WMDE, that could be abstracted, though. You could say Q1[de-ch] depends on Q7.label[de-ch], and Q7.label[de-ch] depends on Q7.actual-label[de-ch] and Q7.actual-label[de]. |
| 119 | 21:30:26 <robla> gwicke: which question are you hoping to get consensus on? |
| 120 | 21:30:55 <gwicke> robla: I wasn't hoping for any decisions in this meeting |
| 121 | 21:30:59 <matt_flaschen> With label depending on which if any actual-label are available. |
| 122 | 21:31:09 <TimStarling> robla: I think this is more requirements gathering |
| 123 | 21:31:19 <DanielK_WMDE> matt_flaschen: yes, exactly. in case of wikibase with nice structured data, it would probably be feasible to do this in advance. for wikitext, it isn't - so we don't know until we render on demand. |
| 124 | 21:32:20 <DanielK_WMDE> TimStarling, robla: since the entire rfc is about gathering requirements, it's kind of meta... how could it ever be "approved" or "implemented"? what does that mean? |
| 125 | 21:32:25 * robla is trying to figure out if/when he should be capturing any of this with #info commands, and can't figure out how to make that useful |
| 126 | 21:33:10 <DanielK_WMDE> i have tried to formulate the requirements we have for wikidata. i can try and put that into #info tags, but i try no to do that too often with my own comments... |
| 127 | 21:33:23 <robla> DanielK_WMDE: I think that's why I'm hoping to move it to MediaWiki. The finished RFC can be a clear description of the options. |
| 128 | 21:33:42 <DanielK_WMDE> robla: right, but the rfc process seems useful for gathering requirements |
| 129 | 21:33:47 <TimStarling> #info significant use: track dependencies when rendering pages with {{int}} and <translate> so that they can be purged when conditional dependencies change |
| 130 | 21:33:53 <matt_flaschen> The biggest open question I see is how to handle the case where the dependency graph itself varies depending on user language. |
| 131 | 21:34:07 <matt_flaschen> Sorry, didn't see Tim's #info before saying that. |
| 132 | 21:34:14 <robla> DanielK_WMDE: gathering them in your head? |
| 133 | 21:34:34 <gwicke> I'm not sure that this is a problem we can or should try to solve in general |
| 134 | 21:34:47 <DanielK_WMDE> matt_flaschen: just thing of renderings as natural parts of the dependency graph. that should Just Work (tm). |
| 135 | 21:34:53 <DanielK_WMDE> *think |
| 136 | 21:35:17 <gwicke> unpredictable dependencies are always going to be a pain to handle, and even if we fully handled them, it would be fairly expensive to do so |
| 137 | 21:35:18 <DanielK_WMDE> one consequence is actually: wikitext *never* depends on wikitext. |
| 138 | 21:35:21 <matt_flaschen> DanielK_WMDE, yeah, I'm trying to get my mind around that perspective. |
| 139 | 21:35:24 <DanielK_WMDE> you never purge wikitext |
| 140 | 21:35:38 <TimStarling> I think MW's link tables need to be moved out of the core DBs fairly urgently, I've thought so for years |
| 141 | 21:35:46 <DanielK_WMDE> what needs purging is *gernerated* content, so we need to track what it depends on |
| 142 | 21:36:29 <TimStarling> I guess it is fine to start with a new thing like shadow namespaces, but the existing LinksUpdate system is not so awesome that I would want to see it preserved |
| 143 | 21:36:42 <DanielK_WMDE> TimStarling: i think there are two use cases for those: dependency tracking for puring (which should be moved and improved), and tracking references (in whatlinshere, related changes, etc) for maintenance (which we should keep, i think=) |
| 144 | 21:36:45 <matt_flaschen> TimStarling, could you explain about moving the links tables out? |
| 145 | 21:36:49 <gwicke> one work-around for the unpredictable dependency problem is to eliminate the direct depenency by a) making things modular, or b) rendering things dynamically on the client |
| 146 | 21:37:18 <DanielK_WMDE> TimStarling: i think we need to consider these two things separately. with more dynamic content, we can no longer treat them the same |
| 147 | 21:37:33 <TimStarling> matt_flaschen: the *links tables are large and heavily-updated, it's not scalable |
| 148 | 21:37:47 <DanielK_WMDE> #info what needs purging is *gernerated* content, so we need to track what it depends on |
| 149 | 21:38:51 <TimStarling> DanielK_WMDE: yes, they can be split up |
| 150 | 21:38:56 <DanielK_WMDE> gwicke: but in order to do that, i need to know the dependencies, right? the client somehow needs to know what resources to load. |
| 151 | 21:39:11 <TimStarling> maybe we can stop putting entries in whatlinkshere when someone uses #ifexist |
| 152 | 21:39:14 <DanielK_WMDE> TimStarling: i think that is actually the cure issue this rfc is about |
| 153 | 21:39:18 <DanielK_WMDE> *core |
| 154 | 21:39:24 <gwicke> DanielK_WMDE: yes, but the response of each part can vary independently, which avoids a lot of dependencies to the composite |
| 155 | 21:39:54 <DanielK_WMDE> it avoids materializing the composite |
| 156 | 21:40:07 <DanielK_WMDE> the dependencies that need tracking are the same |
| 157 | 21:40:09 <Scott_WUaS> (DanielK_WMDE, matt_flaschen, gwicke: re https://commons.wikimedia.org/wiki/Template:LangSwitch - long term question - in what ways could this be extended to include translation from WMF Content Translation as well as later with Wiktionary - and in combination conceptually with Google Translate?) |
| 158 | 21:40:19 <TimStarling> well, the RFC is requirements gathering, so I am stating my support for a use case which is on the list of things this could solve |
| 159 | 21:40:26 <gwicke> DanielK_WMDE: no, you no longer need to track the dependency to the composite |
| 160 | 21:40:33 <gwicke> instead, you can just purge one component |
| 161 | 21:40:49 <gwicke> & have it update wherever that component is pulled in dynamically |
| 162 | 21:41:10 <gwicke> the same way CSS can be updated without re-rendering everything, or tracking dependencies |
| 163 | 21:41:33 <DanielK_WMDE> Scott_WUaS: this is one small step towards allowing people to view pages in their favorite language without logging in. but it's fairly technical/low level, not directly related to translation. |
| 164 | 21:41:39 <Scott_WUaS> (leading eventual very sophisticated machine translation with artificial intelligence re natural language processing in extending WMF ~300 languages to all languages even?) Thanks! |
| 165 | 21:41:52 <gwicke> of course, this only works with reasonable efficiency for a limited number of components |
| 166 | 21:41:54 <Scott_WUaS> DanielK_WMDE: Thanks |
| 167 | 21:42:37 <DanielK_WMDE> gwicke: the dependencies still have to be known somewhere. but perhaps they can just live on the client |
| 168 | 21:43:23 <gwicke> DanielK_WMDE: the difference is that you only need to store one edge (the reference), and don't need to track the backwards edge |
| 169 | 21:43:41 <DanielK_WMDE> ah, yea... i guess you are right |
| 170 | 21:43:41 <gwicke> it's an example of polling |
| 171 | 21:43:50 <AaronSchulz> 2:25 PM <gwicke> for item pages, is there a concern beyond CDN purging? |
| 172 | 21:43:55 <AaronSchulz> what's the answer to that? |
| 173 | 21:44:20 <DanielK_WMDE> gwicke: anyway - do you agree that the main issue is tracking dynamically generated artifacts, in order to determin when they need to be re-generated? and the proposal is to no longer try to do this based on link tables? |
| 174 | 21:44:21 * AaronSchulz is curious about xkey too |
| 175 | 21:45:07 <AaronSchulz> if per language renderings where bucketed in cdn, they could be purged via a single URL purge, without tracking each language variant, right? |
| 176 | 21:45:08 <DanielK_WMDE> AaronSchulz, gwicke: about items pages and cdn purging: yes. that's why anons can only view wikidata in english. |
| 177 | 21:45:33 <gwicke> DanielK_WMDE: that is the general problem, yes -- but I think we'll need to work on this from both sides: a) avoiding some dependencies by making things more modular, and b) improving our infrastructure for tracking dependencies |
| 178 | 21:45:46 <AaronSchulz> avoiding the need to track new renderings on GET |
| 179 | 21:46:33 <gwicke> AaronSchulz: for item pages, even a Vary might be able to do it right now |
| 180 | 21:46:40 <TimStarling> "anons can only view wikidata in english" -- what a stupid problem for a project like ours to have |
| 181 | 21:46:45 <matt_flaschen> AaronSchulz, but how do you know which pages to purge, if it varies by language what the dependencies are? |
| 182 | 21:46:59 <TimStarling> embarrassing, we should fix that |
| 183 | 21:47:03 <DanielK_WMDE> AaronSchulz: related to the cdn issue: https://phabricator.wikimedia.org/T114662 |
| 184 | 21:47:10 <AaronSchulz> gwicke: yeah, if we validate the language and 500 on bogus ones, that limits the hash-chain to only a few 100s of possibilities |
| 185 | 21:47:10 <matt_flaschen> Also, regarding the CDN question, parser cache itself also needs to be purged. |
| 186 | 21:47:36 <gwicke> AaronSchulz: yeah, only a teeny bit of fragmentation ;) |
| 187 | 21:47:41 <AaronSchulz> once you hit MW, there are lots of ways to handle validating variant caches of a single source |
| 188 | 21:48:10 <AaronSchulz> (e.g. checking page_touched, some other field, WAN cache check keys,...) |
| 189 | 21:48:21 <DanielK_WMDE> AaronSchulz: no, we would still need that info, so we can purge the parser cache |
| 190 | 21:48:27 <matt_flaschen> AaronSchulz, never mind, "varies by language what the dependencies are" doesn't apply to Wikidata Q pages. |
| 191 | 21:48:59 <AaronSchulz> DanielK_WMDE: why? |
| 192 | 21:49:13 <DanielK_WMDE> AaronSchulz: also, we want selective purging. if only the french rendering depends on resource X, and X changes, it would be nice if we could purge only the french version. |
| 193 | 21:49:19 <DanielK_WMDE> not all of them |
| 194 | 21:49:36 <AaronSchulz> "would be nice" or "actually is worth it"? |
| 195 | 21:49:46 <DanielK_WMDE> AaronSchulz: you'd need multi-dimensional buckets. not just a bucket for "all renderings of Foo", but also a bucket for "all renderings that depends on X" |
| 196 | 21:49:56 <DanielK_WMDE> both on the CDN, and for the parser cache |
| 197 | 21:50:12 <DanielK_WMDE> (that's actually closely related to the recently rejected PSR6 proposal) |
| 198 | 21:50:39 <DanielK_WMDE> https://phabricator.wikimedia.org/T130528 |
| 199 | 21:50:39 <AaronSchulz> what is X? like magic words and templates that vary on language? |
| 200 | 21:50:59 <DanielK_WMDE> AaronSchulz: a template, or a wikidata item. or the phase of the moon. whatever the rendering depends on |
| 201 | 21:51:35 <AaronSchulz> is there a list of concrete use cases for wikidata? |
| 202 | 21:51:46 <DanielK_WMDE> AaronSchulz: the fact that only the french rendering depends on X would be due to {{langswitch}} or <translate> or something, yes |
| 203 | 21:51:59 <AaronSchulz> for MW core, I'd rather discourage/deprecate stuff like that (use lower TTLs where it is needed) |
| 204 | 21:52:25 <DanielK_WMDE> AaronSchulz: the one that has the most need for language dependant tracking is file description pages on commons. |
| 205 | 21:52:39 <DanielK_WMDE> that'S why i implemented on-render usage tracking for entities. |
| 206 | 21:52:54 <matt_flaschen> DanielK_WMDE, couldn't that be solved (the same way it's non-dynamic for Q pages like discussed above) once Commons uses Wikibase? |
| 207 | 21:53:03 <matt_flaschen> To avoid the "actual dependency graph varies by language" issue? |
| 208 | 21:53:08 <matt_flaschen> At least for file description pages. |
| 209 | 21:53:11 <gwicke> there are a lot of trade-offs here between cost of dependency tracking, cost of purges, accuracy of tracking non-deterministic dependencies etc |
| 210 | 21:53:13 <DanielK_WMDE> AaronSchulz: well, if i understand gwickes vision correctly, he would like to habve a *lot* more of this kind of thing. |
| 211 | 21:53:21 <AaronSchulz> it adds a lot of complexity to go from just varying on rendering of entities/pages to using different helper entities/pages to render an asset/entity |
| 212 | 21:53:48 <DanielK_WMDE> matt_flaschen: once commons *only* uses wikibase, and doesn't generate wikitext from it - then yes, to an extend. |
| 213 | 21:54:00 <DanielK_WMDE> i expect that transition period to take about 5 to 10 years. |
| 214 | 21:54:15 <gwicke> for wikipedia, we kind of track dependencies per language already by virtue of having one project per language |
| 215 | 21:54:26 <AaronSchulz> heh, another decade anniversary cake ;) |
| 216 | 21:55:09 <DanielK_WMDE> AaronSchulz: this rfc is about a generic mechanism to allow this kind of tracking, not just for wikidata, but for all kinds of content. this allows us to re-use rendered snippets/widgets, and purge them when appropriate |
| 217 | 21:55:17 <TimStarling> you know {{int:}} varying by user language was an accident |
| 218 | 21:55:31 <DanielK_WMDE> gwicke: indeed. the language issue arises for multilingual pages. |
| 219 | 21:55:38 <DanielK_WMDE> TimStarling: yea :D |
| 220 | 21:55:43 <TimStarling> if I hadn't made that implementation error then commons would have been stuck with JS hacks to hide languages other than the current one |
| 221 | 21:55:51 <TimStarling> which is what they did to start with |
| 222 | 21:55:52 <gwicke> or CSS |
| 223 | 21:56:16 <DanielK_WMDE> they would just have extended <translate> to cover this |
| 224 | 21:57:16 <gwicke> my personal inclination is still to reduce the reasons for such variance in the content |
| 225 | 21:57:45 <gwicke> but it's clear that there are so many sources and use cases for this already that it won't be possible to avoid it altogether |
| 226 | 21:58:11 <AaronSchulz> right |
| 227 | 21:58:36 <TimStarling> any action items or #info for the notes before we wrap up? |
| 228 | 21:59:03 <gwicke> I want to thank you all for participating |
| 229 | 21:59:16 <Scott_WUaS> (Glad for this focus on language and translation) |
| 230 | 21:59:25 <DanielK_WMDE> I want to thank you for listening to my rants ;) |
| 231 | 21:59:33 <TimStarling> next week we don't have a particular RFC scheduled, sowea triage session |
| 232 | 21:59:33 <mobrovac> one action would seem to be to clarify the anticipated concrete dependency relations |
| 233 | 21:59:42 <mobrovac> lang variants seem to be the most painful point so far |
| 234 | 21:59:44 <gwicke> this kind of discussion is a perfect reminder of all the tricky issues that we are trying so hard to forget |
| 235 | 21:59:58 * robla looks up link to next week |
| 236 | 22:00:00 <TimStarling> s/sowea/so we are planning on doing a |
| 237 | 22:00:29 <gwicke> mobrovac: yes, along with update volume & suitable APIs |
| 238 | 22:00:30 <robla> next week: https://phabricator.wikimedia.org/E187 |
| 239 | 22:00:43 <robla> as TimStarling said, it's a triage |
| 240 | 22:00:52 <TimStarling> mobrovac: that is an action item for gwicke? |
| 241 | 22:01:21 <mobrovac> i'd say this is an action item for all of us that want to get the most out of this |
| 242 | 22:01:34 <gwicke> yeah, I think we'll look into this further as a team |
| 243 | 22:01:35 <mobrovac> the clearer the problem, the simpler the solution as always |
| 244 | 22:01:38 <TimStarling> #action update the RFC to clarify the anticipated concrete dependency relations |
| 245 | 22:02:18 <gwicke> we might also want to split out the dependency tracking part |
| 246 | 22:02:24 <TimStarling> I think it's usually best if an action item is assigned to a single person, since shared responsibility is equivalent to no responsibility |
| 247 | 22:02:41 <TimStarling> you know the bystander effect |
| 248 | 22:03:06 <gwicke> TimStarling: I can be that person, but mobrovac is leading changeprop development, so is heavily involved |
| 249 | 22:03:16 <gwicke> along with pchelolo |
| 250 | 22:03:22 <TimStarling> ok |
| 251 | 22:03:45 <TimStarling> #endmeeting |