We are going to look into that and see how bad this is.

I think we should only do that if the data is not there before... that only happens once per page after the initial entity usage deploy.

To achieve that we should compare the MAX( eu_touched ) (from slave) with the touched timestamp we get from the ParserOutput or Title given to us by the ParserCacheSaveComplete hook. If these match up, we don't need to do any updates.

@hoo: we need to update usage tracking whenever a new target language is viewed after a page was touched. Relying on the modified date sadly doesn't work, because it's not set when the update is triggered via the job queue (learned that the hard way).

In T103429#1416787, @daniel wrote:

@hoo: we need to update usage tracking whenever a new target language is viewed after a page was touched. Relying on the modified date sadly doesn't work, because it's not set when the update is triggered via the job queue (learned that the hard way).

Ok, that makes sense… in that case I guess we should try to compare the data from slave with what we want to write first. If it matches, almost certainly no changes are needed.

Would that defuse the problem enough?

Also would my original idea work if the client setting allowDataAccessInUserLanguage is set to false? If so, that might be worth special casing (that would work for the ParserCacheSaveComplete hook which is more important here, at least).

Change 222463 had a related patch set uploaded (by Daniel Kinzler):
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/222463

The patch above makes the update always go via the job queue. Not sure whether that is the Right Thing.

Change 222463 had a related patch set uploaded (by Aude):
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/222463

Change 222463 merged by Aude:
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/222463

Change 227632 had a related patch set uploaded (by Aude):
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/227632

Change 227634 had a related patch set uploaded (by Aude):
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/227634

Change 227632 merged by Aude:
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/227632

Change 227634 merged by jenkins-bot:
On page views, update usage tracking via the job queue.

https://gerrit.wikimedia.org/r/227634

In T103429#2127044, @ori wrote:

Doing this via the job queue is not a good solution, in my opinion. There is something fundamentally broken about tying updates to parser cache events.

Why? We are indeed trying to track which parser cache entry uses what information from wikidata. No more and no less.

Subscribing to changes via a ParserCacheSaveComplete hook handler is a bit like a department store updating its inventory any time a customer touches an item: sure, it covers all the relevant transactions, but it also creates a large amount of unnecessary work, because customers may handle an item for a number of reasons (to try it on or read the label), not all of which result in a transaction.

We only track if the parser actually accesses the data item. That is, when the thing in the parser cache actually depends on the data item.

ParserCacheSaveComplete events are the same way: they cover all the cases in which an edit is made, but they also fire in cases which do not involve a transaction. For example, the mobile and desktop web sites use different key-spaces to avoid polluting each other's parser cache entries with platform-specific artifacts, so there are at least two ParserCacheSaveComplete events for each edit.

Hooking into ParserCacheSaveComplete is the wrong thing to do, because (AIUI) it isn't the really the parser cache that Wikibase cares about, and because it muddles the distinction between read-only and read/write requests.

It's exactly the ParserCache that Wikibase cares about. We need to know which information from Wikidata has been used to construct HTML that is in the ParserCache, so we can purge the cache when the data changes. I don't see any other use case. Do you have an alternative suggestion that would achieve this? Once we have T105766: RFC: Dependency graph storage; sketch: adjacency list in DB we will no longer need this, but even then, we will need a place to store dependencies between a generated resource and whatever it was generated from.

When do you think does this generated unnecessary work? Our tracking is fairly fine grained, provided people use the parser function and Lua module in a sane way. We'll not purge a page that uses a label of Q123 when a sitelink on Q123 changes, etc.

One alternative I can think of is to store the tracking information in the parser cache itself, instead of the database. But there are two problems with that:

• the tracking info must never expire before the actual cache entry
• we need to be able to query usages by item. The parser cache is keyed per local page.

So, until we have T105766: RFC: Dependency graph storage; sketch: adjacency list in DB, I don't see an alternative.

@ori Could you please answer Daniel's questions? This task is high priority, yet it's not actionable.

Slightly related T124737.