I updated the ORES utility to 1.2.0 and rerun. It ran for a little bit and then hung for 20 minutes. I then hit control-c and received the following traceback:

@Ladsgroup: I believe this is the only related open patch right now: https://gerrit.wikimedia.org/r/#/c/355104/. Is that right, @hoo?

@hoo: elwiki sounds good to me.

@jcrespo: Related to your first comment, the patch that I provided a link to (in my comment from one month ago) is now invalid since we are no longer planning on using a separate table in order to implement statement tracking. A new patch should be ready soon implementing statement tracking with wbc_entity_usage.

@jcrespo: We have decided to update the existing wbc_entity_usage table in order to allow for statement tracking (probably done via the "eu_aspect_id" field). We would do an initial deployment to a medium-sized Wikipedia at first and calculate database IO load. We will be able to undo any database changes made in this initial deployment, it will not affect current tracking being done in wbc_entity_usage. Three medium-sized Wikipedias that @hoo and I have considered for this deployment are :

@madhuvishy Yes, the file is there now. Thank you!

@jcrespo: Asking my question on here per your request.

To follow up on @Halfak's database usage assessments, the estimate of 5 properties per entity/page relationship seems reasonable and conservative since the average number of statements per entity is in fact ~5 as seen here: https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel-statements (its worth noting the average is increasing). This assumes that most of the time, a Wikipedia page is referencing data from just its corresponding Wikidata entity.

If we choose to go the database route instead of EventLogging, my initial thoughts would be to create a new table containing (modeled after @Halfak's reference and the current entity usage documentation: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/docs/usagetracking.wiki):

• page_id(UNSIGNED INT) -- Wikipedia page using the entity's data
• entity_id(UNSIGNED INT) -- the entity id
• property_id(UNSIGNED INT) -- the property used by the entity

I'd be happy to take the lead on the implementation of this task. Would that be okay?

I think a good way to go about this tracking would be by overriding the Lua direct access method for the table that represents the entity.