Page MenuHomePhabricator

[Epic] Store media information for files on Wikimedia Commons as structured data
Open, NormalPublic

Tokens
"Like" token, awarded by Liuxinyu970226."Mountain of Wealth" token, awarded by SandraF_WMF."Love" token, awarded by Mattias_Ostmar-WMSE."Like" token, awarded by Sadads."Like" token, awarded by Deskana."Like" token, awarded by Jdforrester-WMF."Love" token, awarded by Smalyshev."Mountain of Wealth" token, awarded by Bene."Like" token, awarded by Filceolaire."Love" token, awarded by Ricordisamoa.
Assigned To
None
Authored By
Jdforrester-WMF, Jun 4 2014

Description

Adding structured data based on Wikibase for all media files on Wikimedia Commons is something the Multimedia team are planning to work on with the Wikidata team.

Details

Reference
bz66108

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusAssignedTask
OpenNone
OpenNone
ResolvedAbit
OpenNone
OpenNone
DuplicateNone
DuplicateNone
InvalidNone
InvalidNone
DuplicateNone
ResolvedLydia_Pintscher
InvalidNone
InvalidNone
InvalidNone
InvalidNone
InvalidNone
OpenNone
ResolvedNone
OpenNone
ResolvedLydia_Pintscher
Resolveddaniel
ResolvedTobi_WMDE_SW
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedJdforrester-WMF
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Yann added a subscriber: Yann.Feb 7 2016, 4:10 PM
Meno25 removed a subscriber: Meno25.Feb 8 2016, 7:28 PM
Bene renamed this task from Store media information for files on Wikimedia Commons as structured data to [Epic] Store media information for files on Wikimedia Commons as structured data.Mar 1 2016, 3:09 PM
Bene awarded a token.
Jheald added a subscriber: Jheald.Mar 1 2016, 7:26 PM
Smalyshev added a subscriber: Smalyshev.
DannyH added a subscriber: DannyH.Mar 2 2016, 4:36 PM
Deskana added a subscriber: Deskana.
-jem- added a subscriber: -jem-.Mar 28 2016, 5:34 PM
Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 13 2016, 4:25 PM
Sadads added a subscriber: Sadads.Apr 13 2016, 4:25 PM
matej_suchanek updated the task description. (Show Details)
matej_suchanek removed a project: Multimedia.
matej_suchanek removed a subscriber: wikibugs-l-list.
Sadads moved this task from Backlog to Commons + GLAM Open on the GLAM-Tech board.Jul 6 2016, 2:00 PM
Elitre added a subscriber: Elitre.Jan 23 2017, 10:51 AM

T159884 maybe a use case for this at some point (mentioning for bookkeeping)

DannyH removed a subscriber: DannyH.Mar 9 2017, 9:00 PM
Krinkle removed a subscriber: Krinkle.Mar 18 2017, 3:31 AM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 10 2017, 7:47 PM
SandraF_WMF added a subscriber: SandraF_WMF.

Mentioned in SAL (#wikimedia-operations) [2019-01-09T21:04:30Z] <James_F> Creating Wikibase repo tables on Commons for T68108

jcrespo added subscribers: Marostegui, jcrespo.EditedApr 2 2019, 4:32 PM

I've been told wikibase tables have been created on s4. We would like to have been notified of this- we are not sure wikibase for commons should live on s4, if the growth of structured data is as large of it was for wikidata, we should create a separate cluster, dedicated to it (s4, like s1 and s8 are quite bloated). Changing it before or at the beginning is easy, doing it later is more complicated. Please talk to DBAs to understand hw needs. I prefer to request more hw that we need and later not buy it that needing it and not having the budget available. CC @Marostegui.

I asked to talk to us months before that deployment happened.

@WMDE-leszek - I spoke with @jcrespo and some others about potential issues with the SDC deployments, and we basically decided that it would be best to move the Wikibase tables to a separate cluster out of an abundance of caution. They wanted a WMDE perspective as to whether making that move would be possible and what effects might be seen based on the various refactoring going on, in particular with the wb_terms table. Given you've been our point of contact on Wikibase work recently, I hoped you could chime in or bother the correct person to give feedback on this matter.

It should be noted here, also, that the Wikibase tables on the Commons database are currently all-but-empty, so it's not a huge threat to the cluster. However, we're investigating the impact on the revisions table and will be exploring what is needed to avoid further issues.

@MarkTraceur, @jcrespo et al: We've discussed this briefly at WMDE, and we believe the suggested idea should not be problematic with regards to wb_terms table.
Neither the existing wb_terms table, neither its refactored replacement is expected to be used in joins with "standard" MW tables, hence moving the table to separate cluster shouldn't create an issue. The move would likely require some changes to the Wikibase code, so it would be preferred from WMDE side, that the separation happened after our current storage work is done (1-2 months from now), so we can limit number of variables we deal with.

It should be noted wb_terms (or what is about to replace it) is not the only DB table created and used by Wikibase extension. These other tables (e.g. wbc_changes) couldn't, in our understanding, be easily moved to a separate cluster, and we'd recommend against such move, unless there are important reasons to do so.

It is not clear from the comments above, whether the separation you have been discussing only considers wb_terms table, or all Wikibase-specific tables.

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

Wikibase-specific tables

there is wbc_changes, and maybe other wikibase client tables- we don't have a problem with those, as those exist locally on all (wikidata-enabled) wikis. The ones we are worried about are the wikibase server ones (aka the equivalent of s8 on s4). I am not sure we should wait for the refactoring, but as long as the tables on s4 are empty or almost empty (with I think a single row on wb_id_counters), we are flexible. What we don't want is lots of data there that later is more complex to migrate away (we are assuming there will be a large amount of updates there when SDC is at full steam).

A few hosts were budgeted for the split for FY2019-2020, we are not in a rush, but it would be nice to have some planning in place for next fiscal year so there are no unexpected delays, specially given persistence team is at 50% capacity ATM.

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

What I had in mind is wb_terms (and its successor being introduced currently) are also used in queries that do joins with Mediawiki's "standard" tables, e.g. page or revision. This code/queries will not continue to work if the wb_terms table of commons gets moved to another server than the one mw tables are.

The move would likely require some changes to the Wikibase code

Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources.

What I had in mind is wb_terms (and its successor being introduced currently) are also used in queries that do joins with Mediawiki's "standard" tables, e.g. page or revision. This code/queries will not continue to work if the wb_terms table of commons gets moved to another server than the one mw tables are.

I was just corrected by the WMDE colleague the above point about joins is not correct, there are no such joins

I believe the only code change that Wikibase might be facing might then be a new to have a DB connection for wb_terms table server, and the DB connection for the regular mw tables server. This is not going to be rocket science, but also not something that would just work out of the box.

So just to be clear- based on growth projections I recently got from SDC team of wikibase on commons, the separation is not only convenient or needed for performance, literally s4 would not be able to fit except the initial deployment of data, or do it for very short term. Disk usage is close to 2 TB right now, with and additional 2TB of structured data (metadata only). IOPS would be close to wikidata, which requires a dedicated cluster. This is based on their statistics and projected growth, on top of the current growth and usage. We can scale those out with already budgeted hw, but we need software support. Extra content data would not be a concern as we already planned ES expansion for next fiscal.

I think we should plan a bit the different high level tasks related to the database, to schedule them properly, both the ones that directly affect SDC (implementation, wikibase dedicated db, wb_terms migration, MCR) and the ones that affect it indirectly by increase or release of used resources (actor, comment, *links migrations).