Page MenuHomePhabricator

[Epic] Store media information for files on Wikimedia Commons as structured data
Open, NormalPublic

Tokens
"Like" token, awarded by Liuxinyu970226."Mountain of Wealth" token, awarded by SandraF_WMF."Love" token, awarded by Mattias_Ostmar-WMSE."Like" token, awarded by Sadads."Like" token, awarded by Deskana."Like" token, awarded by Jdforrester-WMF."Love" token, awarded by Smalyshev."Mountain of Wealth" token, awarded by Bene."Like" token, awarded by Filceolaire."Love" token, awarded by Ricordisamoa.
Assigned To
None
Authored By
Jdforrester-WMF, Jun 4 2014

Description

Adding structured data based on Wikibase for all media files on Wikimedia Commons is something the Multimedia team are planning to work on with the Wikidata team.

Details

Reference
bz66108

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusAssignedTask
OpenNone
OpenNone
ResolvedAbit
OpenNone
OpenNone
DuplicateNone
DuplicateNone
InvalidNone
InvalidNone
DuplicateNone
ResolvedLydia_Pintscher
InvalidNone
InvalidNone
InvalidNone
InvalidNone
InvalidNone
OpenNone
ResolvedNone
OpenNone
ResolvedLydia_Pintscher
Resolveddaniel
ResolvedTobi_WMDE_SW
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedJdforrester-WMF

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Yann added a subscriber: Yann.Feb 7 2016, 4:10 PM
Meno25 removed a subscriber: Meno25.Feb 8 2016, 7:28 PM
Bene renamed this task from Store media information for files on Wikimedia Commons as structured data to [Epic] Store media information for files on Wikimedia Commons as structured data.
Bene awarded a token.
Jheald added a subscriber: Jheald.Mar 1 2016, 7:26 PM
Smalyshev added a subscriber: Smalyshev.
DannyH added a subscriber: DannyH.Mar 2 2016, 4:36 PM
Deskana added a subscriber: Deskana.
-jem- added a subscriber: -jem-.Mar 28 2016, 5:34 PM
Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 13 2016, 4:25 PM
Sadads added a subscriber: Sadads.Apr 13 2016, 4:25 PM
matej_suchanek updated the task description. (Show Details)
matej_suchanek removed a project: Multimedia.
matej_suchanek removed a subscriber: wikibugs-l-list.
Sadads moved this task from Backlog to Commons + GLAM Open on the GLAM-Tech board.Jul 6 2016, 2:00 PM
Elitre added a subscriber: Elitre.Jan 23 2017, 10:51 AM

T159884 maybe a use case for this at some point (mentioning for bookkeeping)

DannyH removed a subscriber: DannyH.Mar 9 2017, 9:00 PM
Krinkle removed a subscriber: Krinkle.Mar 18 2017, 3:31 AM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 10 2017, 7:47 PM
SandraF_WMF added a subscriber: SandraF_WMF.

Mentioned in SAL (#wikimedia-operations) [2019-01-09T21:04:30Z] <James_F> Creating Wikibase repo tables on Commons for T68108

jcrespo added subscribers: Marostegui, jcrespo.EditedApr 2 2019, 4:32 PM

I've been told wikibase tables have been created on s4. We would like to have been notified of this- we are not sure wikibase for commons should live on s4, if the growth of structured data is as large of it was for wikidata, we should create a separate cluster, dedicated to it (s4, like s1 and s8 are quite bloated). Changing it before or at the beginning is easy, doing it later is more complicated. Please talk to DBAs to understand hw needs. I prefer to request more hw that we need and later not buy it that needing it and not having the budget available. CC @Marostegui.

I asked to talk to us months before that deployment happened.

@WMDE-leszek - I spoke with @jcrespo and some others about potential issues with the SDC deployments, and we basically decided that it would be best to move the Wikibase tables to a separate cluster out of an abundance of caution. They wanted a WMDE perspective as to whether making that move would be possible and what effects might be seen based on the various refactoring going on, in particular with the wb_terms table. Given you've been our point of contact on Wikibase work recently, I hoped you could chime in or bother the correct person to give feedback on this matter.

It should be noted here, also, that the Wikibase tables on the Commons database are currently all-but-empty, so it's not a huge threat to the cluster. However, we're investigating the impact on the revisions table and will be exploring what is needed to avoid further issues.

@MarkTraceur, @jcrespo et al: We've discussed this briefly at WMDE, and we believe the suggested idea should not be problematic with regards to wb_terms table.
Neither the existing wb_terms table, neither its refactored replacement is expected to be used in joins with "standard" MW tables, hence moving the table to separate cluster shouldn't create an issue. The move would likely require some changes to the Wikibase code, so it would be preferred from WMDE side, that the separation happened after our current storage work is done (1-2 months from now), so we can limit number of variables we deal with.

It should be noted wb_terms (or what is about to replace it) is not the only DB table created and used by Wikibase extension. These other tables (e.g. wbc_changes) couldn't, in our understanding, be easily moved to a separate cluster, and we'd recommend against such move, unless there are important reasons to do so.

It is not clear from the comments above, whether the separation you have been discussing only considers wb_terms table, or all Wikibase-specific tables.