Page MenuHomePhabricator
Create Task
Maniphest T159851

Add a column for full entity ID to wb_terms table
Closed, ResolvedPublic
Actions

Description

As suggested in T159718 a column storing the full entity ID (i.e. its string form, not just a numeric part) should be added to wb_terms table.

https://gerrit.wikimedia.org/r/341322 is a draft of the patch adding column like the one considered here. That patch has been only introduced to illustrate the intent. While actually working on this it could either a separate patch (or patches), or that patch might be taken over and changed to the needed form. https://gerrit.wikimedia.org/r/341322 should not be considered something to follow.

Note that in T159718 DBAs suggested to introduce indexes including the new column at the point when the column is added. I.e. unlike @WMDE-leszek initially considered T159718, or it was proposed in the plan in T114903. This decision should be considered here.

Side note: It might be outside of scope of this task but when actually requesting the schema change, make sure to conform to https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change.

Details

SubjectRepoBranchLines +/-
Don't let Wikibase instances read/write terms_full_entity_idoperations/mediawiki-configmaster+3 -0
Writing full IDs to wb_terms on insertmediawiki/extensions/Wikibasemaster+98 -5
Add column for full entity ID to Terms DB tablemediawiki/extensions/Wikibasemaster+22 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclineddchenT118706 Conduct heuristic evaluation of image upload and insert flow in VisualEditor
 OpenNoneT115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
 OpenNoneT115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
 DuplicateNoneT115864 Figure out if the description of the image can be used as the caption on-wiki
 OpenFeatureNoneT53032 When inserting an image, set its caption by default to be the Commons image description
 OpenFeatureNoneT39534 Wikimedia Commons should support searching by color
 DuplicateNoneT39535 Wikimedia Commons should support filtering by color
 ResolvedNoneT19503 Provide metadata support on Wikimedia Commons
 ResolvedNoneT51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
 ResolvedNoneT68108 [Epic] Store media information for files on Wikimedia Commons as structured data
 DuplicateNoneT66288 basic support for structured data on mediawiki files
 InvalidLydia_PintscherT76012 make use of new entity type for multimedia / structured data of media files
 OpenNoneT109579 [Epic] Give more sister projects access to Wikidata
 OpenNoneT187900 There is no way to reference a specific quote on Wikiquote
 StalledNoneT71753 [Story] Wikibase / Wikidata support on Wikiquote
 OpenNoneT67626 [Epic] Support for queries on-wiki (automated list generation)
 ResolvedAddshoreT76019 [Story] Support new types of Entities in Wikibase Client
 ResolvedthiemowmdeT135650 [Task] Migrate PropertySuggester away from assuming all entities are numeric
 ResolvedAddshoreT75496 [Epic] Support new types of Entities in Wikibase Repository
 DeclinedNoneT58711 [Task] Update the wb_terms table so it does not have a numeric entity id
 OpenNoneT30599 Deadlock tracking bug (tracking)
 ResolvedhooT111535 Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
 ResolvedLydia_PintscherT146637 Wikidata 2016 Q4 goals
 ResolvedNoneT86530 Replace wb_terms table with more specialized mechanisms for terms (tracking)
 DeclinedNoneT51982 Add missing wb_entity_per_page entries on LinksUpdate
 InvalidLydia_PintscherT70176 EntityPerPageTable class should be usable from the client
 ResolvedLadsgroupT67333 Wikibase\EntityPerPageTable::getItemsWithoutSitelinks slow query with large LIMIT offset
 ResolvedAddshoreT114902 Remove numeric entity IDs from database schema
 ResolvedLadsgroupT95685 Drop wb_entity_per_page table
 DeclinedNoneT114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs
 ResolvedNoneT159851 Add a column for full entity ID to wb_terms table
 ResolvedWMDE-leszekT168708 Populating full entity column in wb_terms table when running MW update.php should be fast and run only once

Event Timeline

WMDE-leszek created this task.Mar 7 2017, 5:09 PM
WMDE-leszek added a parent task: T114903: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs.
gerritbot subscribed.Mar 8 2017, 6:48 PM

Change 341322 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: WMDE-leszek):
[mediawiki/extensions/Wikibase] [DNM] Add column for full entity ID to Terms DB table

https://gerrit.wikimedia.org/r/341322

gerritbot added a project: Patch-For-Review.Mar 8 2017, 6:48 PM
WMDE-leszek triaged this task as High priority.Mar 10 2017, 7:23 AM
WMDE-leszek removed a project: Patch-For-Review.Mar 13 2017, 7:03 AM
WMDE-leszek moved this task from Proposed to Backlog on the Wikidata-Former-Sprint-Board board.Mar 22 2017, 10:19 AM
Lydia_Pintscher moved this task from incoming to in progress on the Wikidata board.Mar 23 2017, 2:30 PM
daniel added a comment.EditedMar 29 2017, 2:15 PM

The schema update is not sufficient, we also need code to populate the new column. That code should be triggered by a schema update, but also needs a maintenance script. The script needs to operate in batches of configurable size, and should wait for slaves to catch up between batches.

Besides this, the code that updates the terms table needs to be able to write to the old and the new column at the same time. Similarly, the code that reads from the table needs to fall back to using the old columns in case the new column doesn't exist, or is empty. So we will need a feature switch with three states: numeric ids (legacy), full ids (new), and both (compat).

We should probably file separate tickets for these.

With respect to doing the schema update during the data center switchover: we can do that with only the schema change itself in place. It would however be nice to have the rest of the code in place before, so we can actually test the new schema before deploying it.

gerritbot added a comment.Apr 3 2017, 3:50 PM

Change 346169 had a related patch set uploaded (by Aleksey Bekh-Ivanov (WMDE)):
[mediawiki/extensions/Wikibase@master] Writing full IDs to wb_terms on insert

https://gerrit.wikimedia.org/r/346169

gerritbot added a project: Patch-For-Review.Apr 3 2017, 3:50 PM
gerritbot added a comment.Apr 7 2017, 3:54 PM

Change 341322 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add column for full entity ID to Terms DB table

https://gerrit.wikimedia.org/r/341322

aude subscribed.Apr 8 2017, 10:48 PM

see T162533 for creating script to populate the column

Ladsgroup created subtask T162539: Deploy schema change for adding term_full_entity_id column to wb_terms table.Apr 9 2017, 9:19 AM
Ladsgroup moved this task from Backlog to Done on the Wikidata-Former-Sprint-Board board.Apr 10 2017, 3:32 PM
WMDE-leszek created subtask T162673: Adjust code reading from wb_terms table to use full entity ID.Apr 11 2017, 11:06 AM
gerritbot added a comment.Apr 11 2017, 2:58 PM

Change 346169 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Writing full IDs to wb_terms on insert

https://gerrit.wikimedia.org/r/346169

Ladsgroup subscribed.Apr 17 2017, 7:21 AM

gerritbot doesn't add comment when I make a patch:

Ladsgroup moved this task from Done to Review on the Wikidata-Former-Sprint-Board board.Apr 17 2017, 7:21 AM
gerritbot added a comment.Apr 24 2017, 12:51 PM

Change 348413 merged by jenkins-bot:
[operations/mediawiki-config@master] Don't let Wikibase instances read/write terms_full_entity_id

https://gerrit.wikimedia.org/r/348413

WMDE-leszek moved this task from Review to Done on the Wikidata-Former-Sprint-Board board.Apr 25 2017, 3:11 PM
WMDE-leszek moved this task from Done to Doing on the Wikidata-Former-Sprint-Board board.
Reedy removed a project: Patch-For-Review.Apr 26 2017, 3:16 PM
• Marostegui closed subtask T162539: Deploy schema change for adding term_full_entity_id column to wb_terms table as Resolved.May 12 2017, 5:53 AM
Ladsgroup created subtask T165197: Change configuration of test Wikidata to write term_full_entity_id.May 12 2017, 7:54 PM
daniel mentioned this in T165328: Allow entity sub-structures to be treated like entities (so called sub-entities)..May 15 2017, 2:37 PM
WMDE-leszek closed subtask T162673: Adjust code reading from wb_terms table to use full entity ID as Resolved.May 31 2017, 8:39 AM
Ladsgroup created subtask T167114: Open view for term_full_entity_id in wb_terms table in labs.Jun 6 2017, 11:09 AM
Ladsgroup created subtask T167229: Change configuration of Wikidata in production to write term_full_entity_id.Jun 6 2017, 10:06 PM
• chasemp changed the status of subtask T167114: Open view for term_full_entity_id in wb_terms table in labs from Open to Stalled.Jun 8 2017, 2:24 PM
WMDE-leszek added a subtask: T168708: Populating full entity column in wb_terms table when running MW update.php should be fast and run only once.Jun 23 2017, 11:22 AM
daniel removed a subtask: T162539: Deploy schema change for adding term_full_entity_id column to wb_terms table.Jul 24 2017, 11:15 AM
daniel removed a subtask: T162673: Adjust code reading from wb_terms table to use full entity ID.
daniel removed a subtask: T167114: Open view for term_full_entity_id in wb_terms table in labs.
daniel removed a subtask: T167229: Change configuration of Wikidata in production to write term_full_entity_id.
daniel removed a subtask: T165197: Change configuration of test Wikidata to write term_full_entity_id.
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 25 2017, 8:14 AM
• Aleksey_WMDE closed subtask T168708: Populating full entity column in wb_terms table when running MW update.php should be fast and run only once as Resolved.Jul 25 2017, 2:51 PM
WMDE-leszek closed this task as Resolved.Oct 6 2017, 2:40 PM
WMDE-leszek claimed this task.

The column has been added and populated on wikidata production. Not open for view yet though, this is tracked in https://phabricator.wikimedia.org/T167114.

WMDE-leszek removed WMDE-leszek as the assignee of this task.Oct 6 2017, 2:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL