Maniphest T208330

MediaInfo extension should not use the wb_terms table
Closed, ResolvedPublic
Actions

Description

After creating some captions on commons on beta I went to check the wb_terms table and it is indeed being used.

MariaDB [commonswiki]> select * from wb_terms;
+-------------+----------------+---------------------+------------------+---------------+-----------+--------------------------------------+-----------------+-------------+
| term_row_id | term_entity_id | term_full_entity_id | term_entity_type | term_language | term_type | term_text                            | term_search_key | term_weight |
+-------------+----------------+---------------------+------------------+---------------+-----------+--------------------------------------+-----------------+-------------+
|           1 |              0 | M59928              | mediainfo        | en            | label     | Make a caption                       |                 |           0 |
|           2 |              0 | M59928              | mediainfo        | fr            | label     | Seulement une légende                |                 |           0 |
|           3 |              0 | M58796              | mediainfo        | en            | label     | Procedurally generated crystal in 3D |                 |           0 |
|           4 |              0 | M59928              | mediainfo        | es            | label     | another caption                      |                 |           0 |
+-------------+----------------+---------------------+------------------+---------------+-----------+--------------------------------------+-----------------+-------------+
4 rows in set (0.00 sec)

Lexeme has been implemented without using this table.

Related is T198866 which lists the usecases within wikibase itself that we want to migrate away from.

Quote from IRC:

<duesen> Daniel Kinzler addshore: MediaInfo has no uniqueness constraints
5:10 PM we will want "label lookups" at some point

So it sounds like this might accidently be getting used right now while it is not needed?

Details

Subject	Repo	Branch	Lines +/-
EntityHandler, don't write to TermIndex for all entity types	mediawiki/extensions/Wikibase	wmf/1.33.0-wmf.12	+20 -10
EntityHandler, don't write to TermIndex for all entity types	mediawiki/extensions/Wikibase	wmf/1.33.0-wmf.9	+20 -10
EntityHandler, don't write to TermIndex for all entity types	mediawiki/extensions/Wikibase	master	+20 -10

Customize query in gerrit

Related Objects

Mentioned In: T205732: Get WikibaseMediaInfo to the state that we're happy to proceed with initial production deployment (captions)
Mentioned Here: T208425: [EPIC] Kill the wb_terms table
T198866: Uses of wb_terms SQL table to be migrated away from

Event Timeline

Addshore created this task.Oct 30 2018, 4:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2018, 4:48 PM

Jdforrester-WMF added a project: Structured Data Engineering.Oct 30 2018, 4:51 PM

Jdforrester-WMF added a parent task: T205732: Get WikibaseMediaInfo to the state that we're happy to proceed with initial production deployment (captions).

Erm ... why not? "Terms" are labels, description and aliases, afaik.

In T208330#4706901, @Cparle wrote:

Erm ... why not? "Terms" are labels, description and aliases, afaik.

Indeed, but the table is badly designed and doesn't scale.

Hmmm ok. Its use of that table was in place when I got here, I think, might take a little while to figure out how to extract it (edit: and since Matthias's baby arrived I am now the only dev working fulltime on this). Is this necessary for going to production?

In T208330#4706909, @Cparle wrote:

Hmmm ok. Its use of that table was in place when I got here, I think, might take a little while to figure out how to extract it (edit: and since Matthias's baby arrived I am now the only dev working fulltime on this). Is this necessary for going to production?

I guess you could go to production with this, but you would have to start thinking about migrating away from it as a priority.
It would be worth poking the DBA s, if you decide to move to production with this.

The table should see a similarly linear growth as that of the wb_terms table on wikidatawiki, although i guess there are some different characteristics:

captions will generally be longer than labels?
The rate of increase in row count could be greater than that on wikidatawiki, as youll be creating a bunch of new entities quickly during the migration.

Addshore moved this task from incoming to monitoring on the Wikidata board.Oct 31 2018, 10:13 AM

Addshore added a project: DBA.Nov 1 2018, 2:29 PM

Addshore moved this task from Triage to Blocked external/Not db team on the DBA board.

I would prefer if we don't enable more stuff in production that uses wb_term table, that table is already in a _very_ bad state and we should stop using it as soon as we can.
So getting more stuff to use this table is probably not what we want to do, as it will be another road in the way of getting rid of this huge table.

In T208330#4738627, @Marostegui wrote:

I would prefer if we don't enable more stuff in production that uses wb_term table, that table is already in a _very_ bad state and we should stop using it as soon as we can.
So getting more stuff to use this table is probably not what we want to do, as it will be another road in the way of getting rid of this huge table.

One thing to note is this will be commonswiki.wb_terms rather than wikdatawiki.wb_terms, so the existing table won't get bigger with MediaInfo, however we will have a new table for commonswiki that will be growing.

In T208330#4742192, @Addshore wrote:

In T208330#4738627, @Marostegui wrote:

I would prefer if we don't enable more stuff in production that uses wb_term table, that table is already in a _very_ bad state and we should stop using it as soon as we can.
So getting more stuff to use this table is probably not what we want to do, as it will be another road in the way of getting rid of this huge table.

One thing to note is this will be commonswiki.wb_terms rather than wikdatawiki.wb_terms, so the existing table won't get bigger with MediaInfo, however we will have a new table for commonswiki that will be growing.

Good point - however, it is still the same thing. Something that will needed to be migrated away.
So I would prefer if things are done rightly from the start so we can "forget" about it, rather than having another thing to keep in mind whilst getting rid of wb_terms

Jdforrester-WMF mentioned this in T205732: Get WikibaseMediaInfo to the state that we're happy to proceed with initial production deployment (captions).Nov 16 2018, 7:44 PM

• Ramsey-WMF subscribed.Nov 19 2018, 5:21 PM

I've briefly looked into our code around this, and I'm afraid I don't see it as remotely plausible for us to move WBMI away from the table in the next few months' work; we're blocked on Wikibase moving. WBMI doesn't have any code that decides where our data goes (it's all done by Wikibase itself, as they're "just" labels). Consequently we'll have to postpone this work. :-(

So, it doesn't look like WBMI actually uses the data in the table.
If that is the case then we can likely put something in place to prevent the table being written to.

WMDE-leszek subscribed.Jan 10 2019, 9:41 AM

Change 483376 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483376

gerritbot added a project: Patch-For-Review.Jan 10 2019, 9:42 AM

Change 483376 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483376

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.13; 2019-01-15).Jan 10 2019, 12:00 PM

Change 483388 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.9] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483388

Change 483389 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.12] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483389

These will be backported in a slot before the deployment of media info to real commons today

Change 483388 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.9] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483388

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.9; 2018-12-18); removed MW-1.33-notes (1.33.0-wmf.13; 2019-01-15).Jan 10 2019, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-10T15:12:42Z] <addshore@deploy1001> Synchronized php-1.33.0-wmf.9/extensions/Wikibase/repo/includes/Content: [[gerrit:483388|T208330 dont write to wb_terms for mediainfo]] (duration: 00m 55s)

Change 483389 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.12] EntityHandler, don't write to TermIndex for all entity types

https://gerrit.wikimedia.org/r/483389

Mentioned in SAL (#wikimedia-operations) [2019-01-10T15:20:55Z] <addshore@deploy1001> Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/includes/Content: [[gerrit:483388|T208330 dont write to wb_terms for mediainfo]] (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-01-10T15:24:45Z] <addshore> T208330, MariaDB [testcommonswiki]> TRUNCATE TABLE wb_terms; # Was https://phabricator.wikimedia.org/P7973

The extension no longer writes into the table

Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 10 2019, 3:25 PM

Thanks so much!

This is only going to be Resolved for a couple of weeks until we enable Properties; fixing that is tracked in T208425.

Jdforrester-WMF moved this task from To Do to Monitoring on the Structured Data Engineering board.Jan 10 2019, 3:36 PM

Jdforrester-WMF removed a project: Patch-For-Review.

In T208330#4869968, @Jdforrester-WMF wrote:

This is only going to be Resolved for a couple of weeks until we enable Properties; fixing that is tracked in T208425.

The good thing about properties is, well, there won't be many rows in the table :)

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.12; 2019-01-08); removed MW-1.33-notes (1.33.0-wmf.9; 2018-12-18).Jan 10 2019, 4:00 PM

greg added a project: Multimedia.Mar 7 2019, 10:59 PM

MediaInfo extension should not use the wb_terms tableClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

MediaInfo extension should not use the wb_terms table
Closed, ResolvedPublic
Actions