Description

Write statements with their qualifiers into the cirrus search index

Each statement should be stored without qualifiers, and then separately with each qualifier

For example, on wikidata the Mona Lisa has the statement 'depicts woman' P180=Q467 and the statement 'depicts sky' P180=Q527 with the qualifiers 'applies to part background' P518=Q13217555 and 'color green' P462=Q3133.

We propose to store these in statement_keywords as follows:

'statement_keywords' : [
    'P180=Q467',
    'P180=Q527',
    'P180=Q527[P518=Q13217555]',
    'P180=Q527[P462=Q3133]',
]

In this way the user will be able to search using the haswbstatement keyword. If they want to find items with 'depicts sky' then can search for haswbstatement:P180=Q527, where if they want to find items that depict a green sky they can search for haswbstatement:P180=Q527[P462=Q3133]

Details

	Subject	Repo	Branch	Lines +/-
	Write wikibase statement qualifiers to the search index	mediawiki/extensions/Wikibase	master	+103 -35

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
Open		None	T115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
Duplicate		None	T115864 Figure out if the description of the image can be used as the caption on-wiki
Open	Feature	None	T53032 When inserting an image, set its caption by default to be the Commons image description
Open	Feature	None	T39534 Wikimedia Commons should support searching by color
Duplicate		None	T39535 Wikimedia Commons should support filtering by color
Resolved		None	T19503 Provide metadata support on Wikimedia Commons
Resolved		None	T51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
Resolved		None	T68108 [Epic] Store media information for files on Wikimedia Commons as structured data
Resolved		• Ramsey-WMF	T199352 Deploy Structured Data on Commons with arbitrary Statements
Resolved		None	T215305 "Depicts and other statements on a bicycle": Qualifiers, and search by depicts statements, and other statements
Resolved		Cparle	T191633 Implement searching of 'depicts' on commons
Resolved		Cparle	T193012 Think about how to index and search qualifiers for 'depicts' statements
Resolved		Cparle	T193407 Store wikibase statement qualifiers in cirrus search index

Event Timeline

Cparle triaged this task as Medium priority.Apr 30 2018, 3:45 PM

Cparle created this task.

Cparle updated the task description. (Show Details)

Cparle moved this task from To Do to Doing on the Multimedia-Team-Working-Board board.Apr 30 2018, 3:48 PM

Change 430078 had a related patch set uploaded (by Cparle; owner: Cparle):
[mediawiki/extensions/Wikibase@master] Write wikibase statement qualifiers to the search index

https://gerrit.wikimedia.org/r/430078

gerritbot added a project: Patch-For-Review.May 1 2018, 3:55 PM

Cparle moved this task from Doing to Code Review on the Multimedia-Team-Working-Board board.May 2 2018, 11:20 AM

The scheme looks fine, but I have a question about which qualifiers we index. There are options:

Use the same config as statement option does. The downside of this is that there may be qualifiers that we want to see that make no sense to index on statements, and vice versa.
Index all qualifiers - after all, most statements only have one or two, so indexing all of them probably won't hurt. Of course, that is conditioned on the value type actually having indexing callback.
Have separate config for qualifiers.

I am slightly preferring option 2, but I am not set on this.

Option 2 sounds reasonable, uploaded a new version of the patch

debt moved this task from needs triage to Current work on the Discovery-Search board.May 3 2018, 5:28 PM

debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.

debt moved this task from Current work to watching / waiting on the Discovery-Search board.

debt edited projects, added Discovery-Search; removed Discovery-Search (Current work).

• Ramsey-WMF moved this task from Untriaged to Next up on the Multimedia board.May 7 2018, 5:00 PM

Cparle mentioned this in T194185: Implement searching of 'depicts' on commons with the 'inscription' qualifier.May 8 2018, 5:05 PM

Cparle mentioned this in T194245: Implement searching of 'depicts' on commons with the 'quantity' qualifier.May 9 2018, 9:52 AM

Cparle moved this task from Backlog to CURRENT - Work in progress on the SDC General board.May 31 2018, 4:59 PM

Change 430078 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Write wikibase statement qualifiers to the search index

https://gerrit.wikimedia.org/r/430078

Ladsgroup moved this task from incoming to monitoring on the Wikidata board.Jun 28 2018, 3:47 PM

Smalyshev mentioned this in T198091: PropertyDataTypeLookupException from line 50 of vendor/wikibase/data-model-services/src/Lookup/EntityRetrievingDataTypeLookup.php: Property data type lookup failed for: P400.Jun 28 2018, 5:46 PM

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)).Jun 30 2018, 5:02 PM

• Vvjjkkii renamed this task from Store wikibase statement qualifiers in cirrus search index to wydaaaaaaa.Jul 1 2018, 1:13 AM

• Vvjjkkii removed Cparle as the assignee of this task.

• Vvjjkkii raised the priority of this task from Medium to High.

• Vvjjkkii edited projects, added CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish); removed MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed subscribers: gerritbot, Aklapper.

Yann renamed this task from wydaaaaaaa to Store wikibase statement qualifiers in cirrus search index.Jul 1 2018, 12:46 PM

Yann assigned this task to Cparle.

Yann lowered the priority of this task from High to Medium.

Yann removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

Yann updated the task description. (Show Details)

Yann added subscribers: gerritbot, Aklapper.

AntiCompositeNumber added a project: MW-1.32-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)).Jul 1 2018, 1:11 PM

@EBernhardson I see you merged the patch here, but this task is still open since late June - is there further work to do? This is still in the code review column on our working board, so either we should close this task or move it into "next up" or "doing" as appropriate..

I think the next step now that it's deployed is to reindex.

OK, is the search team going to do that or should the Multimedia team keep this around as a to-do item?

Yes, Search Platform team will be responsible for the reindex. Note that it should already be available on recently edited items, but not on items that hasn't been edited since the patch was merged.

Smalyshev mentioned this in T147505: [tracking] CirrusSearch: what is updated during re-indexing.Aug 3 2018, 6:31 PM

MarkTraceur moved this task from Code Review to Ready for Production Testing on the Multimedia-Team-Working-Board board.Aug 3 2018, 7:37 PM

MarkTraceur moved this task from Ready for Production Testing to Blocked on the Multimedia-Team-Working-Board board.

• Ramsey-WMF moved this task from Blocked to Ready for Production Testing on the Multimedia-Team-Working-Board board.Aug 13 2018, 4:50 PM

Reindex is done.

Fuzheado subscribed.Jan 22 2019, 6:58 PM