Make noop script be able replace whole fields with nested subfields
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Smalyshev
	Mar 20 2017, 5:22 PM

Description

Currently, noop script requires incoming data to have null/empty value for a field that is being deleted. However, it is a problem for field with massive amount of subfields, such as labels/description fields in Wikidata (see T125500).

We need to be able to tell noop script to replace content of the nested field wholesale, e.g. having data:

labels: { x: hello, y: world }

and incoming data:

labels: { x: bonjour }

it would replace the whole labels field and not just labels.x. Ideally this would be a parameter on labels mapping.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Allow setting super_noop_script handlers	mediawiki/extensions/CirrusSearch	master	+119 -1
	Clarify equals handler and make it nullsafe by default	search/extra	master	+208 -28

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Wikidata-bugs	T77898 query taking 10s: TermSqlIndex::getMatchingIDs
Open	None	T46529 Wikidata search problems (tracking)
Resolved	aude	T110648 [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata
Resolved	Smalyshev	T119067 Adjust rescoring config for Wikidata to consider sitelink count
Resolved	aude	T119066 Add sitelink count to search index for Wikidata
Resolved	None	T120217 [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins
Invalid	None	T161334 Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata
Resolved	Smalyshev	T78157 [Story] Use ElasticSearch for entity search on wikidata.org
Duplicate	Smalyshev	T117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
Declined	None	T170392 Create gadget that enables the use of the elastic search backend for the entity selector
Invalid	None	T170547 Metrics to evaluate new search for item suggestor
Declined	None	T170549 Provide A/B test for item suggestor
Resolved	Smalyshev	T162292 Reindex wikidata to pick up labels/descriptions mappings
Declined	None	T141813 Add full-text search support to Query Service
Resolved	Smalyshev	T125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Resolved	dcausse	T160926 Make noop script be able replace whole fields with nested subfields
Resolved	dcausse	T166589 Update wikidata code to take advantage of nested fields noop script
Resolved	Smalyshev	T178851 Use label & description index for fulltext search
Resolved	Smalyshev	T176903 Index wikidata descriptions
Resolved	Smalyshev	T180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
Resolved	Smalyshev	T182271 Wikidata fulltext search should handle search syntax properly
Resolved	dcausse	T182293 Tune wikidata fulltext search similarity parameters

Event Timeline

Smalyshev created this task.Mar 20 2017, 5:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2017, 5:22 PM

Smalyshev added a parent task: T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.Mar 20 2017, 5:22 PM

Smalyshev edited projects, added Discovery-Search (Current work); removed Discovery-Search.Mar 21 2017, 5:12 PM

• debt triaged this task as Medium priority.Mar 21 2017, 5:13 PM

dcausse claimed this task.Mar 30 2017, 3:28 PM

dcausse moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

Change 346172 had a related patch set uploaded (by DCausse):
[search/extra@master] Add pruneMap handler to super_noop_script

https://gerrit.wikimedia.org/r/346172

gerritbot added a project: Patch-For-Review.Apr 3 2017, 4:16 PM

Smalyshev added a parent task: T162292: Reindex wikidata to pick up labels/descriptions mappings.Apr 5 2017, 5:56 PM

Change 347380 had a related patch set uploaded (by DCausse):
[mediawiki/extensions/CirrusSearch@master] Allow setting super_noop_script handlers

https://gerrit.wikimedia.org/r/347380

There's no need to add new handlers unfortunately the equals handler needs to be fixed to be nullsafe.
Using it right now without deploying a new version of the plugin will cause NPE during updates.

dcausse moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Apr 11 2017, 12:13 PM

Change 346172 merged by jenkins-bot:
[search/extra@master] Clarify equals handler and make it nullsafe by default

https://gerrit.wikimedia.org/r/346172

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Apr 11 2017, 6:18 PM

Change 347380 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Allow setting super_noop_script handlers

https://gerrit.wikimedia.org/r/347380

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)).May 1 2017, 10:00 PM

@dcausse, after 347380 is merged, do we need to do some changes in the Wikidata code to actually use proper noop operations?

Plugin is ready for deployment, this ticket is done.

Smalyshev mentioned this in T166589: Update wikidata code to take advantage of nested fields noop script.May 30 2017, 5:28 PM

Smalyshev created subtask T166589: Update wikidata code to take advantage of nested fields noop script.

Smalyshev closed subtask T166589: Update wikidata code to take advantage of nested fields noop script as Resolved.Jul 21 2017, 4:55 PM

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 21 2017, 4:55 PM

Make noop script be able replace whole fields with nested subfieldsClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Make noop script be able replace whole fields with nested subfields
Closed, ResolvedPublic
Actions

Related Objects
Search...