Page MenuHomePhabricator
Create Task
Maniphest T160926

Make noop script be able replace whole fields with nested subfields
Closed, ResolvedPublic
Actions

Description

Currently, noop script requires incoming data to have null/empty value for a field that is being deleted. However, it is a problem for field with massive amount of subfields, such as labels/description fields in Wikidata (see T125500).

We need to be able to tell noop script to replace content of the nested field wholesale, e.g. having data:

labels: { x: hello, y: world }

and incoming data:

labels: { x: bonjour }

it would replace the whole labels field and not just labels.x. Ideally this would be a parameter on labels mapping.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Allow setting super_noop_script handlersDCaussemediawiki/extensions/CirrusSearchmaster+119 -1
Clarify equals handler and make it nullsafe by defaultDCaussesearch/extramaster+208 -28
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• Wikidata-bugsT77898 query taking 10s: TermSqlIndex::getMatchingIDs
 OpenNoneT46529 Wikidata search problems (tracking)
 ResolvedaudeT110648 [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata
 ResolvedSmalyshevT119067 Adjust rescoring config for Wikidata to consider sitelink count
 ResolvedaudeT119066 Add sitelink count to search index for Wikidata
 ResolvedNoneT120217 [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins
 InvalidNoneT161334 Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata
 ResolvedSmalyshevT78157 [Story] Use ElasticSearch for entity search on wikidata.org
 DuplicateSmalyshevT117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
 DeclinedNoneT170392 Create gadget that enables the use of the elastic search backend for the entity selector
 InvalidNoneT170547 Metrics to evaluate new search for item suggestor
 DeclinedNoneT170549 Provide A/B test for item suggestor
ResolvedSmalyshevT162292 Reindex wikidata to pick up labels/descriptions mappings
 DeclinedNoneT141813 Add full-text search support to Query Service
 ResolvedSmalyshevT125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
 ResolveddcausseT160926 Make noop script be able replace whole fields with nested subfields
 ResolveddcausseT166589 Update wikidata code to take advantage of nested fields noop script
 ResolvedSmalyshevT178851 Use label & description index for fulltext search
 ResolvedSmalyshevT176903 Index wikidata descriptions
 ResolvedSmalyshevT180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
 ResolvedSmalyshevT182271 Wikidata fulltext search should handle search syntax properly
 ResolveddcausseT182293 Tune wikidata fulltext search similarity parameters

Event Timeline

Smalyshev created this task.Mar 20 2017, 5:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2017, 5:22 PM
Smalyshev added a parent task: T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.Mar 20 2017, 5:22 PM
Smalyshev edited projects, added: Discovery-Search (Current work); removed: Discovery-Search.Mar 21 2017, 5:12 PM
• debt triaged this task as Medium priority.Mar 21 2017, 5:13 PM
dcausse claimed this task.Mar 30 2017, 3:28 PM
dcausse moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.
gerritbot subscribed.Apr 3 2017, 4:16 PM

Change 346172 had a related patch set uploaded (by DCausse):
[search/extra@master] Add pruneMap handler to super_noop_script

https://gerrit.wikimedia.org/r/346172

gerritbot added a project: Patch-For-Review.Apr 3 2017, 4:16 PM
Smalyshev added a parent task: T162292: Reindex wikidata to pick up labels/descriptions mappings.Apr 5 2017, 5:56 PM
gerritbot added a comment.Apr 10 2017, 1:37 PM

Change 347380 had a related patch set uploaded (by DCausse):
[mediawiki/extensions/CirrusSearch@master] Allow setting super_noop_script handlers

https://gerrit.wikimedia.org/r/347380

dcausse added a comment.Apr 10 2017, 2:00 PM

There's no need to add new handlers unfortunately the equals handler needs to be fixed to be nullsafe.
Using it right now without deploying a new version of the plugin will cause NPE during updates.

dcausse moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Apr 11 2017, 12:13 PM
gerritbot added a comment.Apr 11 2017, 6:02 PM

Change 346172 merged by jenkins-bot:
[search/extra@master] Clarify equals handler and make it nullsafe by default

https://gerrit.wikimedia.org/r/346172

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Apr 11 2017, 6:18 PM
gerritbot added a comment.May 1 2017, 9:07 PM

Change 347380 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Allow setting super_noop_script handlers

https://gerrit.wikimedia.org/r/347380

ReleaseTaggerBot added a project: MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)).May 1 2017, 10:00 PM
Smalyshev added a comment.May 2 2017, 1:17 AM

@dcausse, after 347380 is merged, do we need to do some changes in the Wikidata code to actually use proper noop operations?

• debt closed this task as Resolved.May 30 2017, 5:27 PM
• debt subscribed.

Plugin is ready for deployment, this ticket is done.

Smalyshev mentioned this in T166589: Update wikidata code to take advantage of nested fields noop script.May 30 2017, 5:28 PM
Smalyshev created subtask T166589: Update wikidata code to take advantage of nested fields noop script.
Smalyshev closed subtask T166589: Update wikidata code to take advantage of nested fields noop script as Resolved.Jul 21 2017, 4:55 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 21 2017, 4:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits