Page MenuHomePhabricator
Create Task
Maniphest T145426

Provide a way to add new unit normalizations to the query service without a full reload
Closed, ResolvedPublic
Actions

Description

We want to be able to add normalization for additional units without needing a full reload.

Note that we don't need to be able to change an existing conversion; that would be nice to have, bit it's not a requirement.

Possible implementation strategy, option I:

  • Find all values referencing the respective unit using a SPARQL query
  • Find the statements using that value, and the items using that statement

...option II:

  • Find all values referencing the respective unit using a SPARQL query
  • Find the statements using that value
  • Compute the normalized value using the new mapping, and add them to the triple store.

...option II|:

  • Find all statement values referencing the respective unit while scanning a JSON dump
  • Compute the normalized value using the new mapping, and output them (as n-triples or turtle).
  • Bulk-load the new triples into the query service

Details

SubjectRepoBranchLines +/-
Script to produce RDF mappings for new normalized unitsmediawiki/extensions/Wikibasewmf/1.28.0-wmf.23+724 -45
Script to produce RDF mappings for new normalized unitsmediawiki/extensions/Wikibasemaster+724 -45
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT88728 Improve Wikimedia dumping infrastructure
 OpenNoneT88991 improve Wikidata dumps [tracking]
 OpenNoneT46581 Partial dumps
 ResolvedSmalyshevT46580 Script for creating RDF dumps of all entities
 DuplicateNoneT211495 Dump(s) of Wikidata classes
 DuplicateNoneT211497 Dump(s) of Wikidata instances of Q5
 OpenNoneT162351 Create a "page prop" RDF dump for Wikidata entities ("pagePropertiesRdf")
 OpenNoneT98320 [Task] Create dump of entity redirects (JSON or n-triples)
 OpenNoneT285307 Create randomly split partial entity dumps
 OpenNoneT44063 [Epic] Provide a plain linked data interface for accessing entities
 ResolvedhooT101837 [Story] switch default rdf format to full (include statements)
 OpenNoneT50143 Implement complete RDF mapping for entities (tracking)
 OpenNoneT56318 Quantity datatype (tracking)
 ResolvedFeatureMichaelT68580 Better support for exact values in Quantity DataType
 ResolveddanielT140997 When guessing quantity precision from digits, use the +/-0.5 rule rather than the +/-1 rule
 OpenNoneT133042 Quantity datatype precision (tracking)
 ResolvedLydia_PintscherT115269 [Story] When a Quantity is entered with no uncertainty/bounds given, do not guess uncertainty/bounds until needed.
 ResolvedthiemowmdeT115270 Allow QuantityValue objects to have empty/no bounds set
 ResolveddanielT105623 [Task] Investigate quantification of quantity precision (+/- 1 or +/- 0.5)
 OpenNoneT77977 [Epic] Unit support
StalledNoneT111770 [Story] Decide how to represent quantities with units in the "truthy" RDF mapping
 StalledNoneT115934 Represent simple values with units in RDF
 ResolvedSmalyshevT117031 Represent normalized unit values in full values RDF
 OpenNoneT145424 Represent normalized unit values in simple values RDF
 OpenNoneT77978 [Story] Support unit conversion
 ResolvedSmalyshevT117032 Create configuration for specifying units conversions
 ResolvedSmalyshevT145426 Provide a way to add new unit normalizations to the query service without a full reload
 ResolvedSmalyshevT144768 Decide how to enable/disable unit conversions in flavors
 InvalidNoneT150656 Light-year and gigaparsec do not show up in the query for the longest things on Wikidata.
 ResolveddanielT150877 Unit conversion does not create psn:P2043 for Q531
 ResolvedSmalyshevT150881 add support for additional dimensions for unit conversion
 OpenNoneT234809 Support normalized quantities in Lua
 ResolvedLydia_PintscherT142086 [Task] announce quantity changes
 ResolvedLadsgroupT142087 [Task] prepare bot for quantity change fixes
 OpenNoneT112247 [RFC] Create a "number" datatype for exact values
 ResolvedLadsgroupT154168 Quantity changes broke ORES

Event Timeline

daniel created this task.Sep 12 2016, 5:43 PM
daniel mentioned this in T117032: Create configuration for specifying units conversions.Sep 12 2016, 5:50 PM
daniel updated the task description. (Show Details)Sep 12 2016, 6:16 PM
Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Sep 12 2016, 7:52 PM
Smalyshev moved this task from Current work to SDAW on the Wikidata-Query-Service board.Sep 12 2016, 10:36 PM
Smalyshev claimed this task.Sep 24 2016, 12:02 AM
Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.
Smalyshev added a comment.Sep 24 2016, 12:24 AM

Right now I think this should be the plan:

The tool gets two .json config files - new config and old config. Old config can be optional. Then:

  1. Diff the configs and produce list of new units
  2. For each new primary unit:
    1. Run SPARQL query to find all values using it and generate self-referencing normalized statements with wikibase:quantityNormalized
    2. Run SPARQL query to find all statements using those values (need to see if we have too many we may have to split it in batches) and generate parallel normalized statements for those, with the same value.
  3. For each new non-primary unit:
    1. Run SPARQL query to find all values using it and generate new converted value for each one. Generate SPARQL for those new values and also wikibase:quantityNormalized statements on the old values.
    2. Run SPARQL query to find all statements using those values and generate parallel normalized statements for those, with the new converted value.

The output of the tool will be RDF/TTL that can be bulk-loaded into the instance.

We need to see if we will be able to hold all the values described in memory. So far the most popular unit - square kilometre - has 13398 usages, it should not be a problem to hold all of them in memory I think.

Smalyshev moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.Sep 24 2016, 12:25 AM
gerritbot added a comment.Sep 24 2016, 4:55 AM

Change 312627 had a related patch set uploaded (by Smalyshev):
Script to produce RDF mappings for new normalized units

https://gerrit.wikimedia.org/r/312627

gerritbot added a project: Patch-For-Review.Sep 24 2016, 4:55 AM
Smalyshev moved this task from In progress to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.Oct 2 2016, 9:15 PM
gerritbot added a comment.Nov 2 2016, 8:01 PM

Change 312627 merged by jenkins-bot:
Script to produce RDF mappings for new normalized units

https://gerrit.wikimedia.org/r/312627

gerritbot added a comment.Nov 2 2016, 8:28 PM

Change 319402 had a related patch set uploaded (by Smalyshev):
Script to produce RDF mappings for new normalized units

https://gerrit.wikimedia.org/r/319402

Smalyshev closed this task as Resolved.Nov 2 2016, 11:39 PM
gerritbot added a comment.Nov 3 2016, 4:58 PM

Change 319402 abandoned by Smalyshev:
Script to produce RDF mappings for new normalized units

Reason:
we can do it without backporting

https://gerrit.wikimedia.org/r/319402

Smalyshev removed a project: Discovery-Wikidata-Query-Service-Sprint.Jul 14 2017, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL