Page MenuHomePhabricator
Create Task
Maniphest T146356

[Bug] RDF export misses extreme values with day precision
Closed, ResolvedPublic
Actions

Description

Because the Julian to Gregorian conversion we do is currently based on PHP's limited cal_to_jd and jdtogregorian functions, many dates the Wikibase data model supports are currently not accessible from the RDF export and the QueryService.

Trivial solution: Based on the assumption that a date value at, let's say, 10.000 years BCE with day precision is pointless anyway, we can just export the year. This is already much better than no export.

Reported here: https://www.mediawiki.org/wiki/User_talk:Thiemo_Mättig_(WMDE)#RDF_Julian_Gregorian_conversion

Details

SubjectRepoBranchLines +/-
Skip Julian to Gregorian date conversion for extreme, unsupported datesmediawiki/extensions/Wikibasemaster+84 -35
Cleaner JulianDateTimeValueCleaner implementationmediawiki/extensions/Wikibasemaster+69 -34
Fix (Julian)DateTimeValueCleaner for extreme yearsmediawiki/extensions/Wikibasemaster+40 -14
Cleanup DateTimeValueCleanerTestmediawiki/extensions/Wikibasemaster+66 -51
Customize query in gerrit

Related Objects
Search...

Event Timeline

thiemowmde created this task.Sep 22 2016, 9:39 AM
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptSep 22 2016, 9:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
thiemowmde added a parent task: T87764: Bugs related to time datatype (tracking).Sep 22 2016, 9:40 AM
gerritbot subscribed.Sep 22 2016, 9:44 AM

Change 312214 had a related patch set uploaded (by Thiemo Mättig (WMDE)):
Cleanup DateTimeValueCleanerTest

https://gerrit.wikimedia.org/r/312214

gerritbot added a project: Patch-For-Review.Sep 22 2016, 9:44 AM

Change 312215 had a related patch set uploaded (by Thiemo Mättig (WMDE)):
Fix (Julian)DateTimeValueCleaner for extreme years

https://gerrit.wikimedia.org/r/312215

thiemowmde triaged this task as Low priority.Sep 22 2016, 9:45 AM
thiemowmde moved this task from Proposed to Review on the Wikidata-Sprint-2016-09-21 board.
thiemowmde moved this task from Needs triage to WDQS on the Discovery-ARCHIVED board.
thiemowmde moved this task from incoming to in progress on the Wikidata board.
thiemowmde moved this task from Incoming to SDAW on the Wikidata-Query-Service board.
Smalyshev added a comment.Sep 23 2016, 11:40 PM

I think this idea makes a lot of sense. While I think all far-ago dates that have day precision are most probably data errors (you can't have May 4th in 200000BC, really, at least not using Earthly calendars :) handling these errors in more sane manner is good. However, I think we need then to change precision, if we operate under year precision, we can't just take year and then claim day precision. It'd be claiming false data.

Jc3s5h added a comment.Sep 24 2016, 12:59 AM
In T146356#2663757, @Smalyshev wrote:

I think this idea makes a lot of sense. While I think all far-ago dates that have day precision are most probably data errors (you can't have May 4th in 200000BC, really, at least not using Earthly calendars :) handling these errors in more sane manner is good. However, I think we need then to change precision, if we operate under year precision, we can't just take year and then claim day precision. It'd be claiming false data.

A person using the RDF may just want the best idea of the date that the RDF processing can give. But the person might also be performing quality control on the underlying data that flowed into the internal representation to RDF conversion process. It would be nice to somehow indicate that the date had to be approximated due to limitations of the RDF conversion process. (Perhaps leave it for the data consumer to figure out for him/her/itself that May 4th in 200000BC is silly?)

Smalyshev added a comment.Sep 24 2016, 1:03 AM

Perhaps leave it for the data consumer to figure out for him/her/itself that May 4th in 200000BC is silly

It is an option, and here we have a tradeoff, of either:

A. Try to represent the wrong data in a way that they would be useful, preserving the useful part and dropping the bad part, or
B. Don't try to fix the data and present it as is, broken if it so happens

Depending on your goals, you may want either of them, or maybe both - i.e. have both original and normalized values. We may try doing that too.

gerritbot added a comment.Sep 27 2016, 6:08 PM

Change 312214 merged by jenkins-bot:
Cleanup DateTimeValueCleanerTest

https://gerrit.wikimedia.org/r/312214

gerritbot added a comment.Oct 4 2016, 10:17 AM

Change 313984 had a related patch set uploaded (by Thiemo Mättig (WMDE)):
Add extreme Julian date test to TimeRdfBuilderTest

https://gerrit.wikimedia.org/r/313984

thiemowmde closed this task as Resolved.Oct 4 2016, 10:19 AM
thiemowmde moved this task from Review to Done on the Wikidata-Sprint-2016-09-21 board.
gerritbot added a comment.Oct 4 2016, 10:28 AM

Change 312215 merged by jenkins-bot:
Fix (Julian)DateTimeValueCleaner for extreme years

https://gerrit.wikimedia.org/r/312215

gerritbot added a comment.Oct 5 2016, 9:31 AM

Change 314240 had a related patch set uploaded (by Thiemo Mättig (WMDE)):
Cleaner JulianDateTimeValueCleaner implementation

https://gerrit.wikimedia.org/r/314240

gerritbot added a comment.Oct 5 2016, 7:07 PM

Change 313984 merged by jenkins-bot:
Skip Julian to Gregorian date conversion for extreme, unsupported dates

https://gerrit.wikimedia.org/r/313984

gerritbot added a comment.Oct 6 2016, 2:49 PM

Change 314240 merged by jenkins-bot:
Cleaner JulianDateTimeValueCleaner implementation

https://gerrit.wikimedia.org/r/314240

Jc3s5h reopened this task as Open.Oct 7 2016, 4:15 PM

I have been experimenting with various extreme values of the Julian year, as great as positive 10 billion, and am finding erratic results for the value of the julian date ($jd) and the Gregorian date ($gregorian). I found that for values that are too large, $jd could be 0, negative, or of the same order of magnitude as the year (when it should be about 365 times the year). Also, for values that are too large, $gregorian might or might not be "0/0/0".

I should add that I don't have another tool at my disposal to independently convert Julian to Gregorian for enormous dates, so I can't say for sure if the correction in PHP is correct between the years 101,994 and 1,465,071, only that there are no obvious problems.

I found the largest Julian calendar date that does not produce any obvious problem is y = rMW1465072dbe23, m = 9, d = 17. This results in $jd = 536,838,867. The value of $gregorian is "10/17/1,465,102" (thousand separators added).

A simple solution, if you don't mind not converting less than a year's worth of dates in the distant future, is to add a test that the Julian year must be < rMW1465072dbe23.

hoo closed this task as Resolved.Oct 7 2016, 4:19 PM
In T146356#2699680, @Jc3s5h wrote:

A simple solution, if you don't mind not converting less than a year's worth of dates in the distant future, is to add a test that the Julian year must be < rMW1465072dbe23.

We already do this, see https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/35e6a6bbcd1c94565443404a04d60ad355eb9704/repo/includes/Rdf/JulianDateTimeValueCleaner.php#L62 (since d6f5080ca43a4a7064c7a7e7eb558141a73bee54).

Jc3s5h added a comment.Oct 7 2016, 4:26 PM
In T146356#2699683, @hoo wrote:

We already do this, see https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/35e6a6bbcd1c94565443404a04d60ad355eb9704/repo/includes/Rdf/JulianDateTimeValueCleaner.php#L62 (since d6f5080ca43a4a7064c7a7e7eb558141a73bee54).

That's great. I must have been looking at an earlier version.

thiemowmde added a comment.Oct 18 2016, 12:59 PM

Here is a query to try the effect of this change: https://query.wikidata.org/#SELECT%20%3Fitem%20%3Ftime%7B%0A%3Fitem%20wdt%3AP31%20wd%3AQ36507.%0A%3Fitem%20wdt%3AP585%20%3Ftime%0A%7D%0AORDER%20BY%20%3Ftime%0ALIMIT%2020 It should now find Items with both Julian and Gregorian dates. Before dates at (roughly) 5000 BC and before could not be found and not ordered by date.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL