I won't take issue issue with Delane13 changing the priority. I do think the general case for visualizing data in time is that the data about the events was originally recorded in a mixture of calendars, and usually there will be a desire to retrieve the calendar information so all the results of a query are in the same calendar so that they can be compared to each other. I think it is commonplace among researchers to have to convert dates themselves into their desired calendar and format. When large amounts of data are being handled, I think researchers will need automated tools for calendar conversions.

I see no reason to keep it open.

In T207705#7704353, @Pigsonthewing wrote:

Your proposed alternative is..?

In T207705#7703514, @Epidosis wrote:

The task above could be updated: ISO 8601-2:2019 effectively supported EDTF (https://en.wikipedia.org/wiki/ISO_8601).

One of the chief arguments against date autoformatting was that the only way to make it work was for the user to set a preference. Since the vast majority of readers were anonymous (not using an account) the change would only serve a small minority of readers. The main effect would be to conceal poor choices, which would be visible to most readers but invisible to the editors who would be in a position to fix the poor choices.

In T132308#7084371, @Vladis13 wrote:

Is the date format will has a mark to indicate which the calendar used? As a kind of wiki extension to the EDTF format. Like, Julian date "J44-03-15".
Or will need to keep specifying an additional tag about the date calendar? Like a string description in the text next to the date, an additional parameter in templates, a switch / qualifier in the Wikidata properties.

In T132308#7081934, @Pigsonthewing wrote:

In T132308#7081868, @Jc3s5h wrote:

In the case of Julius Caesar, for example, among historians there is an uncertainty of a few days

Which is precisiely why we need a method of marking-up dates as imprecise, and where applicable as imprecise within defined limits.

In T132308#7081793, @Iniquity wrote:

...These formats are also needed in the dates of birth and death, or in the dates of the work...

In T132308#7071845, @Iniquity wrote:

...Damn guys, really. I'm doing some work here in my community to reduce the used technical dates to one kind YYYY-MM-DD: to drop 12.12.12, 12 July 2020, Jule 12, 2020, 20-12-2000, 12/12/12 etc and you throw that in here. I am very angry.

In T132308#7077938, @Iniquity wrote:

In T132308#7072004, @Iniquity wrote:

...I have almost all dates in templates in projects go through this parser function...

In T132308#6856735, @Pigsonthewing wrote:

In T132308#6856343, @Trappist_the_monk wrote:

Year and month specified, day unspecified in a year-month-day expression (day precision)
Example 4 ‘1985-04-XX’

Does that not provide a way to do month year dates that don't require English?

1985-04-XX means "an unknown individual day in April 1985"; 1985-04 means "the month of April 1985".

In T132308#6856720, @Pigsonthewing wrote:

In T132308#6856130, @Trappist_the_monk wrote:

For the YYYY-MM date format to be accepted at en.wiki, you will have to get it accepted at MOS:NUM.

That may be true for output; but is not true for input. Per Postel's Law, templates should accept YYYY-MM dates (valid in ISO 8601) when entered, regardless of how we decide they should display such values.

Some comments on the status of EDTF.

@Mvolz you write "we've been using EDTF". Who is we? And what discussion on the English Wikipedia are you referring to.

The change in the Dates help page was made by Jarekt at 14:28, 17 July 2018 UTC. The edit summary states

In T57755#6593446, @Bouzinac wrote:

Well, how one could write a timezone for {{Q|Q3062714}}, whose guillotine worked on 10 h 22, Paris Time (given the fact that in 21 JAN 1793, UTC didn't exist) ?

In T57755#6576646, @Multichill wrote:

In T57755#6576583, @Jc3s5h wrote:

....

There is no indication about what time zone will be assumed for the creation or first publication date. You can't expect users to provide correct dates and times if you don't tell them at the time of upload that whatever they enter will be assumed to have the time zone that was in force at the time and place where the photo was taken.

That's how basic human interaction works? If I say that we'll meet next Saturday at 13:00, the other person knows it's in local time (just switched from CEST to CET). Works the same way on Commons. We don't ask users for time zones and we don't record that data.

In T57755#6576706, @Sannita wrote:

In T57755#6576583, @Jc3s5h wrote:

I don't understand how a time could be without a timezone and at the same time have a default timezone.

UTC can be used for this.

Situation 1: time can be accompanied by a time zone, which could be UTC. In this case, the time zone is specified.

! In T57755#6576484, @Multichill wrote:
without timezones and default it to the local time? That is also current practice on Commons, see for example https://commons.wikimedia.org/wiki/File:Northstar_California_Prosser_2.jpg where the time is "3 July 2017, 13:28:12". If someone is really interested in the timezone, they can derive it from the coordinates.

I stopped reading after the first paragraph of the "Description" because it is false. I quote it here, in case it is edited, so my comments will remain clear.

Most of Extended Date/Time Format (EDTF) Specification has been adopted into ISO 8601-1:2019 and ISO 8601-2:2019. These cost 158 Swiss francs and 178 Swiss francs respectively. I am concerned that due to the widespread use of ISO standards, the Library of Congress specification will be left by the wayside. However, because the volunteer developers and editors at the Wikimedia Foundation won't be able to afford the expensive ISO standards, they will be used, but used incorrectly, because the users will rely on unreliable summaries of the standards.

In T207705#4691973, @Pigsonthewing wrote:

In T207705#4690349, @Jc3s5h wrote:

ISO 8601 being one of the rare exceptions

As noted above: "ISO 8601-2019, due in the middle of that year, is expected to support all of the features of EDTF."

Furthermore, as you can read at the linked page: "the EDTF specification is included as a profile of ISO 8601."

In T207705#4690134, @Pigsonthewing wrote:

Jc3s5h: This is now the third venue in which I have told you that there is absolutely no reason why we couldn't apply this only to Gregorian dates, as intended.

@Mvolz I'm not sure what you mean by front end and back end. In any case, there are several methods that may be used to put information into the database, and several that may be used to extract information from the database. Some of these methods like RDF follow standards set by external organizations and we lack the ability to change them.

This specification is a non-starter because Wikidata supports both the Gregorian and Julian calendar while this spec only supports the Gregorian calendar.

@Huji The task description talks about inputting data, not reading data that is already present.

The task description gives a link and claims Arabic, Persian, Hebrew, Thai solar, Minguo and Japanese nengo are supported. But the link claims a time object must be acceptable to PHP's strtotime() function. The linked documentation for that function refers to https://php.net/manual/en/datetime.formats.php for the acceptable formats. That page has no mention of non-Gregorian calendar support.

The first day the Gregorian calendar was used was 15 October 1582. Dates before that should be converted to the other calendar supported by Wikidata, the Julian calendar.

In T95553#4430944, @Jarekt wrote:

Phabricator tickets are not a good place to debate the definitions of phrases like "20th century". The meaning of that term was defined a long time ago and should not depend on Wikidata or other software. I am no expert on the matter but articles on individual centuries on Wikipedias (I checked English and German ones) list "20th century" as timeperiod between January 1, 1901 and ended on December 31, 2000, and that did not change in last 15 years. It might be counter intuitive and not compatible with some software, but it seems like that is the understanding of the term.

I would say the data model must be obeyed, and every aspect of the user interface must obey the data model. The presence of the word "century" in the user interface is a lie. This ticket should be closed as requesting the developers to refine a lie.

In T196674#4430825, @Jarekt wrote:

Current Wikibase definition of beginning and ending years of a century and millennium are in synch with the definitions in the English or German Wikipedia, see :de:19._Jahrhundert for example. Maybe Wikibase software could just provide a link to the item like Q6955 to clarify the definition, and where we can specify the date range.

In T95553#4430864, @Jc86035 wrote:

(My earlier description edits were incorrect and were based on a bad interpretation of the project chat discussion with complaints about external tools. The thing that causes the issue is that some software doesn't realize that Wikibase has nonstandard date handling and has slightly different ranges for centuries and millennia.)

One benefit would be to educate editors that the proper characters to indicate arcminutes or arcseconds is not the apostrophe or double quote respectively.

In T95553#4265262, @kaldari wrote:

@Jc3s5h: I created a new bug for you here: T196674. Now I respectfully ask that you stop spamming this bug with unrelated discussion. Thank you.

In T95553#4262897, @kaldari wrote:

@Jc3s5h: This bug has nothing to do with accurately expressing ranges. This bug is about the fact that "5. century" is not valid English and is confusing to English speakers and should be changed to "5th century" (or if that's too hard, "5 century"). If the word "century" is abandoned, then this bug is moot, but that doesn't mean that this is the right place to talk about accurately representing 100-year ranges.

In T95553#4259766, @kaldari wrote:

Can y'all please open separate bugs for these other issues and not just dump them all in this bug? Thanks.

In T95553#4259798, @kaldari wrote:

It looks like y'all are actually discussing T73459, so please feel free to continue there.

In T95553#4256969, @Lucas_Werkmeister_WMDE wrote:

The insignificant digits are not always set to zero – you can manually set any date you like and then adjust the precision (example edit). And when you do that, the input shows you how your edit will be interpreted:

daniel's statement at 08:28 May 30 2018 is not correct. The spec daniel quotes states "...the time zone should be determined from the timezone field." The time zone field is always set to 0. Therefore the time zone is always UT. Daniel's statement "the spec is quite clear in saying that the time is indeed local time, always" is a direct contradiction of the spec.

@Marsupium , perhaps the routines could be modified to recognize 11 as "day in local time, no time zone specified" and 11.5 as "day with time zone specified". But currently the data model emits precision as an integer, so all the consumers would have to be modified to accept 11.5 as a float. Not only that, if we start emitting 11 as a float, it will break all the existing emitters. If we emit 11 as an integer and 11.5 as a float, that will complicate consuming software because it will have to be flexible and accept either an integer or a float in that position.

All existing dates have been entered without the ability to specify time zones. Sources for dates would typically include encyclopedias, biographic dictionaries (like ''American National Biography''), newspaper obituaries, etc. For such sources, the time zone is the time zone where the event took place, but this was '''falsely''' entered into Wikidata as UTC. The sources generally do not give the time of day of the event, so it is impossible to convert the date in local time to UTC.

This is being discussed at Help talk:Dates on Wikidata.

I'm skeptical about Wikidata considering that it can't get foot right. ( See https://www.wikidata.org/wiki/Q3710 )

I disagree with Lucas_Werkmeister_WMDE. Help:Dates#Precision is correct. It agrees with[[ https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format | the RDF Dump Format ]] documentation and the the JSON documentation.

In T145532#3818480, @Mike_Peel wrote:

Don't forget to include the time zone!

In T112075#3484280, @Lydia_Pintscher wrote:

I think we need to keep consistency in mind here. We have at least one other place where we use the same mechanism right now: selecting a language for monolingual text values. We should also think about selecting globes, precision and time zone in the future. Which of those should behave the same and which differently?

Gut feeling: I like two fields better because it makes it a bit clearer that you are selecting an existing item for the unit and can't just type anything.

Since, historically, some countries have considered the year to begin, and incremented the year number on, a date other than January 1, we should specify that we always consider the year to begin on, and always increment the year number on, January 1, for bot the Gregorian and Julian calendars.

While being clear about the specification, we should specify that for both the Julian and Gregorian calendars, we consider the year to begin on, and the year number to be incremented on, January 1. Historically some countries have had other conventions.

In T146499#3406504, @Esc3300 wrote:

As a first step, we might just consider that "unused" means "unspecified".

Making the timezone default to unspecified would require deciding how "unspecified" will be represented in all possible representations, including JSON, RDF, and the internal database value. If this approach is followed, I would like to see a bot go through the database and change all the existing timezone values to unspecified. Then correct values can be added once the facility to specified them becomes available, and as editors determine which values are correct.

In T132308#3273517, @Trappist_the_monk wrote in part:

In T132308#3273017, @Mvolz wrote in part:

I have opened up the conversation to the wikicite-discuss group as well: https://groups.google.com/a/wikimedia.org/forum/#!topic/wikicite-discuss/a2kRHayAiyo

In T132308#3273017, @Mvolz wrote in part:

Someone there suggested EDTF (extended date time format) https://www.loc.gov/standards/datetime/ - this looks like exactly what we need.

In T132308#3272213, @Whatamidoing-WMF wrote, in part:

In T132308#3269009, @Whatamidoing-WMF wrote:

|date= already doesn't follow IS0 8601. It has never followed ISO 8601. The decision that |date= would not follow ISO 8601 was made years before citoid was created. I'm not sure why "editor doesn't follow ISO 8601 while typing manually" should be separated from "editor still doesn't follow ISO 8601 while using a semi-automated script".

In T132308#3270373, @Trappist_the_monk wrote:

In response to T132308#3268915, @Whatamidoing-WMF:

In T132308#3269009, @Whatamidoing-WMF wrote:

|date= already doesn't follow IS0 8601. It has never followed ISO 8601. The decision that |date= would not follow ISO 8601 was made years before citoid was created. I'm not sure why "editor doesn't follow ISO 8601 while typing manually" should be separated from "editor still doesn't follow ISO 8601 while using a semi-automated script".

If you're not going to follow a standard, you should go far away from the standard to avoid confusion. I'd suggest keeping the date and year field just as they are. In the absence of both those fields, the template could look for citoid-month, citoid-year, citoid-day, citoid-season, and whatever else is required.

Lots of tools and bots examine citations. I'm not really sure which ones examine the rendered HTML, which ones examine the COinS metadata, and which ones examine the wikitext. Anything that examines the wikitext and finds yyyy-mm-00 should reject it as completely non-standard.

In T132308#3260425, @Mvolz wrote:

Worth noting that if you try this in wikidata, i.e. "Fall 2003" you will get "date is malformed..." so they haven't figured it out either. Since iso8601 allows ranges, I can imagine us translating say, "Fall 2003" to be Sept-Nov 2003, which would be 200709/200711 instead of having the special formats. And converting Christmas to December 25th :) (which we could actually do right now with the current set-up actually.) And quarter 1 is Jan-March?

In T132308#3260417, @Mvolz wrote:

Thanks, that's helpful. That suggests that maybe the 00-ed out version that wikidata uses i.e. 2007-05-00 might preferable since there's no way to confuse that with a date range.

The WMF should provide any of its employees working on this an official copy of ISO 8601 and they should be required to read it. Among other things, they would find that the only correct way to represent the current year is "2017". "2017-00" or "2017-00-00" are just wrong. Likewise, the only correct ways to represent the current year and month are "201705" or "2017-05"; "2017-05-00" is wrong.

In T165116#3257456, @Astinson wrote, in part:

In T145898#2645449, @Esc3300 wrote:

Sounds like another calendar model trap (W3C spec requires Gregorian)

I offer a list of test cases.

In T146356#2699683, @hoo wrote:

I have been experimenting with various extreme values of the Julian year, as great as positive 10 billion, and am finding erratic results for the value of the julian date ($jd) and the Gregorian date ($gregorian). I found that for values that are too large, $jd could be 0, negative, or of the same order of magnitude as the year (when it should be about 365 times the year). Also, for values that are too large, $gregorian might or might not be "0/0/0".

In T105100#2629763, @Addshore wrote:

In T105100#2665020, @Addshore wrote:

Yes, so once added the statements are safe to remove, they will not be re added by a future run as I have a list of GUIDs to be skipped in the future! (the lists are essentially the same as the lists published earlier in this ticket).
The GUIDS will remain the same throughout changes to the statement, the one exception is when items are merged, where statements are moved from one item id to another.

In T146356#2663757, @Smalyshev wrote:

I think this idea makes a lot of sense. While I think all far-ago dates that have day precision are most probably data errors (you can't have May 4th in 200000BC, really, at least not using Earthly calendars :) handling these errors in more sane manner is good. However, I think we need then to change precision, if we operate under year precision, we can't just take year and then claim day precision. It'd be claiming false data.

I have made a parallel request at the mediawiki talk page for Wikibase/Indexing/RDF_Dump_Format. There is no mention there of needing to submit a Phabricator ticket to change that documentation.

In T105100#2659241, @Addshore wrote:

@Jc3s5h This should be as simple as removing the instance of qualifier.
We of course have the lists of all statements that will be touched in this run of the script.
If we do ever do a future run we will still have that list of guids that we can avoid!

In T105100#2659241, @Addshore wrote:

@Jc3s5h This should be as simple as removing the instance of qualifier.
We of course have the lists of all statements that will be touched in this run of the script.
If we do ever do a future run we will still have that list of guids that we can avoid!

In T105100#2662859, @Addshore wrote:

In T105100#2659966, @Multichill wrote:

@Addshore looking at https://upload.wikimedia.org/wikipedia/commons/c/ca/Wikidata_Calendar_Model_Decision_Tree.svg your bot is not following it.

The bot followed this tree up to the first level, at which point human intervention is needed.

@Multichill the two edits edits that you have linked appear to be for statements that have no references. Per the decision tree we should try and find references for these to ensure they are correct.

@Addshore asked for some examples of off-by-one years where the year is less than one.

Now that the bot has begun to mark items, what is the procedure to follow when a marked item has been reviewed by an editor and found to be correct?

In T105100#2647800, @Esc3300 wrote:

If you limit Gregorian before 1584 (or 1582) to precision > 9, do you get comparable numbers?

That being said, many of the years marked as Gregorian on https://tools.wmflabs.org/reasonator/?q=Q208233&lang=en seem to be off by one year.
Unfortunatly not all. For years before year 1, probably all of them.

Maybe a special check for anything before year 1 is needed.

Thanks to Addshore for the answer of Sep. 12, 17:54 UT.

It might be useful to state what range of dates will be checked, and what the source of the dates will be, since JSON and RDF dates state dates in different formats, and in the case of the flavor of RDF used in this link the form of the date depends on whether the software that produces the rdf was able to convert the date from Julian to Gregorian or not. Such irregularities might cause some dates that should be marked to be missed.

I see the documentation of the JSON data model has been revised today; see this version from mediawiki. So now we have different formats, but our documentation now explains to users that these formats are intended to be different. That's a lot better. Now a few tweaks are still needed to make various parts reject or vilify dates they consider invalid, but this is a big step forward.

In T142198#2544939, @Smalyshev wrote:

In T142198#2544647, @Smalyshev wrote:

In T142198#2544647, @Smalyshev wrote:

Also, there is controversy over whether the author(s) of the RDF formatter have correcty interpreted what is stored in the data base and are converting the year correctly.

Could you please explain what you mean? I'm the author of the RDF format (at least one of :) and I am not aware of any "controversy" - RDF follows XSD 1.1, and it's the industry standard. If you have observed some bugs there please submit a ticket.

But of course the converter can't be written until the notation used in the source the converter is receiving input from is known with certainty.

We can not know with certainty what whoever have put the dates into Wikibase meant. What however we can do is to define how we interpret the user input and stored data. And we have no choice but to do that - we have to put *something* in the output, and by the fact of putting that into output we are defining our interpretation.

Right now the dates in the DB and JSON follow (roughly) the XSD 1.0 standard for serializing dates, but the RDF follows XSD 1.1 since it is accepted standard in semantic data world.

The last date - 0000-01-30 - doesn't seem to be valid, i.e. it does not describe any actual point on the date scale. Wikibase may sometimes allow to enter nonsensical values (including February 30th, etc.), but RDF can not really represent it.

After the revision of the task description around 1700 UT on Aug 11, I see that the if the date was a Julian calendar date, WQS would convert the Gregorian date received from the RDF to the equivalent Julian date, and not mention in the display which calendar this is. I seriously question the decision of the UI authors to make fussy decisions about when to display the calendar and when not to. I think it would be better, and more resilient to any changes in this area in the UI, to always display the calendar.

I made a chart of what happens around AD 1. All these dates were entered in the sandbox today and all have had the calendar manually set to Gregorian during the entry process. (I accidentally entered 0000-01-22 while taking the default for this, Julian calendar, so I'm ignoring that date.) The diff column is the timestamp value one sees while viewing diffs between different versions of the sandbox.

Oh dear. Blazegraph recently went through an upheaval, switching from XSD 1.0 (where year 0000 does not exist and -0002 = 2 BCE) to XSD 1.1 (where year 0000 does exist and -0001 = 2 BCE). Also, there is controversy over whether the author(s) of the RDF formatter have correcty interpreted what is stored in the data base and are converting the year correctly.

I tend to favor displaying years before AD 1 with a combination of letters and numbers in the user interface, because different conventions have been used about what negative year numbers mean; no such ambiguity exists with "2 BC" or "2 BCE". I don't care whether AD/BC are used vs. CE/BCE.

There is considerable doubt about the correctness of the user interface. I suggest anyone undertaking this must obtain an ironclad definition of the meaning of the exact source WQS is obtaining the data from. Data entered into the database may have been entered with a variety of tools, not just the user interface. There is reason to suspect some of these tools would have treated -0001-01-01 as January 1, 2 BCE, while others would have treated it as January 1, 1 BCE. Thus, looking at examples of well-known BCE dates in the user interface is not a reliable way to judge the definition of the source of data that the user interface pulls from.