Page MenuHomePhabricator

Support `null` to indicate a missing value
Closed, ResolvedPublic

Description

We should support null to indicate that the value is not available. We should also support notNull field schema parameter to indicate that the value must exist.

W3C version of the tabular schema also supports "json-y" way of values - instead of storing data as a list of lists [ [1,2,3], [4,5,6] ], it also allows an object approach - [ {"hdr1": 1, "hdr2": 2, "hdr3": 3}, { ... } ]. This allows for an easy way to indicate missing values - simply don't specify it. It is almost the same as setting it to null, but not entirely the same: missing != null. If we don't support it, we basically equate null and missing.

Event Timeline

Yurik created this task.Dec 9 2016, 1:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 9 2016, 1:25 AM
Yurik added a comment.Dec 12 2016, 7:50 AM

From extensive conv at https://gitter.im/frictionlessdata/chat - seems by the standard, null is always implicitly allowed, and to prevent that, the schema supports constraints.required = true. See full schema definition

Change 326390 had a related patch set uploaded (by Yurik):
Add support for "null" as a valid tab value

https://gerrit.wikimedia.org/r/326390

Change 326393 had a related patch set uploaded (by Yurik):
Bumped graph-shared lib - null support

https://gerrit.wikimedia.org/r/326393

Yurik moved this task from Backlog to Needs review on the Maps-Sprint board.Dec 12 2016, 8:37 AM

Change 326500 had a related patch set uploaded (by Yurik):
Bumped graph-shared lib - null support

https://gerrit.wikimedia.org/r/326500

Change 326393 merged by jenkins-bot:
Bumped graph-shared lib - null support

https://gerrit.wikimedia.org/r/326393

Change 326540 had a related patch set uploaded (by Yurik):
Add support for "null" as a valid tab value

https://gerrit.wikimedia.org/r/326540

Change 326390 merged by jenkins-bot:
Add support for "null" as a valid tab value

https://gerrit.wikimedia.org/r/326390

Change 326540 merged by jenkins-bot:
Add support for "null" as a valid tab value

https://gerrit.wikimedia.org/r/326540

Change 326853 had a related patch set uploaded (by Yurik):
Revert special null treatment

https://gerrit.wikimedia.org/r/326853

MaxSem reopened this task as Open.Dec 13 2016, 12:02 AM

Change 326500 abandoned by Yurik:
Bumped graph-shared lib - null support

https://gerrit.wikimedia.org/r/326500

Change 326853 merged by jenkins-bot:
Revert special null treatment

https://gerrit.wikimedia.org/r/326853

Yurik closed this task as Resolved.Dec 13 2016, 3:49 AM

We can still close it - that special null treatment was working as is in production without alternations. my previous patch to "fix it" was actually a mistake - i realized that the fix would introduce weird consequences for graphs, so reverted it. Both patches will be on the next train, negating one another.

debt added a subscriber: debt.Dec 13 2016, 3:48 PM

Does this mean that if a user doesn't enter in a value - errors will still show? And that we're continuing to not support "null" values?

Yurik added a comment.Dec 13 2016, 4:40 PM

Per JSON specification, you cannot "not enter a value". If the user enters [ 1, 2, 3 ], they entered a three values for a row. The system would reject those values unless the schema at the top specifies that there must be exactly 3 fields, each being a number. Initially [ 1, 2, null ] would have been rejected, but with this minor rule relaxing, the system is slightly more lenient - it allows the null to be used instead of any of the valid value.

Deskana moved this task from Needs review to Done on the Maps-Sprint board.Apr 19 2017, 4:27 PM