| Architecture meetings | ||
|---|---|---|
| 13:00 PT ArchCom Planning Meetings | upcoming | all since 2016-03-30 |
| 14:00 PT ArchCom-RFC Meetings | upcoming | all since 2015-09-09 |
@Yurik recently posted to wikitech-l:
We have had some good feedback for the new shared tabular data feature, and we are getting ready to deploy it in production. It would be amazing if you can give it a final look-over to see if there are any blockers left.
The first stage will be to enable Data:*.tab pages on Commons, and allow all other wikis direct access to it via Lua code and Graph extension. All data at this point must be licensed under CC0. More licensing options are still under discussion, and can be easily added later.
In line with the "release early, release often", we will not have any elaborate data editing interface beyond the raw JSON code editor for the first release. Our initial target audience is the more experienced users who will evaluate and test the new technology. Once the underlying tech is stable and prooven, we will work on making it more accessible to the general audience.
This was followed with a series of links, including to T134426, http://data.wmflabs.org,
mw:Extension:JsonConfig/Tabular, and a Commons discussion
The subsequent discussion made clear these are (some of?) the relevant tasks:
...and that there are many other tasks in Commons-Datasets. TechCom agreed this would be a good topic for this week's RFC meeting, if @Yurik is available.
<wm-labs-meetbot`> Meeting ended Wed Jun 15 22:05:08 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
15:05:10 W<wm-labs-meetbot`> Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-06-15-21.02.html
15:05:10 W<wm-labs-meetbot`> Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-06-15-21.02.txt
15:05:10 W<wm-labs-meetbot`> Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-06-15-21.02.wiki
15:05:10 W<wm-labs-meetbot`> Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-06-15-21.02.log.html
#wikimedia-office: T120452 Technical aspects of Data namespace blob storage
Meeting started by robla at 21:02:08 UTC (full logs).
Meeting summary
Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ (robla, 21:02:35)
https://phabricator.wikimedia.org/T114454 (cscott, 21:10:28)
much of the first half of the discussion was about defining datatypes (robla, 21:30:21)
DanielK_WMDE and yurik agree to try to avoid naming conflicts (e.g. with wikibase types) (robla, 21:32:11)
yurik agrees to create a Phab task for use as an ArchCom-RFC (robla, 21:47:13)
https://phabricator.wikimedia.org/T137929 (yurik, 22:02:52)
conversation will continue at https://phabricator.wikimedia.org/T137929 (robla, 22:03:22)
https://phabricator.wikimedia.org/T137930 (yurik, 22:03:24)
https://phabricator.wikimedia.org/T137930 geojson (robla, 22:03:56)
Meeting ended at 22:05:08 UTC (full logs).
Action items
(none)
People present (lines said)
yurik (125)
DanielK_WMDE (66)
cscott (51)
robla (35)
SMalyshev (29)
brion (28)
aude (23)
stashbot (9)
TimStarling (4)
wm-labs-meetbot` (4)
MaxSem (2)
YairRand_ (1)
Krinkle (0)
Generated by MeetBot 0.1.4.
Log:
| 1 | 21:02:08 <robla> #startmeeting T120452 Technical aspects of Data namespace blob storage |
|---|---|
| 2 | 21:02:08 <wm-labs-meetbot`> Meeting started Wed Jun 15 21:02:08 2016 UTC and is due to finish in 60 minutes. The chair is robla. Information about MeetBot at http://wiki.debian.org/MeetBot. |
| 3 | 21:02:08 <wm-labs-meetbot`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. |
| 4 | 21:02:08 <wm-labs-meetbot`> The meeting name has been set to 't120452_technical_aspects_of_data_namespace_blob_storage' |
| 5 | 21:02:08 <stashbot> T120452: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452 |
| 6 | 21:02:09 <yurik> hi everyone, thx for making it to the discussion that shall change the face of the earth... again |
| 7 | 21:02:35 <robla> #topic Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ |
| 8 | 21:02:45 <cscott> let's change all the things! |
| 9 | 21:02:58 <yurik> cscott, oh no, not again! |
| 10 | 21:04:12 <SMalyshev> question: so this thing is using content handler from JsonConfig? |
| 11 | 21:04:38 <robla> so, Yuri, things ArchCom just discussed in our last meeting (E212) is "is this a question for ArchCom?" and "should this be a formal RFC?" |
| 12 | 21:04:56 <yurik> SMalyshev, correct |
| 13 | 21:05:34 <DanielK_WMDE> I propose that we don't discuss "on which domain shall it live" today. that's a community/product question which should be discussed elsewhere, i think |
| 14 | 21:05:53 <yurik> robla, there are clearly two topics, just as brion said. The social aspect - should this be on commons - should probably be discussed with the community, and so far the majority of commons seems to be in favour. What I do want this meeting to address is the technical aspects |
| 15 | 21:06:00 <DanielK_WMDE> that is to say, that's the part of this that i think is not a question for archcom |
| 16 | 21:06:21 <yurik> ok, seems like we are in an agreement on that one |
| 17 | 21:06:21 <SMalyshev> yurik: ok. It may be worth it to make it more search-friendly, but I guess that is generic for all JsonConfig things then |
| 18 | 21:06:49 <yurik> SMalyshev, sure, search friendliness is definitly on the todo list |
| 19 | 21:06:51 <DanielK_WMDE> SMalyshev: how well do you think will the structured search interface you are working on work with tabluar data? |
| 20 | 21:07:20 <DanielK_WMDE> would it be possible to search for values in spoecific columns, for example? if the content handler does it right, i mean |
| 21 | 21:07:23 <SMalyshev> DanielK_WMDE: it might, but I wouldn't go that far for starters. I'd just start with something like being able to search the description of the dataset |
| 22 | 21:07:28 <cscott> (i personally like the idea of making a namespace available from every wiki, then letting social pressures move stuff around as necessary. eg if i want to index the winners of the US "Dancing with the Stars" maybe that live on enwiki in Data:DancingWithTheStars.json, since that's US-specific and there's a different "dancing with the stars" in basically every country.) |
| 23 | 21:07:36 <SMalyshev> DanielK_WMDE: search inside the data is much bigger fish to fry |
| 24 | 21:08:12 <cscott> hm. i agree that search can be done later. |
| 25 | 21:08:20 <DanielK_WMDE> it actually shouldn't be much harder than making the description searchable. but yea, let's start6 small |
| 26 | 21:08:21 <SMalyshev> yurik: my next question: is it attached to specific namespace or can be on any namespace? |
| 27 | 21:08:27 <cscott> like search of wiki articles is essentially separate from storage/viewing those articles |
| 28 | 21:08:31 <brion> of course we'll have articles on several different countries' shows in each language... |
| 29 | 21:08:49 <SMalyshev> cscott: not anymore :) |
| 30 | 21:09:06 <cscott> brion: right, at that point it starts to make sense to move it to commons. but the community can decide this themselves organically. |
| 31 | 21:09:12 <SMalyshev> cscott: at least not completely. I'm working on a patch that makes ContentHandler handle both |
| 32 | 21:09:14 <brion> hmmm, interesting |
| 33 | 21:09:35 <cscott> SMalyshev: "not anymore" re search? not sure what you were responding to. |
| 34 | 21:09:36 <DanielK_WMDE> SMalyshev: i suggest to rely on the content model, not the namespace, for any special processing. currently, content models are mostly bound to namespaces, but that can change |
| 35 | 21:09:38 <brion> cscott: so put the system on *every* wiki that doesn't opt-out, plus a common backing, like File: uploads? |
| 36 | 21:09:50 <SMalyshev> cscott: yes, re search. But let's not get too offtopic :) |
| 37 | 21:09:58 <cscott> brion: that would be my suggestion. it also ensures we're running the same software everywhere. |
| 38 | 21:10:12 <cscott> i have a proposal for a new template system that could really use data blob storage. for example. |
| 39 | 21:10:13 <SMalyshev> DanielK_WMDE: yes, I agree. But how you specify that you want to create page in certain content model? |
| 40 | 21:10:19 <brion> *nod* sensible :D |
| 41 | 21:10:27 <yurik> cscott, i am not too happy about allowing it everywhere. We already got burnt by it waaay too many times - i would much rather keep it in one place, allow multiple wikis to reuse it, and name it accordingly - "dancing with the stars US 2015" |
| 42 | 21:10:28 <cscott> https://phabricator.wikimedia.org/T114454 |
| 43 | 21:10:50 <DanielK_WMDE> SMalyshev: currently, via the namespace or a title suffix (like .css or .js) |
| 44 | 21:10:56 <cscott> yurik: sure, default should be commons, but that should be social pressure not technical restriction. |
| 45 | 21:11:02 <SMalyshev> DanielK_WMDE: ok, got it |
| 46 | 21:11:11 <SMalyshev> sounds reasonable |
| 47 | 21:11:17 <DanielK_WMDE> SMalyshev: but that'S just at the time of creation. after that, we should just look at the model associated with the page |
| 48 | 21:11:26 <robla> #chair DanielK_WMDE brion robla TimStarling Krinkle |
| 49 | 21:11:26 <wm-labs-meetbot`> Current chairs: DanielK_WMDE Krinkle TimStarling brion robla |
| 50 | 21:11:32 <DanielK_WMDE> it's actually stored separately for every revision |
| 51 | 21:11:41 <yurik> cscott, i would wait until community explicitly demands that feature. It is much easier to enable than to disable |
| 52 | 21:11:55 <yurik> we can always enable it later if needed, but disabling is next to impossible |
| 53 | 21:12:05 <yurik> so for the step 1 i would really like just 1 wiki |
| 54 | 21:12:09 <DanielK_WMDE> yurik: T120452 says "CSV, TSV, JSON, XML". But we are down to just a specific JSON format now, right? |
| 55 | 21:12:10 <stashbot> T120452: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452 |
| 56 | 21:12:11 <yurik> and wait for the feedback |
| 57 | 21:12:18 <SMalyshev> only thing is that now we are using model that's called JsonConfig for some things that are definitely not config... but I guess it's too late to rename... |
| 58 | 21:12:30 <yurik> DanielK_WMDE, my first goal is to get two types of data out : tabular and maps |
| 59 | 21:12:41 <cscott> yurik: will https://phabricator.wikimedia.org/T91162 let me refer to the data store on commons as if it were local? (like instantcommons does)? |
| 60 | 21:12:50 <yurik> maps would allow geojson storage (so that all maps can overlay with extra stuff), and tabular - .... is tabular :) |
| 61 | 21:13:00 <DanielK_WMDE> yurik: maps as in geo-shapes? |
| 62 | 21:13:05 <yurik> DanielK_WMDE, yes |
| 63 | 21:13:06 <DanielK_WMDE> or full maps? |
| 64 | 21:13:08 * robla will brb |
| 65 | 21:13:08 <cscott> yurik: one issue might be (for example) country-specific data, which then gets hung up on different countries/wiki's ideas of which are valid countries |
| 66 | 21:13:09 <yurik> and pushpins |
| 67 | 21:13:12 <cscott> ie, is "taiwan" a country? |
| 68 | 21:13:37 <cscott> yurik: allowing zhwiki to override the table from commons with a zhwiki-specific table is a valuable way to defuse that situation |
| 69 | 21:13:50 <cscott> and i think wikidata already has something like this, where certain facts are only true for certain wikis? |
| 70 | 21:13:59 <DanielK_WMDE> cscott: they can just choose to use a different table. much simpler solution. |
| 71 | 21:14:13 <cscott> DanielK_WMDE: how does wikidata handle this? |
| 72 | 21:14:15 <yurik> cscott, no, not via shadow. The current implementation will only target Lua and Graph users at first, which means Lua will simply say mw.data.get('Page.tab') - and use that data |
| 73 | 21:14:23 <SMalyshev> yurik: I wonder if we need a higher level API to operate with such data. I.e. if I want to store a tabular data set, I don't want to need to know specific JSON scheme (which could also change)... |
| 74 | 21:14:23 <DanielK_WMDE> but i thin the "where does it live" question is out of scope here. |
| 75 | 21:14:28 <yurik> we can add shadow later if requested |
| 76 | 21:15:10 <SMalyshev> yurik: and by operate I mean not just read (Lua probably covers that) but also write |
| 77 | 21:15:15 <TimStarling> presumably Lua will see the decoded object, not the JSON-encoded format? |
| 78 | 21:15:22 <cscott> i'm fine with deploying first without instantcommons/shadow and on a single wiki, but i'd like to state for the record that, if this functionality turns out to be useful, we'll eventually need that functionality. we should ensure that we're not *foreclosing* that possibility, even if we're not initially enabling it. |
| 79 | 21:15:29 <DanielK_WMDE> cscott: in theory, wikis can pick specific statements over the default for a specific property, or they can filter by well-known authorities being cited as sources. i don't think this is actually being done, but the data model is specifically designed to allow this |
| 80 | 21:15:29 <yurik> cscott, the idea here is to provide the most basic usage that will cover 80%. If we get strong desire for 1) multi-wiki storage, or 2) multi-wiki overrides, we can always add that |
| 81 | 21:16:02 <yurik> TimStarling, correct |
| 82 | 21:16:17 <yurik> TimStarling, more specifically, Lua will get the json as table |
| 83 | 21:16:40 <DanielK_WMDE> yurik: i see some overlap between the use cases of wikidata queries and tabular data. it would be nice if the formats and interfaces would be very similar, of not identical. |
| 84 | 21:16:40 <yurik> so that it has access to all the meta fields |
| 85 | 21:16:47 <yurik> BUT, we can provide additional helpers to resolve the multi-lingual resolution |
| 86 | 21:16:47 <cscott> again, not many people have grokked T114454 yet, but the basic idea there is to separate code, content, and presentation, so every template potentially will have a "data" component, along with the "code" and the "presentation" component. |
| 87 | 21:16:47 <stashbot> T114454: [RFC] Visual Templates: Authoring templates with Visual Editor - https://phabricator.wikimedia.org/T114454 |
| 88 | 21:17:02 <yurik> DanielK_WMDE, agree - we have discussed it briefly with Lydia_WMDE |
| 89 | 21:17:41 <cscott> assuming some basic separation of that happens in the future, we'll want the data namespace to be roughly on par with the template namespace. ie, shadowed from a default on commons, override-able from specific wikis. |
| 90 | 21:17:56 <YairRand_> maybe have data blobs be a wikidata datatype, like commons file |
| 91 | 21:18:05 <DanielK_WMDE> yurik: to be compatible with wikidata, the representation of data values would have to become more complex. but i'm not sure whether we should require that, or offer it as an optional ferature. |
| 92 | 21:18:12 <DanielK_WMDE> yurik: btw, multilingual is pretty complex when it gets to the nitty gritty. do you think it's really needed from the start? |
| 93 | 21:18:14 <yurik> DanielK_WMDE, lets sync up afterwards to see if we can match wikidata api with this, or if they should go different routes |
| 94 | 21:18:46 <brion> could also do things like instead of multilingual text, refer to a wikidata item and then look up its data by name .... though that may have perf issues with large batches |
| 95 | 21:18:47 <brion> :D |
| 96 | 21:18:49 <DanielK_WMDE> yurik: not tonight, i'm going to bed after this :) will you be at wikimania? |
| 97 | 21:19:06 <yurik> DanielK_WMDE, absolutelly - i really think simple "multi-lingual" feature that will allow a fallback is something we need from the start |
| 98 | 21:19:07 <TimStarling> ideally Lua would see a read-only wrapper object like what is returned from mw.loadData() |
| 99 | 21:19:12 <DanielK_WMDE> brion: yes, wikidata Q-id is a very useful datatype to have |
| 100 | 21:19:16 <yurik> DanielK_WMDE, sadly, no Wikimania for me - no budget :( |
| 101 | 21:19:28 <TimStarling> that way you don't have to clone it for each #invoke instance |
| 102 | 21:19:39 <DanielK_WMDE> yurik: sad. let's find another time and place then. |
| 103 | 21:20:01 * aude wavez |
| 104 | 21:20:10 <DanielK_WMDE> yurik: it would be nice to re-use the language-fallback stuff we have in wikidata. we should factor it out into a library, i guess. |
| 105 | 21:20:15 <yurik> DanielK_WMDE, indeed it is :( Yes, lets. I will schedule a hangout with you. Anyone else - pls poke me if you want to parttake- |
| 106 | 21:20:19 <DanielK_WMDE> hey aude |
| 107 | 21:20:33 <yurik> DanielK_WMDE, sure, but i already have something like that for the zero banners that i'm reusing |
| 108 | 21:20:35 <yurik> but sure |
| 109 | 21:20:58 * robla returns to the meeting he's allegedly chairing :-) |
| 110 | 21:21:11 <SMalyshev> yurik: so what do you think about having higher-level API to manipulate specifically the tabular data? |
| 111 | 21:21:17 * yurik thinks chairing !== participating ;) |
| 112 | 21:21:40 <yurik> SMalyshev, "manipulate" is out of the scope i think at this point. I'm all for it though :) |
| 113 | 21:21:54 <DanielK_WMDE> yurik: do you want to start out with your own (simpler) data types for now (and spend tiem to specify them properly)? or do you want to go with the representation that is used by the wikidata? that's already in reusable libraries |
| 114 | 21:21:55 <SMalyshev> yurik: thinking forward :) |
| 115 | 21:21:57 <yurik> especially because i can totally see some pages being custom-defined to store data in the backing SQL |
| 116 | 21:22:03 <yurik> SMalyshev, ^ |
| 117 | 21:22:38 <yurik> DanielK_WMDE, i would like to match datatypes in wikidata as much as possible, but probably only provide a subset of them from the start |
| 118 | 21:23:04 <SMalyshev> yurik: well, there's two venues here: a) run external query, store results on wiki (I don't want to know too much details about how wiki stores it) |
| 119 | 21:23:14 <DanielK_WMDE> yurik: yea, an "table aware" storage backend is an interesting idea, and fits in with the blob-store refactoring i'm thinking about. but it's for later. |
| 120 | 21:23:14 <yurik> if that means re-implementing some of it first - lets, because otherwise we might spend years making it perfect only to realize that community needs are totally orthogonal to wikidata usage |
| 121 | 21:23:25 <SMalyshev> yurik: and b) represent internal query as data set on wiki (e.g. WDQS query) |
| 122 | 21:23:47 <yurik> SMalyshev, yep, that's what DanielK_WMDE is talking about i think. But again, lets not discuss it now :) |
| 123 | 21:23:49 <SMalyshev> all that needs clean API so that clients don't know too much |
| 124 | 21:23:57 <yurik> otherwise we might be redesigning SQL engine next ;) |
| 125 | 21:24:16 <SMalyshev> that's why I mention it - if we make it too specific now, it'd be hard to change it ater |
| 126 | 21:24:18 <SMalyshev> *later |
| 127 | 21:24:47 <yurik> SMalyshev, that i agree. But remember, the use case here is for Lua to GET EVERYTHING, and deal with it. If we say we want SQL-like GET EVERYTHING THAT MATCHES THE WHERE CLAUSE, we might get into all sorts of weird issues |
| 128 | 21:25:13 <yurik> especially because we might go the route that is not needed (yet or ever) |
| 129 | 21:25:26 <brion> huge-data-set needs are quite different yes |
| 130 | 21:25:27 <DanielK_WMDE> yurik: ok then. have you looked at how wikidata represents data values? e.g. look at https://www.wikidata.org/wiki/Special:EntityData/Q42.json |
| 131 | 21:25:38 <SMalyshev> yurik: Lua is a good enough API if we don't get too specific about the structure |
| 132 | 21:26:08 <yurik> exactly brion - that's what we actually discussed in the task earlier - dealing with large datasets is a very different beast, with a different reqs |
| 133 | 21:26:11 <aude> DanielK_WMDE: i'm not sur eabout duplicating all the json for each value |
| 134 | 21:26:12 <DanielK_WMDE> yurik: e.g. we have something like {"snaktype":"value","property":"P577","datavalue":{"value":{"time":"+2002-01-01T00:00:00Z","timezone":0,"before":0,"after":0,"precision":9,"calendarmodel":"http://www.wikidata.org/entity/Q1985727"},"type":"time"},"datatype":"time"} |
| 135 | 21:26:13 <stashbot> P577 (An Untitled Masterwork) - https://phabricator.wikimedia.org/P577 |
| 136 | 21:26:24 <aude> if all the values have the same calendar, e.g. |
| 137 | 21:26:30 <DanielK_WMDE> yurik: the "value" thing is what i think should be in your table fields. |
| 138 | 21:26:54 <aude> then have calendar, before /after + array of timestamps? |
| 139 | 21:27:15 <DanielK_WMDE> yurik: the "time" data type is akind of a nice nasty example. it'S json representation really isn't too great, and i'd love to change it... we'll probably have to use a new type id for the new version, not sure yet |
| 140 | 21:27:24 <aude> maybe precision might vary though |
| 141 | 21:27:37 <yurik> DanielK_WMDE, i would prefer to go with the tabular data as defined by the industry for tabular data (see the bug), but for specific datatypes like time - sure |
| 142 | 21:28:01 <DanielK_WMDE> aude: yea, i at least wouldn't dublicate the time. we could have a "defaults" row, that gets merged into every value. |
| 143 | 21:28:11 <aude> the json is very verbose |
| 144 | 21:28:12 <DanielK_WMDE> a bit hacky, but woudl work... |
| 145 | 21:28:21 <aude> just saying... |
| 146 | 21:28:24 <yurik> DanielK_WMDE, btw, time is not part of this proposal just yet :) |
| 147 | 21:28:32 <yurik> too complex to have it in ver1 |
| 148 | 21:28:42 * aude agrees with yurik |
| 149 | 21:28:49 <aude> start simple |
| 150 | 21:29:06 <yurik> it can be easily added later - simply add a new type, and make the value object mean as DanielK_WMDE described above |
| 151 | 21:29:22 <DanielK_WMDE> yurik: yea, sure. for string-based types, sime literals work fine. for numbers, too. once we get into measured quantities, things get more complex |
| 152 | 21:29:38 <yurik> DanielK_WMDE, default row is fairly complex - should it be the "null" that gets used as the default? |
| 153 | 21:29:41 <aude> counts could be ok |
| 154 | 21:29:45 <SMalyshev> oh let's not get into units :) |
| 155 | 21:30:07 <yurik> agree, lets get back to overall strategy :) |
| 156 | 21:30:13 <DanielK_WMDE> yurik: we should make sure to avoid naming conflicts. if you define a type name and use it with a different format than wikibase does, that will become annoying |
| 157 | 21:30:21 <robla> #info much of the first half of the discussion was about defining datatypes |
| 158 | 21:30:24 <yurik> DanielK_WMDE, agree |
| 159 | 21:31:14 <DanielK_WMDE> yurik: the idea with the defaults row wats that you can e.g. say that all dates in a column use the same caledar, or all coordinates refer to earth, without havign to repeat that info for every field. but that's an optimization that can be added later. |
| 160 | 21:31:49 <yurik> ver1 datatypes: strting, numbers, multilingual strings. I don't even know if i want to allow bools for now. |
| 161 | 21:32:11 <robla> #info DanielK_WMDE and yurik agree to try to avoid naming conflicts (e.g. with wikibase types) |
| 162 | 21:32:13 <aude> multilingual gets somewhat complex |
| 163 | 21:32:23 <yurik> these three types should cover almost 90% of the usecases from the start - simply because it will be Lua doing the processing and presenting of the dat |
| 164 | 21:32:24 <yurik> data |
| 165 | 21:32:25 <SMalyshev> yurik: bools are just numbers 1/0. Or strings yes/no :) |
| 166 | 21:32:33 <aude> and numbers (units? no units / counts?) |
| 167 | 21:32:34 <DanielK_WMDE> yurik: i think the way you represent multilingual is different from what the DataValues lib does. but wikidata doesn't use multilingual yet, so it can be changed |
| 168 | 21:32:44 <yurik> aude, simple JSON numbers |
| 169 | 21:32:55 <yurik> which means if you want units, you add a string column |
| 170 | 21:32:56 <aude> yurik: like counts |
| 171 | 21:33:00 <SMalyshev> aude: I don't think we need units and associated headache. We havent' properly figured them on wikidata even |
| 172 | 21:33:14 <aude> SMalyshev: that's why i am asking :) |
| 173 | 21:33:16 <DanielK_WMDE> robla: i think data types are a crucial issue. but i agree that we should leave some room for other topics ;) |
| 174 | 21:33:41 <yurik> DanielK_WMDE, lets make this part of our wikidata-jsonconfig sync up meeting |
| 175 | 21:34:06 <yurik> are there any other issues that people are concerned about? |
| 176 | 21:34:22 <aude> yurik: btw, maybe you can visit berlin before SOTM in belgium? |
| 177 | 21:34:27 * DanielK_WMDE thinks that values from a query api will actually be full "snaks"... |
| 178 | 21:34:30 <SMalyshev> yurik: are there any limits on how big it can get? |
| 179 | 21:34:36 <aude> and we can talk more of the details |
| 180 | 21:34:44 <yurik> SMalyshev, 2mb - same as a wiki page |
| 181 | 21:34:50 <DanielK_WMDE> oh, a visit sounds nice! |
| 182 | 21:34:53 <SMalyshev> ok |
| 183 | 21:34:55 <yurik> because it uses storage engine |
| 184 | 21:35:36 <DanielK_WMDE> do you think we will want to expand to very large data sets later?` |
| 185 | 21:35:37 <robla> yurik: can/should you formalize T120452 as an ArchCom-RFC? |
| 186 | 21:35:38 <stashbot> T120452: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452 |
| 187 | 21:35:53 <yurik> DanielK_WMDE, aude, do we really want to wait until september to deploy this? JsonConfig has been in production for the past 2 years, for all wikis (as part of the zero system) |
| 188 | 21:35:55 <robla> (at least the technical side)? |
| 189 | 21:36:28 <yurik> DanielK_WMDE, i don't want to tackle large datasets until after this thing has had some usage, e.g. half a year |
| 190 | 21:36:30 <robla> (perhaps T134426 is the right one to focus on) |
| 191 | 21:36:30 <stashbot> T134426: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426 |
| 192 | 21:36:51 <DanielK_WMDE> yurik: that's a pretty brisk pace ;) |
| 193 | 21:37:04 <yurik> DanielK_WMDE, agree, i will wait a year until large datasets :D |
| 194 | 21:37:05 <DanielK_WMDE> if you really want to support data sets by then, you better start thinking abotu that early |
| 195 | 21:37:33 <yurik> but yes, it should be in the back of our minds, but shouldn't be fully speced until later |
| 196 | 21:38:03 <yurik> robla, i think there is another task that formalizes how the system works |
| 197 | 21:38:05 * yurik looks |
| 198 | 21:39:13 * robla waits patiently |
| 199 | 21:40:23 <DanielK_WMDE> how about directly transclusing a table into a wiki page? how would that work? do we newed that? or do we rely on lua for that? |
| 200 | 21:40:29 <yurik> robla, i think its in https://www.mediawiki.org/wiki/Extension:JsonConfig/Tabular |
| 201 | 21:40:55 <yurik> DanielK_WMDE, even though i do have it implemented (as a template expansion), i don't think its a usecase |
| 202 | 21:41:02 <yurik> simply because there is really no big reason for it |
| 203 | 21:41:18 * robla notes that the Extension:JsonConfig points to T120452 |
| 204 | 21:41:18 <yurik> it is always very usage dependent - e.g. show a list generated from a table |
| 205 | 21:41:36 <DanielK_WMDE> yea, you'd always want som custom stuff anyway |
| 206 | 21:41:39 <robla> yurik: is T120452 the right Phab task? |
| 207 | 21:41:40 <stashbot> T120452: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452 |
| 208 | 21:42:35 <DanielK_WMDE> can we remove the formats from the title? i think they are misleading now. |
| 209 | 21:43:09 <yurik> DanielK_WMDE, agree, but please keep in mind that as part of this discussion i would really like geojson (maps overlays) to be agreed on as well |
| 210 | 21:43:20 <yurik> actually geojson is much simpler than tabular |
| 211 | 21:43:33 <yurik> it is a well established format, and we are already heavily using it in maps |
| 212 | 21:43:36 <robla> yurik: what Phab task do you want to declare as an ArchCom-RFC? |
| 213 | 21:43:55 <yurik> robla, that one is fine i think - we some refining to the title and description |
| 214 | 21:43:55 <cscott> I wonder if extensions are really the right way to select data type? |
| 215 | 21:44:06 <yurik> cscott, what do you mean? |
| 216 | 21:44:22 <cscott> since there is some discussion of types, for instance, it might be that we start with a very simple "json" but later have a more typeful "json" with the date-type figured out, etc. |
| 217 | 21:44:31 <cscott> .json is going to get overloaded quickly |
| 218 | 21:44:40 <cscott> mime types would be much nicer |
| 219 | 21:44:47 <cscott> but then that begs the question of where they get stored |
| 220 | 21:44:51 <yurik> cscott, i actually don't want .json - it will be heavily misused from the begining, no? |
| 221 | 21:44:58 <cscott> still, storing the data type separately from the data/article name is not a bad thing. |
| 222 | 21:44:59 <yurik> and we won't be able to do some proper editing |
| 223 | 21:45:15 <DanielK_WMDE> cscott: internally, it will be represented as a content model id. the extension is one way to indicate that. |
| 224 | 21:45:17 <yurik> if we from the begining define a rigid structure, we can add useful tools |
| 225 | 21:45:33 <yurik> so for tabular, VE can have a nice editor of values (like a spreadsheet) |
| 226 | 21:45:39 <SMalyshev> .json is way too generic... I think it'd be better if tables and geojson had their own spaces |
| 227 | 21:45:40 <yurik> actually it won't even be a VE on commons |
| 228 | 21:45:40 <cscott> DanielK_WMDE: will we be able to eventually just associate a mime type with the content? |
| 229 | 21:45:43 <DanielK_WMDE> hm... will geo-shapes and tables live in the same namespace? with different suffixes/extensions? |
| 230 | 21:45:53 <robla> yurik: If that becomes an ArchCom-RFC, then you won't be the assignee, and Danny Horn will be the author. Is that the desired outcome? |
| 231 | 21:45:56 <yurik> DanielK_WMDE, yes |
| 232 | 21:46:05 <cscott> i don't mind Data: as the namespace. i'd rather have that than GeoJson: Tables: etc etc |
| 233 | 21:46:14 <yurik> robla, i hear you, ok, i will create a new task |
| 234 | 21:46:37 <yurik> DanielK_WMDE, example: Data:Don Qixote Trip in Spain.geojson |
| 235 | 21:46:37 <DanielK_WMDE> cscott: a content model id, not a mime type. the mime type specifies a serialization format, like json or xml. that's also stored, but kind of redundant. the important info is what model/vocabulary/scheme the data is using. |
| 236 | 21:46:45 <cscott> fwiw Scribunto/JS has this same issue -- there's no way to specify which *language* the module is in, in the Module: namespace. |
| 237 | 21:46:49 <MaxSem> I agree with cscott |
| 238 | 21:46:51 <DanielK_WMDE> cscott: we already do that. that's how contenthandler works. |
| 239 | 21:46:57 <brion> my one concern about separating type is that if a table changes type, will that break usage? :) |
| 240 | 21:47:11 <yurik> that's why we from the begining define extensions |
| 241 | 21:47:13 <robla> #info yurik agrees to create a Phab task for use as an ArchCom-RFC |
| 242 | 21:47:19 <brion> (eg if you change an image from .png to .svg you can still use it the same from wiki side, but for tables it may matter more) |
| 243 | 21:47:22 <cscott> brion: possibly, but that's no different from a rename breaking usage, or any other edit breaking usage. |
| 244 | 21:47:28 <yurik> JsonConfig will be set up to only allow pages that match REGEX |
| 245 | 21:47:32 <brion> *nod* |
| 246 | 21:47:36 <DanielK_WMDE> cscott: you could indeed use a file extension to indicate whether a modules is JS or Lua. Just add .js or .lua |
| 247 | 21:47:40 <brion> and we really should rename JsonConfig ;) |
| 248 | 21:47:44 <yurik> so it will be Data:.*\.tab |
| 249 | 21:47:54 * cscott is not a fan of file name extensions |
| 250 | 21:47:56 <yurik> no other pages will be creatable in the data namespace |
| 251 | 21:48:00 <cscott> not i18n friendly |
| 252 | 21:48:04 <DanielK_WMDE> cscott: internally, that would just define the content model to use when creating the page |
| 253 | 21:48:06 <cscott> not human friendly, really |
| 254 | 21:48:18 <yurik> cscott, the only other option is to have multiple namespaces - and the community (and i personally) really hate that |
| 255 | 21:48:35 <brion> well, the other option is to have some sort of content model selection in the creation process |
| 256 | 21:48:40 <DanielK_WMDE> i kind of like to have that info in the title, cscott |
| 257 | 21:48:41 <brion> which implies UI etc |
| 258 | 21:48:44 <cscott> no, i'm just saying that the content model should be defined separately (as DanielK_WMDE indicates is already the case under the covers) and not rely on filename extensions |
| 259 | 21:48:45 <yurik> brion, sure, that can also work |
| 260 | 21:49:15 <cscott> DanielK_WMDE: but the info in the title doesn't mean anything unless you speak english -- or "hacker english" at least |
| 261 | 21:49:29 <cscott> and "geojson" doesn't really mean anything to even english speakers |
| 262 | 21:49:41 <DanielK_WMDE> cscott: i like to do both. we *can* handle different models without any indicator in the title, but it's *nice* to have that indicator there. we already do this for .css and .js in the MediaWiki and User namespaces |
| 263 | 21:49:56 <yurik> cscott, brion, we could create an elaborate system for model selection - is that an absolute blocker/requirement? I really feel that since data will be very technically oriented, people will actually find it better usable |
| 264 | 21:50:01 <cscott> DanielK_WMDE: that will probably have to be good enough for now. |
| 265 | 21:50:02 <yurik> just like we have File:Blah.json |
| 266 | 21:50:16 <yurik> exactly |
| 267 | 21:50:21 <yurik> i really like that indicator |
| 268 | 21:50:44 <TimStarling> "geojson" hopefully means something to the people who are writing lua modules |
| 269 | 21:50:45 <cscott> yurik: i'd just like it clear during the document/evangelization process that filename extensions may be a convenient *shortcut* for specifying the data type, they are only a stopgap and not strictly speaking required. (especially if your native language is not english) |
| 270 | 21:50:47 <brion> i'm happy enough with extensions given the existing ecosystem |
| 271 | 21:51:09 <cscott> hopefully we'll eventually have more robust article metadata editors, so you can just directly edit the content model |
| 272 | 21:51:12 <yurik> remember that we are targeting a very tech savvy community with this until a nice editor system is in place. And when it is, I wouldn't mind a VE to edit the data remotely, without even switching to commons (like we do in Wikidata) |
| 273 | 21:51:28 <brion> mmmm, spreadsheet editor |
| 274 | 21:51:30 <cscott> brion: and i'm lobbying against them based on where i'd like to see the ecosystem eventually go. ;) |
| 275 | 21:51:31 <robla> cscott: file extensions and file types are tied up with one another, despite years of standards bodies trying to make that not be true |
| 276 | 21:51:33 <yurik> brion, exactly |
| 277 | 21:51:37 <DanielK_WMDE> i notice we are getting close to the end of the meeting. |
| 278 | 21:51:46 <DanielK_WMDE> are there any thoughts or comemnts about geojson? |
| 279 | 21:51:53 <yurik> brion, T134618 |
| 280 | 21:51:53 <stashbot> T134618: Implement spreadsheet-like cell editing for tabular data - https://phabricator.wikimedia.org/T134618 |
| 281 | 21:51:54 <DanielK_WMDE> yurik: how do youo render geo shapes? |
| 282 | 21:51:56 <cscott> "tech savvy community" == we systematically exclude potential community members who are not tech savvy |
| 283 | 21:51:59 <cscott> that's what i hear, at least |
| 284 | 21:52:04 <brion> and if you really want to have fun with file extensions <-> type, try dealing with video containers vs codecs! </runs away> |
| 285 | 21:52:19 <yurik> DanielK_WMDE, easy - you just put that geojson inside the <mapframe>...</> wikitext element :) |
| 286 | 21:52:34 <robla> brion: amen |
| 287 | 21:53:06 <brion> cscott: a legit concern, yes |
| 288 | 21:53:15 <DanielK_WMDE> yurik: so there is a hard dependency on the maps extension? |
| 289 | 21:53:16 <yurik> cscott, i am by no mean trying to exclude them, but rather understand the users. Non-tech savvy community is the ones that will provide the most value (simply because there is probably a bigger nontechsavy community there), but we should make it nicer and easier for them. |
| 290 | 21:53:17 <brion> usability will become a bigger concern once there are tools built up on top of this system |
| 291 | 21:53:26 <yurik> DanielK_WMDE, when supporting geojson as storage - yes |
| 292 | 21:53:29 <cscott> so long as the file extensions aren't baked hard into the design, i'm happy. |
| 293 | 21:53:32 <brion> eg if you already have graphing/table-formatting templates+lua modules ready to use |
| 294 | 21:53:34 <DanielK_WMDE> yurik: what do you do if it's not there? just show json as text? |
| 295 | 21:53:36 <brion> and a good editor |
| 296 | 21:53:41 <DanielK_WMDE> would be ok-ish, i guess |
| 297 | 21:53:42 <cscott> just like i'm happy so long as we can *eventually* enable shadow namespaces or instantcommons on this |
| 298 | 21:53:44 <yurik> DanielK_WMDE, we could - as a backup |
| 299 | 21:54:03 <yurik> cscott, i am having very big doubts about shadow namespaces to be honest |
| 300 | 21:54:05 <DanielK_WMDE> cscott: +1 |
| 301 | 21:54:06 <robla> I think the file extension issue needs to go to wikitech-l |
| 302 | 21:54:09 <yurik> but that's a separe discussion :) |
| 303 | 21:54:22 <cscott> yurik: instantcommons then. or data: namespaces on every wiki. what you will. |
| 304 | 21:54:55 <yurik> cscott, i'm not against it, just doubting the long term viability of it ;) |
| 305 | 21:55:10 <cscott> i have faith in kartik ;) |
| 306 | 21:55:11 <yurik> but again, we can totally support it if we decide that's the way forward |
| 307 | 21:55:18 <DanielK_WMDE> can we confirm that geojson is good to go? i have no objection, but i also know next to nothing about it |
| 308 | 21:55:32 <robla> so yurik, thanks for bringing this conversation up on wikitech-l generally. I think there's a lot more to discuss here...and I'm not sure how to do it |
| 309 | 21:55:33 <DanielK_WMDE> is anyone around who aqctually knows something about geojson? |
| 310 | 21:55:37 <cscott> sure. that's all i'm lobbying for at the moment. leave space for the future, don't do anything that would make it impossible later. |
| 311 | 21:55:39 <yurik> DanielK_WMDE, https://www.mediawiki.org/wiki/Help:Extension:Kartographer |
| 312 | 21:55:40 <brion> so the alternative on the extension is probably "don't enforce an extension, have everything in the Data: namespace be this tabular format _for now_" |
| 313 | 21:55:42 <MaxSem> DanielK_WMDE, /me |
| 314 | 21:55:43 <yurik> it has a geojson sample |
| 315 | 21:55:43 <robla> DanielK_WMDE: I don't want to confirm anything in this meeting |
| 316 | 21:55:51 <brion> with an eventual UI/API extension for picking different content model |
| 317 | 21:56:14 <SMalyshev> DanielK_WMDE: well, I know a little about it... nothing that would prevent us from having it on wiki as format :) |
| 318 | 21:56:17 <DanielK_WMDE> robla: ok, check that there are no objections at this time ;) |
| 319 | 21:56:28 <yurik> brion, i'm not too happy about that - i would much rather say "for now, lets only allow pages in the Data: that match the extension" |
| 320 | 21:56:39 <yurik> this way we can put geojson there as well |
| 321 | 21:56:43 <yurik> and other formats |
| 322 | 21:56:49 <brion> is geojson ready to go? |
| 323 | 21:56:54 <yurik> brion, yep |
| 324 | 21:56:57 <brion> ah fun |
| 325 | 21:56:57 <yurik> it is much easier |
| 326 | 21:57:02 <DanielK_WMDE> robla: as in humming ;) |
| 327 | 21:57:06 * robla doesn't feel like he understands what's being proposed to have had a chance to object |
| 328 | 21:57:08 * aude and soem other people implemented geojson content handler in zurihc |
| 329 | 21:57:14 <aude> 2 years ago? |
| 330 | 21:57:14 <yurik> geojson is very straight forward - we already have it as part of kartograhper ext |
| 331 | 21:57:27 <cscott> i'd say "the content model of the page is defined at page creation type by the extension. but nothing after that point tries to parse the article title for an extension" |
| 332 | 21:57:37 <yurik> cscott, agree |
| 333 | 21:57:43 <aude> not sure exactly how it would work now, but think it's not too complex |
| 334 | 21:57:53 <DanielK_WMDE> cscott: yes, absolutely. |
| 335 | 21:57:58 <brion> cscott: that seems sensible yeah |
| 336 | 21:58:04 <cscott> it also potentially means you could work around the need for an extension by sneaky renames. ;) |
| 337 | 21:58:10 <brion> and allows for the future to drop the extension at creation time |
| 338 | 21:58:12 <yurik> cscott, the only limitation - jsonconfig will not allow renaming if the target page name does not match the original regex |
| 339 | 21:58:13 <DanielK_WMDE> that's how it works for .js and friends |
| 340 | 21:58:25 <cscott> in lieu of having a proper direct edit mechanism for the content model |
| 341 | 21:58:31 <yurik> yep |
| 342 | 21:58:55 <cscott> yurik: yeah, i'm okay with the rename limitation for now. i just don't want to code to have regexp matches against the page title scattered everywhere. |
| 343 | 21:58:59 <robla> we're running out of time. very good discussion; I think I know how to pull open questions out, but I'm not volunteering to do it. |
| 344 | 21:59:13 <aude> not sure we need hard dependency on kartographer |
| 345 | 21:59:15 <yurik> cscott, oh, thats not there . The content id is stored with the page |
| 346 | 21:59:18 <brion> for the .js/.css subpages we also just have predictable naming which is what the exts are for, something not relevant for the primary item |
| 347 | 21:59:20 <DanielK_WMDE> yurik: people got upset when they couldn't rename a misnamed foo.jd to foo.js, because the content model mismatched ;) so now they can re-decalre the content. a bit scary... |
| 348 | 21:59:22 <yurik> aude, its a soft dep |
| 349 | 21:59:36 <aude> yurik: or some generic stuff could be seaprated and used by both things |
| 350 | 21:59:38 <yurik> just like many extensions depend on syntax highlighter |
| 351 | 21:59:38 <robla> yurik: where should people who are interested continue this discussion? |
| 352 | 22:00:47 <yurik> robla, i guess I should create a new task "deploy" ? |
| 353 | 22:00:52 * robla plans to type "#endmeeting" by 22:05 UTC |
| 354 | 22:00:54 <yurik> as we discussed earlier ? |
| 355 | 22:01:05 <cscott> (brion: .js/.css subpages are a bit weird since browsers and webservers still do content-type sniffing based on url extension and other factors; that shouldn't be relevant to the data namespace which is for internal mediawiki use, not for directly serving to web browsers) |
| 356 | 22:01:11 <robla> yurik: could you file a quick placeholder task? |
| 357 | 22:01:19 <yurik> sec |
| 358 | 22:01:32 * robla wishs Phab allowed reassigning the submitter |
| 359 | 22:01:51 <brion> cscott: when we serve them as JS/CSS content it's through RL's load.php; their URLs don't end in .js or .css at all :) |
| 360 | 22:02:52 <yurik> https://phabricator.wikimedia.org/T137929 |
| 361 | 22:03:00 <robla> thanks! |
| 362 | 22:03:22 <robla> #info conversation will continue at https://phabricator.wikimedia.org/T137929 |
| 363 | 22:03:24 <yurik> https://phabricator.wikimedia.org/T137930 |
| 364 | 22:03:31 <yurik> robla, ^ geojson |
| 365 | 22:03:37 <yurik> should there be one common one? |
| 366 | 22:03:51 <yurik> that discusses the underlying tech? like extensions, etc |
| 367 | 22:03:56 <robla> #link https://phabricator.wikimedia.org/T137930 geojson |
| 368 | 22:04:08 <aude> thanks yurik :) |
| 369 | 22:04:22 <aude> suppose maybe we can also talk at SOTM US :) |
| 370 | 22:04:36 <yurik> ok, if needed, i will create another task later |
| 371 | 22:04:41 <robla> let's treat T137929 as the parent task |
| 372 | 22:04:41 <stashbot> T137929: Enable shared tabular data storage on a shared wiki - https://phabricator.wikimedia.org/T137929 |
| 373 | 22:04:54 <robla> ok....let's end the meeting |
| 374 | 22:04:57 <yurik> :) |
| 375 | 22:05:04 <robla> thanks all! |
| 376 | 22:05:08 <yurik> thanks robla! |
| 377 | 22:05:08 <robla> #endmeeting |