I think the initial idea was to use client-side rendering, and mapbox offers a better path for that.

Note that ruwiki has similar settings, and has mapframe enabled, but has not observed any issues (or at least they have not reported them).

In T192695#4201262, @Catrope wrote:

Old revisions are not in the parser cache (AFAIK), so they'd have to be reparsed every time.

Architecture: parser for graph/kartographer generates JSON blobs and their hashes. The hash goes into img URL. Browser fetches image from backend service, which uses the hash to get the original JSON blob from MW API and renders it into an image. There are two levels of caching: image cache (Varnish), and memcached used by the MW API (per hash).

Sure, that's fine. The problem is the cache. If you include revisionID in the URL, your image MUST regenerate on each save (regardless if it depends on time or not). This is an expensive operation, plus users may be annoyed that their graphs/maps don't show up right after saving. If you don't include revID, you cannot regenerate the page, but only get it from the cache when available. BTW, currently json blobs are already stored in memcached, with the longest possible expiration.

similar - you re-run parser for the specific page revision (either new or old), for the whole page, and as part of that process you get a JSON blob. That json blob is what the graphoid/kartotherian services need to render the image. Hash is just a way to uniquely identify that blob. Most of the time, the blob would be identical between revisions, or oven across multiple pages in the same wiki. But in some cases, e.g. when time-related constants are used, the blob will be different with each parsing.

@Tgr the main problem with cache is that sometimes it might not be there when needed, and it would have to be rebuilt. But to rebuild something, you have to somehow provide the context required for the rebuilding. Currently each graph is identified with the hash, making it impossible. The map/graph would have to use a different URL structure, and include all the needed info (e.g. page revision ID and graph index), and graphoid/kartotherian would have to pass all that info to MW API for regeneration. But the moment you include revision ID, you destroy the image cache, because each image would have to be regenerated on each page save - and regenerating these images may take longer that page parsing itself.

@Ghuron use nolabels parameter.

@Ghybu I believe it's already available - https://maps.wikimedia.org/?s=osm#4/40.75/-73.96 (style = "osm")

In T155290#4187962, @Smalyshev wrote:

I think having a separate description slot, where you could put categories, templates, bot instructions, human instructions, etc. would be the best solution. Trying to make JSON into what it's not meant for is far inferior.

In T155290#4187322, @IKhitron wrote:

We do it this way in js and css.

@daniel while i do agree with you in principle, it might be a while to implement. Adding a single field to JSON and passing it through parser is about 5 lines of code, and should take at most an hour of a skilled dev. Also, I wouldn't separate categories from the wiki markup here, simply because most of the time you want templates with categories to auto-add stuff, rather than each page having individual category fields. Also, this method does not preclude future migration to the multi-part system - rather it will be very straightforward.

This might be a bug/limitation of the express engine routing matching?

Awesome, thanks @SBisson for figuring out. At first I thought it was something much more substantial as it coincided with various deployments.

@Abbe98 have you tried creating a single-letter source? I'm ok with closing this issue, as its not really "issue-worthy"

@Urbanecm this is not accurate. changes to existing systems take a long time - people need to agree that the "new" system is better and it is ok to change. That's why introducing a new media viewer was messed up - it required every community to agree on the new way to see images for everyone, and the change wasn't well accepted/delivered.

Storing just the keyvalue hash->spec is not a very good solution, because graphs/map templates/lua may rely on dynamic things like time, which means the data will keep changing with every map rendering, polluting this table without any hope of cleaning it up.

I think @MaxSem was working on this a while back, and might have even had some code. ]

@SBisson I think adding an extra tag to every single OSM object with the name tag is a bit excessive - there are 62 million of them. A much better solution IMO is to make it possible to calculate that language based on the geo position of the object. My suggestion was to create a new "language meta-regions" with the language tags. Another solution is to add language tags to the existing regions. Both have pros/cons, but so far it was only discussed in that mailing thread (link above). If either of these solutions are implemented, it should be possible to generate a geo index for lookups - something that can be done during vtile data generation.

I think you meant name=<local name>, not name:<local name>. "name" should be the last fallback if other options are not available. "int_name" should also be considered (it's the most common tag besides the name itself -- over 400,000). As for telling the language of the "name", I proposed OSM to have regions with primary language code) - currently being discussed. Please participate.

@daniel sorry, let me clarify.

@daniel this is awesome, thanks! The sitelink hack would probably be the best approach for the tag keys (e.g. name, address etc.) I am not yet sure of the best approach for the "enum-like" values -- some tags, e.g. religion tend to have a well-known list of values. It would be good to have a separate namespace for all the possible values for them. Enforcing uniqueness for them does not make sense -- same value could be used for more than one tag, and could have very different meaning. One option to store tag value would be to allow duplicates (use a regular string property to store the actual value), to reference all tag entities, and to use a bot to ensure that tags are referenced only ones per each unique value. Or it could be some on-db-update check...

thx, clearly this wouldn't be used for wikidata. @daniel could you point me to the relevant code plz, or perhaps something similar, or maybe sketch the implementation approach? I was thinking that this would be the only real path to solve this, possibly with a custom database table to prevent accidental duplicates.

Might be related to @Cwek's T98836#4093634

Getting OSM to even discuss standardization is very hard - there are always "territorial" people who say that they prefer things as is for their neck of the woods, even for small things like these (I had a fair share of fights on this one). I do agree that the system should be flexible to cover all use cases, but I also think all communities should work together to harmonize the data in order to increase its value, and to allow data consumers to actually make sense of it.

Not sure if this was discussed before. Many labels in OSM are encoded with an extra param, e.g. Latn (75k), _rm (250k+), _kana, _pinyin, hira, and other. Search for "name:" for the full list. How should these be handled? I think babel used to have some support for it, but it might have been recently removed.

In T192187#4131256, @Catrope wrote:

It does use the content language, not the user language. $this->getLanguage() ends up using $this->parser->getTargetLanguage(), which is the page language (which is generally the content language, except in weird cases like translated pages).

I don't think https://gerrit.wikimedia.org/r/426322 should have been merged. Besides the no tests, it uses user language, not content language. As such, this ends up with weird behavior: if content author created a <mapframe lang=xx>, it will always show the map in xx, but if they didn't specify lang param, it will use User's language, not the content language.

@Pnorman makes sense. @Catrope wrt caching, I was thinking more in terms of caching fully identical tiles in an efficient way (e.g. if tiles is identical in multiple languages, or even if it has identical content such as water to use some optimized caching strategy, but I guess since tiles are tiny, the mgmt overhead won't make it worth while)

@Catrope @Next2u 's propsal doesn't preclude all that work. It could be a one line change in Varnish - simply change domain name into a lang parameter, and let the backend handle it as before. My only only other concern with the proposal is that lang parameter allows anything at all, whereas domain name usually comes from a whitelist. Not sure which is better - merits for both.

@Next2u I actually really like this idea! Makes testing and serving much better, and avoids ugly URL parameters. I just wonder if some sort of caching optimization is needed/possible in this case (cc: @Gehel)

@Pnorman no, the compilation of tm2source yaml -> mapnik xml still has to be done outside of Kartohterian. PRs are welcome :)

i have solved this a bit differently with Sophox's region service (github). I think this approach is much better because it decouples tile generation from the regions service, and because the database is much smaller and can be regenerated and updated much more frequently/rapidly.

Update: more specifically, on English Wikipedia, flagged revs is mostly enabled on biographies of living people (not very map related) - see https://en.wikipedia.org/wiki/Wikipedia:Flagged_revisions -- which means English wiki doesn't have to be prevented from mapframe on the account of flaggedrevs.

@jmatazzoni both lists are correct. I listed just the Wikipedias, whereas @MaxSem listed all other projects too. Note that mapframe is already enabled in all those other projects, and, to my knowledge, noone had any issues. Also note that the <graph> tag uses identical system of data storage to maps, and it has been enabled on all wikis for many years, and noone has raised any major issues either. So while flagged rev issue is real, in reality it seems to affect almost noone. For example, flaggedrev is enabled on enwiki, but there are very limited number of actual articles that enable it.

Crosslinking with T156191 and T127683 - might be related, or addressed in the same way?

@Cwek thx for the heads up on this! I wonder if this also addressed T156191 and T127683

I also think this is not needed for the test endpoint.

@jmatazzoni as far as I know, most consumers of OSM, e.g. Mapbox and Klokane, use Wikidata for international labels. Simply because OSM's international coverage is nowhere as good as what Wikidata has. I have built Sophox query service (based on Wikidata's query service) - its a database that combines both the Wikidata and OSM data into one place, thus allowing to easily merge the two results.

Seem like lowering the upload batch size fixed most of the memory issues. The system still crashes on occasion, but works okayish otherwise. Thx!

@Rical take a look at Template:Graph:Lines -- that template's localized parameter documentation is stored in one place at the single location, and can be changed there without going to each wiki and copy/pasting things inside the complex template. Beyond that, the caption under the graph is also localized -- the string "See or edit raw graph data." is localized using the {{#invoke:TNT|msg|I18n/Template:Graphs.tab|source_table|{{#invoke:TNT|link|{{{table}}}}}}} (a bit cryptic, I know - it uses TNT twice, once to render the link, and once to render the whole message). The localized messages are actually stored here.

@Rical take a look at the https://www.mediawiki.org/wiki/Module:TNT module -- it already allows one global translation table that can be used in any template anywhere. This allows a template or a module to be copy/pasted to any wiki without a single modification. Only requirement - the TNT module must be present on the wiki (many wikis already have it).

In T121470#4069480, @MZMcBride wrote:

In T121470#4069451, @Yurik wrote:

Also, I don't think shadow namespaces are a good approach, as discussed some time ago at a conference.

It depends on the use-case. Some of the past discussions focused on the difference between use-cases like Help pages v. User pages v. Template and Module pages. They may require different solutions, even though they can feel like similar problems.

@MZMcBride I think this task is the implementation of one aspect of T66475 . Also, I don't think shadow namespaces are a good approach, as discussed some time ago at a conference. Shadow implies local (per-language) override, whereas most of the time you don't want to override per language, you want to override per usecase, and that override may be useful for more than one language. To give an example, lets say you have an infobox that most wikis find useful, but it breaks in RTL wikis. You have two options - either fix the infobox so that it works on both (preferable, but might be very hard), or you could create a new infobox for RTL wikis, in which case it is useful to all RTL wikis, not just a single one. So instead of having a global + several local overrides, there should be two globals.

I disagree - I find sourcing data extremely useful, but either of these are hearsay. I agree that we should first and foremost honor the licensing requirement - so the easiest solution is to actually show copyright info first, followed by wikidata links. This will solve the immediate need, and we can work on a proper solution without removing otherwise perfectly useful feature.

I agree, it would be great to have a separate "source data" UI separate from copyrights. While this UI is not there, I think we should simply set a maximum of how many WD items are shown (e.g. at most 3, plus add an ellipses if needed) - this will quickly address the problem without sacrificing the useful feature others rely on.

I merged it - as it seemed a valid patch. @MaxSem could you think of a better alternative to dynamically patch upstream lib? We could also submit it upstream.

Shouldn't all small wikis that do not use flagged revisions extension just get this? Otherwise this seems like a busy work - there are 300 wikis, each "enablement" takes probably around 5 hours of combined human time, making this into a two months of wasted human time... that doesn't really gain anything. WMF has never done anything remotely similar to any of its content features that have been so widely requested. Most of these 2 months of time is being paid by the donations too...

Not exactly - this.lang always depends on which "source" you are talking about. The this.lang in babel instance is different from the this.lang in tilelive-vector instance. The pipeline is initiated during the startup - the global storage stores one instance per "source ID", and if one source gets data from another source, that chaining is set up initially. E.g. if view is using generator, there will be one instance of view, which internally references one instance of generator. When user asks for a tile, Kartotherian gets the named instance (e.g. "view" or "osm-intl"), and calls getTile (now its getAsync), passing all the params. Kartotherian would not be able to set any params on the "generator" source because at that point it doesn't know that generator source will be used by the view source.

Interesting idea! The problem is that this (IIRC) is not per request, it's per tilelive instance. Each instance gets its state from the config file, so you have as many instances as sources. Per-request state is only passed via parameters, and that's where the limitation comes in - original tilelive API params are only x,y,z, not opts. That's why I had to introduce a new getAsync method to pass additional state.

@SBisson the custom version of vector should handle that, but I don't remember if that was fully implemented (most of it was for sure). See https://github.com/kartotherian/tilelive-vector ...

@MaxSem I think static map should still be site-language based, which would prevent cache bloat, while still be useful to the readers (map in Japanese wiki will show in Japanese)

@Mooeypoo are you sure about that? I was under the impression that the parser has user (or at least site) language, which means it could generate a proper language-dependent url for the static image.

That seems correct. Try looking at the tile at every state, see if labels are there initially, and if so, when does it disappears

@SBisson I wrote some docs in https://github.com/kartotherian/babel#usage-examples -- I think prod source configs should follow that pattern. The tilerator config should use the json2tags as part of the tile generation pipeline - converting one giant json string into proper pbf tags. The babel should be used in kartotherian config, to convert the tile on the fly for the user. I would recommend using a language map option with it - to create a proper language fallback map.

@Gryllida you are right. There is an extensive Vega doc site, which also includes a number of tutorials, but unfortunately it documents Vega 3, whereas Wikipedia is still using Vega 2. The concepts are very similar, but there are some syntax changes between versions. Also, there is a much simpler language called Vega-Lite, used for the most typical X/Y graphs, and eventually I hope WP will support both.

@kaldari I agree that when we have this many data points, they become bad. This was initially designed when you have just a few wikidata objects, e.g. the outline of a single country or the shape of some famous street. This allows users to quickly navigate to the Wikidata item in question, similar to how French Wikipedia has "edit" link next to each item in the infobox that gets generated from Wikidata.

I think the links to Wikidata should remain, but need to be restyled. The reason these links were placed there is to allow users to view the source of the data, not to satisfy a legal requirement. One option would be to restrict the links to just the first few - this will help when the map has only a few items, but won't clutter it.

I replied to https://www.mediawiki.org/w/index.php?title=Topic:U8qudxbgf2xujfsw&topic_showPostId=u8rrlm6pbjgyg7gz&fromnotif=1#flow-post-u8rrlm6pbjgyg7gz -- take a look at this example -- you need to set "contentmodel": "Tabular.JsonConfig"

@Smalyshev the reason I made type as a string is to allow additional parsing parameters, e.g. ?start tabular:startDate 'date:yyyy-mm-dd'

Kartotherian's babel does exactly this, but it needs to be configured with the right fallback rules.

Just FYI, Kartotherian babel already supports the fallback. The big question is actually decide what to fall back to for each language. Currently it simply accepts a dictionary of language -> [ fallback chain ]

snapshot service might not allow z19, but this is probably in a config somewhere.

There could be a different, more powerful alternative to improving [[ https://github.com/mapbox/simplestyle-spec | simplestyle support (e.g. maki icons, etc), especially as it appears to have been mostly abandoned by Mapbox (at least I haven't heard of any changes with it)

snapshot service needs to accept zoom 19 too.

I think this task can be closed because it can be done with https://en.wikipedia.org/wiki/Template:OSM_Location_map

@Pnorman my apologies, added.

@Pnorman, I have added you to the kartotherian npmjs project a week ago. Were you not able to publish?

The first version of this feature has been implemented in Sophox -- see docs. At this point, it supports any GET request that returns CSV-style data (parsable by Java's CSVParser, with many parameters).

@Lucas_Werkmeister_WMDE I agree - I am planning to implement this feature for both WDQS and Sophox QS. For WDQS, it should only support tabular datasets, or possibly other respected sources.

In T179991#3754444, @Base wrote:

Label service is also very slow, like 2 times slower than to just query labels a normal way, considering that map data processing is probably more complex than just fetching a label I am afraid that it won't work for any real queries with current timeout…

In T180314#3754292, @Lucas_Werkmeister_WMDE wrote:

Duplicate of T175578?