done :)

Are there any updates/progress on this issue? The OpenStreetMap Wikibase has both the local images and it supports Commons, which means every Wikibase "image" property is actually two properties - one of "image" type, and another being a manually copy/pasted string of the local (OSM) wiki file - in case the file is not on Commons. It would greatly simplify things for OSM community if an image property would be a single "media", regardless of where it is actually stored. What could be done to solve this?

I agree with @Legoktm -- storing data blobs in page props was a hack. But to my knowledge, there is no good alternative storage for the parser-generated blobs. Essentially any system that needs independent access to those blobs would require something like this, essentially solving T119043

See related approach by Module:TNT -- it stores templatedata as a table (example). In this case <templatedata> will be dynamically generated during the parse time, and is available to every language/every wiki.

@akosiaris also, please add usage before the Varnish - to see how often graphoid objects are actually requested by the user, rather than how often there is a cache miss. Plus a similar stats for getting map snapshots.

However, that begs a question. If the caller of the graphoid service API already knows the hash of the graph, that means they either got it via talking to mediawiki, or that they calculated it themselves (aka they already have the entire graph). If they already have the entire graph, why not POST it to the service and obtain the resulting PNG themselves? Things become even more convoluted if the user is mediawiki itself (that's to my knowledge actually the case) at which case having mediawiki instruct another service to create requests back to it (the API endpoint actually) causes a cascading number of requests hitting the mediawiki API. Up to now this hasn't caused an outage, in my opinion simply because of the low traffic the graphoid receives.

In T184933#4813382, @hoo wrote:

In T184933#4812900, @TheDJ wrote:

Fancy...
Just a followup/enhancement idea: since we have precision.. wouldn't this be a good place to draw a precision circle ?

This is a very good point: Basically we can alter the generated mapframes like in any article… in fact we generate them from wikitext (see CachingKartographerEmbeddingHandler::getWikiText).

I created a new task (T211676) for this… suggestions on how exactly to implement this are very welcome there :)

I suspect the same issue is in Graphoid - it pulls the same bin data blobs via api

After some extensive work with Docker containers, it seems they are much cleaner, more portable, and much easier to deploy approach, without any performance or other issues. I would highly recommend repackaging under Docker.

Upstream issue

@Lucas_Werkmeister_WMDE I think both should be fixed - it should never use unneeded parenthesis for single values, and when the VALUES list is short enough, it should be shown singleline. For a very long list, it should be shown like this:

Turns out the --conceptUri http://wiki.openstreetmap.org should have been http instead of https. Thanks @Smalyshev for your help!

I added bot approach to the wish list -- https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2019/Miscellaneous/Shared_Multilingual_Templates_and_Modules_available_to_all_wikis
Please comment and help improve. Thanks!

IIRC, OSM can store polar regions just fine, but it is the OSM's own rendering that causes problems. Wikipedia does not use OSM's renderer, but it is similar in structure.

@Anomie I don't think that task will have anyone working on it any time soon - that's why I proposed a bot-based solution. The good thing about that solution is that:

• it doesn't need to wait for MW to change
• it is multilingual
• once MW implements something, all shared templates and modules can be switched to the new system without much effort (essentially just disable the bot)

@Capankajsmilyo AFAIK, WMF is not working on this. When I have some time, I will try to set up a bot to make this possible with the existing technology. A typical workflow:

• A template or a module is created on mediawiki.org (MW is better because its community is more dev-focused, whereas Commons tends to be content-focused)
  • all localization strings are placed in the tabular data on Commons to simplify translation
  • template parameters are also placed in a tabular data on Commons
  • All strings are used via the Module:TNT
• Some well known infobox is placed at the top of the template/module documentation to indicate that this module is shared between multiple wikis, and should not be changed anywhere else.
• A bot looks for all modules/templates on MW.org that have that infobox, and copies them automatically to all other wikis using the sitelink list in Wikidata
• If someone wants to use that template/module in a wiki X, they just need to copy/paste it to the wiki X, and add the sitelink - the bot will automatically keep it up to date from there on.
• If a wiki decides to "fork" their version, they can simply remove the infobox from the docs.

P.S. I am not sure Commons is the good place for the shared modules - simply because Commons community tends to be much less technical than the mediawiki.org community. Maybe the main source of templates and modules should be there instead, with the proper things like unit testing.

I agree this is needed, but perhaps we can already set this up without waiting for the complicated change in the system. There is one big reason modules and templates differ - language. So if we move translations out of the templates and modules, we can simply copy/paste them without any changes between wikis. Moreover, I think we can even automate that - e.g. any template or module that has some sort of a flag (e.g. embedded well known template) will be automatically copied from the central wiki to all other ones that want to use it. I wrote a Module:TNT that allows for the translations to be stored on Commons using shared data tables. This way, when you update a message on commons, all existing templates will automatically start using the new translation.

Seems to have been fixed in the past? Thx @MSantos !

@Krinkle the (1) was to give instant feedback to the user -- that's the part that I don't think is feasible in all cases -- because template could grow in size, and blob formation happens post-save. In some cases it might be possible when we are doing page preview to show a warning in case templates already exceed blob size. I do agree the extension should handle it gracefully, and remove some of the data to keep the JSON structure.

Adding a new table shouldn't be too hard, just requires someone to work on it :) @MaxSem actually had some prototype of this a few years back :)
Changing data would help because that data is essentially a key-value dictionary of objects. Each object is independent, and represents a separate graph. If there is a giant single graph on the page, this won't help. But if there are multiple large graphs, deleting some of them would make it possible for at least some of them to show up, rather than the whole page to fail.

Turns out this does break the 3rd party sites, e.g. wiki.openstreetmap.org. I think we should revert and introduce a separate path for the rest API.

More than 64KB is too large? That sounds ... strange when discussing modern systems that are perfectly capable of storing much bigger data rows. Especially when we clearly have a valid use case for it.

The only graceful thing we can do is something like this:

while (True) {
  gzData = gzip(rawData);
  if (length(gzData) < XXX) break;
  unset(rawData[length(rawData)-1]);  // delete last element
}

Could be optimized further.

Ahem... you did not offer any better alternatives :-) The proper solution is to simply change blob to large storage. That said, I don't think page props should be used for it. Instead, it should be a general "large blob storage" key-value store, where different extensions could store hash-based data. This way multiple page revisions could reuse the same blob, and at the same time updates to templates would create new blobs. This would also require usage counting (tricky). There is another big task about this.

(1) is not feasible -- if I build a page with the geojson or a complex graph, and some article adds multiple templates (e.g. an article with 10 graphs), the article blob may not fit all the data, causing this issue. In many cases, there is no way to even give feedback to the user, e.g. if the template gets updated after the article is created.

Agree.

Thx @Aklapper , I think WIWOSM wouldn't be part of it, because, just like Kartotherian and other techs, WIWOSM is using OSM tech/data, but does not affect the OSM project itself. On the other hand, some bug in MediaWiki could affect OSM community because OSM wiki uses MediaWiki engine. Essentially this tag would indicate the affected audience. BTW, the multilingual map efforts would probably fall under this tag because Wikipedia drives contributors back to OSM to contribute the name translations.

@Milimetric v3 and v4+ are not very different. Just some minor integration API changes. 4 is 99% backward compatible. So yes, we could just skip to 4.3

Another possible use case: multiple sitelinks to the same site. For example, Wikidata may have a project page for postal codes in English, and someone wants a similar page in German. But the community does not want direct translation, because English page may want to cover the whole world, while the corresponding German page may only want to concentrate on the aspects specific to Germany.

See my previous comment, I don't think this task should be done at all.

@Smalyshev correct. Ideally we should have T190454, but unless we want to stop using wikibase until it is ready, this is a good interim solution.

See also community discussion thread (update link fixed)

@Jan_Dittrich I think context of use is correct for Wikidata, but not for some other Wikibase installations. Wikibase may not be the centerpiece of the wiki installation.

This issue has raised a lot of concerns at the OpenStreetMap wiki after wikibase installation. Is it possible to disable wikibase search suggestions until this ticket is implemented? Thx!

@Addshore thanks! It does work indeed! I saw that many query modules were marked with Caveat: On a repo wiki, this module only works directly on entity pages, not on pages connected to an entity via a sitelink. This may change in the future., and I couldn't find any way around it. Thanks for the quick help!

Version 4 hasn't changed much from v3, especially in terms of the graphs. So basically it will be v1.5, v2, and v3+, at least from the integration perspective. Copying v1.5 and v2 docs might be a good idea.

@thiemowmde @daniel my last patch only refactors Wikibase extension a bit, without touching the data model. It fixes the last few places I found that used direct Item ID parsing/composition, instead using the same common interface used by other code. Are there any concerns with that? For example, Lua function getSiteLink() used TermIdconstructor, whereas all other code in that same file used EntityParser instance.

I think I figured out most of the code needed for item ID parsing/creation without calling new ItemID() directly. Submitted as a new patch ^, let me know if this approach

Spoke too soon: overriding entityTypeDefinitionsArray in a hook works well for creating/editing Wikibase items, but it does not work in some cases like sitelink lookups (used by wiki pages to find corresponding wikibase items).

Update: I have implemented a custom extension that overrides default item https://github.com/nyurik/OsmWikibase using a hook :

I think lex data dumps should be available independently of the other Wikidata data. For example, https://sklonenie-slov.ru/ shows all Russian noun declensions (30,000+), and I think such sites can greatly benefit from the community work.
P.S. I have began a discussion with the site authors, trying to get them to donate their database to Wikidata.

@Bugreporter and @thiemowmde , thank your for the thorough responses!

Introduced by https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JsonConfig/+/450397/ - looking...

@eranroz thanks, reviewed the code, looks awesome, but need a few minor tweaks. The proper license codes are at https://spdx.org/licenses/ .

xml is only needed to configure Mapnik - so if it is possible to use some standard component to convert yml->xml for it, it would work. I think the issue here is that Mapnik is doing some non-standard xml parsing - hence all the problems. @Pnorman would know this better.