@sbassett Question for the future, is htmlentities() for all input-parameters the right way to go?

I turn off the script. So I think the issue can be closed.

Commons-on-OSM is only the frontend that I maintain, so I can't solve the problems of the backend of project geocommons ( https://tools.wmflabs.org/geocommons/kml?bbox=-0.3,-0.3,0.3,0.3 )
@sladen is perhaps the only active maintainer of project geocommons.

For me it seems so that kmlexport needs new maintainer. I wrote @Dvorapa a mail but he makes a break until July, if his user-page is correct. With osm4wiki it looks better but it doesn't work without kmlexport.

As far as I know, the osm4wiki use as backend the project kmlexport, which is not running after server move to stretch . So deactivating is in my eyes no problem.
For clarification I'm not a developer of osm4wiki, aim of co-maintenance was to be able to restating a server if necessary. I have no access to project kmlexport and both maintainer have a break or are inactive. It's not so nice as osm4wiki is linked inside different articles in German Wikipedia.

@GTirloni @jcrespo The issue still exist but now it's necessary to import the data as the database was deleted by T216441.
I try to import Ukrainian Wiki a file with 4.3 GB and lost during the import the connection to the database:

@jcrespo Thanks for reinstalling templatetiger info table. But now I ran as predicted in the problem of https://phabricator.wikimedia.org/T184126 .Let's work further under this issue report.

Main problem with repair is that the table "info" is away and this was created by hand and I have no idea about it's structure as it was many years ago. I have the hope that the other tables are generated by script.

A repair on s51071__templatetiger_p would be nice. The easiest way to do this, would be allow me to import me new data from dumps but this is blocked by https://phabricator.wikimedia.org/T184126.
Dump files are here: /data/project/templatetiger/public_html/dumps/

@GTirloni : Thanks for the cleanup script.

Clean up done.

Fine it's working. Thanks.

It works now fine high zoom levels. Thanks.

Sorry I was longer time AFK.

@Bstorm: I think "copy" should import the unpacked file:

copy wp_coords_red0 from '/home/wp-world/new_red0';

Than I can start some some other sql-commands to create the other tables *red2,*red3 and *red4

Luckily I have dumps (primarily as service for external users):
Dumps:
https://tools.wmflabs.org/wp-world/dumps/new_red0.gz
Database_Structure_postGIS:
https://de.wikipedia.org/wiki/Wikipedia_Diskussion:WikiProjekt_Georeferenzierung/Hauptseite/Wikipedia-World/news#Database_Structure_postGIS
Further personal SQL-Commands:
https://tools.wmflabs.org/wp-world/load-dispenser2015-2.php

@Bstorm : Yes the ticket is still a problem. You can self see that under database "gis" that my tables "wp_coords.." are still missing. This is also the error report of my script[1].
If the data was moved to an other database like u_kolossos a hint would be nice and I would need help how to access the other database (see above).

On parts of my tool I'm still have a strong interest to maintain it.
But I deactivated the googlemaps proxy as I'm no expert on XSS security https://phabricator.wikimedia.org/T182341 .
The source code of the script is:
https://tools.wmflabs.org/wp-world/googlmaps-proxy.txt
So if somebody would be able to repair it it would be nice.

After starting it with jsub (without autocommit=0) it lose the connection to the database during the LOAD DATA LOCAL.

The line by line execution shows that the LOAD DATA LOCAL line was the point where it breaks.

The import of some GB need some time. This was no problem in the past because I use "/* SLOW_OK */" in the sql-statement[1].

Hello, it's cleaned up.

I have two points, first the relation between Wikimedia Movement and OSM and second my opinion what we should do with gpsies and apemap as the largest external users.

Changed to wmflabs tiles.

It's really seems related to T180380, but there is no solution.

Done.

Cleaned up to 95 GB.

@yuvipanda Yes it works fine.

@yuvipanda Thank you. Fine.

I update the db to 2017. Everything works fine.
Now the database seems to be deleted/disappear. Anyone any idea?

I would say the community should decides this question in each language. For me as a German it's e.g. ok to get English labels as fallback, for someone in eastern European projects it's perhaps better to get Russian as fallback.
It's in my eyes impossible to configure this from a central point.
Offcourse each user has different background which languages he can read. Each user have different usecase, e.g. sometimes he want to compare the map with real street signs on the street, so he need the pure name-tags. Sometimes he really need the labels in his own languages.

I take I look in my old php-code of templatetiger and must say that it's horrible. There are wild string-construction to generate sql-queries. It's simply not to modify anymore.
I think it could need a complete rewrite in something like Python/Pandas and a complete new UI.

Please, don't use Property P402 !!!
We show it on WIWOSM [1] project that we don't need it to link from Kartographer to an geometry in OSM.
We can use Wikipedia and Wikidata-Tags in OSM and are so more flexible to suport also points and lines without and relation-id. The relation ID's in OSM are not stable.
so please change Kartograher to support OSM data without P402.

@yuvipanda : Do you know why templatetiger doesn't work and why I can't bring it back by restarting webserver?
I get an "502 Bad Gateway" . Thanks

I reduced the volume to the minimum of 150 GB by deleting old files manually.
With the often broken dump files I have no better strategy.

The osm-style have to be reduced to have place and clearance for additional overlays.

templatetiger should be after cleanup now at under 140GB on file system.

templatetiger should be after cleanup now at under 140GB.

I can recreate all data inside templatetiger, in the case of a disaster.

@scfc:
To the first point: Yes to maintain a labs project would be to much overhead for me. Normally templatetiger is only a little php script and a database. The trouble with to sort script cames later.

I will try now -T /tmp/sort. So the temporal dir should be local tmp .
I code this stuff, often after my regular job. So sometimes I'm fine if things seems to work, also if it's not optimal. Sorry if such things happen.

Is it true that "/tmp/" is locally on a server randomly choosen by jstart? So how can I use this files later in the next job to import it to database?

Sorry, I read this too late.
I will sort now to "/tmp/sort/" and hope thats ok.

This task is also know as multilingual maps:
http://wiki.openstreetmap.org/wiki/Multilingual_maps_Wikipedia_project
Should work in the future with vector tiles.

Thanks, it seems to work.

Ping? The problem still exist.

No I change only the scripts we have in production inside Wikipedia and Commons.
This doesn't also help maps that base on hikebikemap-tiles or blackwhite-map.

I switch the standard tiles to osm.org:
https://tools.wmflabs.org/wiwosm/osm-on-ol/kml-on-ol.php?lang=de&uselang=de&title=Pirna&zoom=11&lat=50.96033&lon=13.94839

Hello,
I (Kolossos) and Plenz are only developers for the frontend. We have no access about the backend, also if I asked Para to get access to it.
I believe he code in Python, to query the database about Categories and Subcategories.
So I believe the subcategories are perhaps the performance killer and someone should replace these part of code by using CatGraph or something eise.
Sorry that I'm also very busy in the moment. So perhaps somebody else can help. Perhaps @dschwen has an idea.

Solved.

The idea was to merge Wikiminiatlas and OSM-Gadget[1] to a commons tool that is powered by Leaflet or OpenLayers3.
Now it seems both developers (me and DSchwen) have other things to do. So some support for an update would be nice.
Tiles in WikiminiAtlas have a different projection.

In T57549#1088276, @Lydia_Pintscher wrote:

@Kolossos: You're making a very good point about geojson very likely being too big. Do you have recommendations for other ways to store it?

I think as big solution we should use OpenStreetMap-Software stack (editor, Database, Rendering and APIs) for our own data. The rules we can change than. A lot of objects like "Interstate5" are already in OpenstreetMap and we have with the project WIWOSM over 400.000 links between these objects and Wikidata. The own OSM-Software should be only for objects that are out of scope from the OpenStreetMap (OSM) project, like fuzzy objects e.g. bird habitats. For historical objects that also out of scope from OSM we are in contact with openhistoricalmap project.