Hello taavi,
sorry for that. I clean up large files. You can delete this project completely.

@Nux the blackout was with high probability on my side. I think a repo on github would be very good, would be nice if you could do it.
(BTW: My Desktop is Linux, but i still know WinSCP from the good old times.)

Yes I did this. Some of the files have only write access for user kolossos but not for group tools.wp-world.

The files in project wp-world are in the command line read only for me. I contacted @Nux yesterday how to solve this issue.
Than the change of one line code should be no problem.

In T323159#8588380, @dschwen wrote:

@Kolossos how old are these POI tables? ....

From my side I don't need the u_kolossos db
@aude can maybe check if she needs here db.
I have no idea who is behind wikimaps-atlas.

Thanks @fnegri . If you could copy also my 4 tables wp_coords_red* from the old gis-database to the new one, I would be absolutely fine with the read-only status of the new database. (This tables contains coordinates of Wikipedia-POIs for wp-world on different reduction levels. )

@dschwen : As End of January comes closer, I want to ask if you had time to import my dumps from above? Thanks

@dschwen: Sounds very good. Thanks in advanced.

I could really need help. For wp-world I would need a PostGIS database for my database of Wikipedia coordinates (4 static tables "wp_coords_redX" with "X"=[0,2,3,4] ) . The project runs in different Wikipedias in over 100.000 articles. It would be such a shame to stop the project after >15 years without replacement. The problem is that I never was an server admin or an database admin, so help would be neccesary.

@dschwen : Could you please tell me where the new database is and if you copy perhaps the 4 tables "wp_coords_redX" ?
If so, a switch should be easy.

@Andrew : For the WIWOSM part I still think a switch to maps.wikimedia service would make sense.

@Nux: I would also say let's switch to maps.wikimedia service. With Kartographer the service should have long time support by WMF. It's better to have this, than an unmaintained project (I only managed the development of WIWOSM but doesn't write the code.) I hope maps.wikimedia service has a rapid update cycle.

• For a link to the map you can look at German Wikipedia. Nowadays we use the OSM-Icon instead of the "map"-link. Examples :
  • https://de.wikipedia.org/wiki/Berlin
  • https://es.wikipedia.org/wiki/Berl%C3%ADn (mapa)
  • https://pl.wikipedia.org/wiki/Berlin (mapa)
• If the queries to planet_osm_line and planet_osm_point are coming from OSM-updater, it seem to me that they are coming from OSM-to Pgsql Updating process https://osm2pgsql.org/

If I remember correctly the data for WIWOSM are cached in the file system.

@dschwen, WIWOSM also still running but it's afaik not updated.
For this it seems possible to switch to kartographers data like
https://maps.wikimedia.org/geoshape?getgeojson=1&ids=Q158158 but some adoptions seems necessary.

Yes wp-world is still in use and included in tons of Wikipedia-article.
You can find the help page here:
https://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_Georeferenzierung/Anwendungen/OpenStreetMap/en#Open_map_view

@Nux: I believe we did nasty stuff.
Look for the following line:

gg = '{"type":"FeatureCollection", "features":[{"geometry": ' +	  response.responseText + '}]}';

So we ask for the core of the geoJSON and put it in response.responseText than we make a frame with FeatureCollection, etc. around.

@Nux Was my comment helpful?

In the file https://github.com/wikiosm/osm-on-ol/blob/main/kml-on-ol.php in line 189 & line 190.
I believe you can find a way how to handle projections in old Openlayer.
I hope nowadays are the browsers and the computers fast enough for retransformation of coordinates in the browser, maybe also in the past it was never a real problem beside IE 6.0 ...

The large number are from Google -Mercator projection (Projection-Number 900913)
WGS84 is Projection Number 4326. It comes from the history that Openlayers was a clone of Google Maps.
With the used coordinate system the browser doesn't have to re-project and is so a little bit faster.
Explaination in PostGIS: https://postgis.net/workshops/postgis-intro/projection.html

I gave access to projects "wiwosm" and "osm" to Nux and DB111. I think an update from OpenLayers 2.12 to OpenLayers 7 would be a good thing, but I also believe this would be a complete rewrite. Something I can't do anymore.
I also believe that a tool thats in production and linked on 100.000s pages should be in maintainance by WMF/WMDE ... .
(Maybe WMF/WMDE could give financial support for an modern rewrite of the script.)

Please keep s51071__templatetiger_p. Perhaps it can help me to rebuild the tool (maybe in Python/streamlit) to have a running dataset also if don't speak Ukrainian language. Example: https://templatetiger.toolforge.org/tt-table4.php?template=Infobox%20Film&lang=ukwiki_neu

After thinking about my priorities I think I will concentrate on other projects. @Bgwhite a dump file for e.g. dewiki would still be nice for local experiements.

Thanks a lot to all for the help.

Answers to the questions above:

• Yes I restarted the server each time.
• I tested "OPTIONS,GET" at the same time,but today also one by one or alone.
• Thanks for renaming this task.

I try some changes in .lighttpd.conf but without luck. It seems OpenLayers change the request from GET to OPTIONS methode. Perhaps this is a reason .

Thanks @bd808 for your warm words to the project. Yes, run only a database on a Could-VPS server sounds more realistic for me instead of running a full web server. So I believe I make a project request next weekend. I hope I'm able to configure the database, I don't have experience at this topic so I will need some help.

If I would see a chance to import the data, I would say it could be interesting to bring the project back to life.
Perhaps somebody has an idea.

@sbassett Question for the future, is htmlentities() for all input-parameters the right way to go?

I turn off the script. So I think the issue can be closed.

Commons-on-OSM is only the frontend that I maintain, so I can't solve the problems of the backend of project geocommons ( https://tools.wmflabs.org/geocommons/kml?bbox=-0.3,-0.3,0.3,0.3 )
@sladen is perhaps the only active maintainer of project geocommons.

For me it seems so that kmlexport needs new maintainer. I wrote @Dvorapa a mail but he makes a break until July, if his user-page is correct. With osm4wiki it looks better but it doesn't work without kmlexport.

As far as I know, the osm4wiki use as backend the project kmlexport, which is not running after server move to stretch . So deactivating is in my eyes no problem.
For clarification I'm not a developer of osm4wiki, aim of co-maintenance was to be able to restating a server if necessary. I have no access to project kmlexport and both maintainer have a break or are inactive. It's not so nice as osm4wiki is linked inside different articles in German Wikipedia.

@GTirloni @jcrespo The issue still exist but now it's necessary to import the data as the database was deleted by T216441.
I try to import Ukrainian Wiki a file with 4.3 GB and lost during the import the connection to the database:

@jcrespo Thanks for reinstalling templatetiger info table. But now I ran as predicted in the problem of https://phabricator.wikimedia.org/T184126 .Let's work further under this issue report.

Main problem with repair is that the table "info" is away and this was created by hand and I have no idea about it's structure as it was many years ago. I have the hope that the other tables are generated by script.

A repair on s51071__templatetiger_p would be nice. The easiest way to do this, would be allow me to import me new data from dumps but this is blocked by https://phabricator.wikimedia.org/T184126.
Dump files are here: /data/project/templatetiger/public_html/dumps/

@GTirloni : Thanks for the cleanup script.

Clean up done.

Fine it's working. Thanks.

It works now fine high zoom levels. Thanks.

Sorry I was longer time AFK.

@Bstorm: I think "copy" should import the unpacked file:

copy wp_coords_red0 from '/home/wp-world/new_red0';

Than I can start some some other sql-commands to create the other tables *red2,*red3 and *red4

Luckily I have dumps (primarily as service for external users):
Dumps:
https://tools.wmflabs.org/wp-world/dumps/new_red0.gz
Database_Structure_postGIS:
https://de.wikipedia.org/wiki/Wikipedia_Diskussion:WikiProjekt_Georeferenzierung/Hauptseite/Wikipedia-World/news#Database_Structure_postGIS
Further personal SQL-Commands:
https://tools.wmflabs.org/wp-world/load-dispenser2015-2.php

@Bstorm : Yes the ticket is still a problem. You can self see that under database "gis" that my tables "wp_coords.." are still missing. This is also the error report of my script[1].
If the data was moved to an other database like u_kolossos a hint would be nice and I would need help how to access the other database (see above).

On parts of my tool I'm still have a strong interest to maintain it.
But I deactivated the googlemaps proxy as I'm no expert on XSS security https://phabricator.wikimedia.org/T182341 .
The source code of the script is:
https://tools.wmflabs.org/wp-world/googlmaps-proxy.txt
So if somebody would be able to repair it it would be nice.

After starting it with jsub (without autocommit=0) it lose the connection to the database during the LOAD DATA LOCAL.

The line by line execution shows that the LOAD DATA LOCAL line was the point where it breaks.

The import of some GB need some time. This was no problem in the past because I use "/* SLOW_OK */" in the sql-statement[1].

Hello, it's cleaned up.

I have two points, first the relation between Wikimedia Movement and OSM and second my opinion what we should do with gpsies and apemap as the largest external users.

Changed to wmflabs tiles.

It's really seems related to T180380, but there is no solution.

Done.

Cleaned up to 95 GB.

@yuvipanda Yes it works fine.

@yuvipanda Thank you. Fine.

I update the db to 2017. Everything works fine.
Now the database seems to be deleted/disappear. Anyone any idea?

I would say the community should decides this question in each language. For me as a German it's e.g. ok to get English labels as fallback, for someone in eastern European projects it's perhaps better to get Russian as fallback.
It's in my eyes impossible to configure this from a central point.
Offcourse each user has different background which languages he can read. Each user have different usecase, e.g. sometimes he want to compare the map with real street signs on the street, so he need the pure name-tags. Sometimes he really need the labels in his own languages.

I take I look in my old php-code of templatetiger and must say that it's horrible. There are wild string-construction to generate sql-queries. It's simply not to modify anymore.
I think it could need a complete rewrite in something like Python/Pandas and a complete new UI.

Please, don't use Property P402 !!!
We show it on WIWOSM [1] project that we don't need it to link from Kartographer to an geometry in OSM.
We can use Wikipedia and Wikidata-Tags in OSM and are so more flexible to suport also points and lines without and relation-id. The relation ID's in OSM are not stable.
so please change Kartograher to support OSM data without P402.

@yuvipanda : Do you know why templatetiger doesn't work and why I can't bring it back by restarting webserver?
I get an "502 Bad Gateway" . Thanks

I reduced the volume to the minimum of 150 GB by deleting old files manually.
With the often broken dump files I have no better strategy.

The osm-style have to be reduced to have place and clearance for additional overlays.

templatetiger should be after cleanup now at under 140GB on file system.

templatetiger should be after cleanup now at under 140GB.

I can recreate all data inside templatetiger, in the case of a disaster.

@scfc:
To the first point: Yes to maintain a labs project would be to much overhead for me. Normally templatetiger is only a little php script and a database. The trouble with to sort script cames later.

I will try now -T /tmp/sort. So the temporal dir should be local tmp .
I code this stuff, often after my regular job. So sometimes I'm fine if things seems to work, also if it's not optimal. Sorry if such things happen.

Is it true that "/tmp/" is locally on a server randomly choosen by jstart? So how can I use this files later in the next job to import it to database?

Sorry, I read this too late.
I will sort now to "/tmp/sort/" and hope thats ok.

This task is also know as multilingual maps:
http://wiki.openstreetmap.org/wiki/Multilingual_maps_Wikipedia_project
Should work in the future with vector tiles.

Thanks, it seems to work.

Ping? The problem still exist.

No I change only the scripts we have in production inside Wikipedia and Commons.
This doesn't also help maps that base on hikebikemap-tiles or blackwhite-map.

I switch the standard tiles to osm.org:
https://tools.wmflabs.org/wiwosm/osm-on-ol/kml-on-ol.php?lang=de&uselang=de&title=Pirna&zoom=11&lat=50.96033&lon=13.94839

Hello,
I (Kolossos) and Plenz are only developers for the frontend. We have no access about the backend, also if I asked Para to get access to it.
I believe he code in Python, to query the database about Categories and Subcategories.
So I believe the subcategories are perhaps the performance killer and someone should replace these part of code by using CatGraph or something eise.
Sorry that I'm also very busy in the moment. So perhaps somebody else can help. Perhaps @dschwen has an idea.

Solved.

The idea was to merge Wikiminiatlas and OSM-Gadget[1] to a commons tool that is powered by Leaflet or OpenLayers3.
Now it seems both developers (me and DSchwen) have other things to do. So some support for an update would be nice.
Tiles in WikiminiAtlas have a different projection.

In T57549#1088276, @Lydia_Pintscher wrote:

@Kolossos: You're making a very good point about geojson very likely being too big. Do you have recommendations for other ways to store it?

I think as big solution we should use OpenStreetMap-Software stack (editor, Database, Rendering and APIs) for our own data. The rules we can change than. A lot of objects like "Interstate5" are already in OpenstreetMap and we have with the project WIWOSM over 400.000 links between these objects and Wikidata. The own OSM-Software should be only for objects that are out of scope from the OpenStreetMap (OSM) project, like fuzzy objects e.g. bird habitats. For historical objects that also out of scope from OSM we are in contact with openhistoricalmap project.