Load OSM data for maps1004 and test machine after reimage
- Load a small set of the planet for testing
- Update Kartotherian/Tilerator to latest
- Generate tiles and test all services
- Start full planet sync
- Monitor it until everything is complete
MSantos | |
Sep 25 2018, 5:17 PM |
F26306474: osm-initial-import.log | |
Oct 4 2018, 1:50 PM |
Load OSM data for maps1004 and test machine after reimage
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | MSantos | T152196 Kartotherian snapshots always return low DPI images | |||
Resolved | Gehel | T198622 migrate maps servers to stretch with the current style | |||
Resolved | MSantos | T205462 Load OSM data into maps1004 | |||
Resolved | MSantos | T207360 osmosis state file on maps1004 is outdated |
To generate tiles and test the maps1004 instance we need to update kartotherian/tilerator packages so they can work properly with the mapnik 3.0.20.
We are waiting until the next deploy window to deploy the services.
Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:39:12Z] <mholloway-shell@deploy1001> Started deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462)
Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:39:31Z] <mholloway-shell@deploy1001> Finished deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462) (duration: 00m 19s)
Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:59:58Z] <mholloway-shell@deploy1001> Started deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462)
Mentioned in SAL (#wikimedia-operations) [2018-10-01T07:00:15Z] <mholloway-shell@deploy1001> Finished deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462) (duration: 00m 16s)
Change 464439 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Skipping download if PBF file exists
Change 464441 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Skipping download if PBF file exists
Change 464439 abandoned by MSantos:
Skipping download if PBF file exists
Reason:
Duplicate
@Gehel and @Mholloway I am having permission issues with the OSM initial import script. I wrote the patch above to help me debug.
This task is temporarily blocked.
The OSM initial import script finished execution after ≈ 18,20 hours.
I am now looking for a way to confirm that everything is fine before start generating tiles.
The log is attached if you would like to see the outputs.
After performing the osm-brighttm2source steps with the SQLs I got some errors and it need more investigation
$ sudo -u postgres psql -Xqd gis -f sql/admin.sql $ sudo -u postgres psql -Xqd gis -f sql/functions.sql $ sudo -u postgres psql -Xqd gis -f sql/create-indexes.sql psql:sql/create-indexes.sql:250: ERROR: duplicate key value violates unique constraint "pg_class_relname_nsp_index" DETAIL: Key (relname, relnamespace)=(planet_osm_ways_nodes, 2200) already exists. CONTEXT: SQL statement "CREATE INDEX planet_osm_ways_nodes ON planet_osm_ways USING gin (nodes) WITH (fastupdate=off)" PL/pgSQL function inline_code_block line 12 at SQL statement $ sudo -u postgres psql -Xqd gis -f sql/names.sql $ sudo -u postgres psql -d gis -c 'select populate_admin();' ERROR: Geometry type (Point) does not match column type (LineString) CONTEXT: PL/pgSQL function insert_admin_row(admin,geometry) line 3 at assignment SQL statement "SELECT insert_admin_row(theRow, ST_GeometryN(geom, i))" PL/pgSQL function insert_admin_rows(admin,geometry,boolean) line 10 at PERFORM SQL statement "SELECT insert_admin_rows(row, result, TRUE)" PL/pgSQL function populate_admin() line 26 at PERFORM
Change 468257 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: restrict OSM sync check to maps1001
Change 468257 merged by Gehel:
[operations/puppet@production] maps: restrict OSM sync check to maps1001
Change 482648 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Restore settings for maps1004
Change 482648 merged by Gehel:
[operations/puppet@production] Restore settings for maps1004
@Gehel, @Jhernandez and @Mholloway the instance maps1004 is ready to go and the updated report is available here https://www.mediawiki.org/wiki/Wikimedia_Maps/Tile_generation_report
Change 464441 abandoned by MSantos:
Skipping download if PBF file exists
Reason:
Change is rebased and without conflicts at I6a6ea30949620ebc6ac7eb24e68b754ba76f8d01