Page MenuHomePhabricator

Load OSM data into maps1004
Closed, ResolvedPublic

Description

Load OSM data for maps1004 and test machine after reimage

  • Load a small set of the planet for testing
  • Update Kartotherian/Tilerator to latest
  • Generate tiles and test all services
  • Start full planet sync
  • Monitor it until everything is complete

Event Timeline

MSantos created this task.Sep 25 2018, 5:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2018, 5:17 PM
MSantos triaged this task as High priority.Sep 25 2018, 5:17 PM
MSantos updated the task description. (Show Details)Sep 28 2018, 5:43 PM

To generate tiles and test the maps1004 instance we need to update kartotherian/tilerator packages so they can work properly with the mapnik 3.0.20.

We are waiting until the next deploy window to deploy the services.

Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:39:12Z] <mholloway-shell@deploy1001> Started deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462)

Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:39:31Z] <mholloway-shell@deploy1001> Finished deploy [tilerator/deploy@22f90ee] (maps1004): Update tilerator to latest (T205462) (duration: 00m 19s)

Mentioned in SAL (#wikimedia-operations) [2018-10-01T06:59:58Z] <mholloway-shell@deploy1001> Started deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462)

Mentioned in SAL (#wikimedia-operations) [2018-10-01T07:00:15Z] <mholloway-shell@deploy1001> Finished deploy [kartotherian/deploy@ab6cb74] (maps1004): Update kartotherian to latest (T205462) (duration: 00m 16s)

Mholloway updated the task description. (Show Details)Oct 1 2018, 7:01 AM
Mholloway added a subscriber: Mholloway.

Kartotherian and Tilerator are now up to date on maps1004.

MSantos updated the task description. (Show Details)Oct 1 2018, 7:16 PM
MSantos updated the task description. (Show Details)
MSantos updated the task description. (Show Details)Oct 3 2018, 7:22 PM

Change 464439 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Skipping download if PBF file exists

https://gerrit.wikimedia.org/r/464439

Change 464441 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Skipping download if PBF file exists

https://gerrit.wikimedia.org/r/464441

Change 464439 abandoned by MSantos:
Skipping download if PBF file exists

Reason:
Duplicate

https://gerrit.wikimedia.org/r/464439

MSantos added a subscriber: Gehel.Oct 3 2018, 10:00 PM

@Gehel and @Mholloway I am having permission issues with the OSM initial import script. I wrote the patch above to help me debug.

This task is temporarily blocked.

The OSM initial import script finished execution after ≈ 18,20 hours.
I am now looking for a way to confirm that everything is fine before start generating tiles.

The log is attached if you would like to see the outputs.

After performing the osm-brighttm2source steps with the SQLs I got some errors and it need more investigation

$ sudo -u postgres psql -Xqd gis -f sql/admin.sql
$ sudo -u postgres psql -Xqd gis -f sql/functions.sql
$ sudo -u postgres psql -Xqd gis -f sql/create-indexes.sql
psql:sql/create-indexes.sql:250: ERROR:  duplicate key value violates unique constraint "pg_class_relname_nsp_index"         
DETAIL:  Key (relname, relnamespace)=(planet_osm_ways_nodes, 2200) already exists.                                           
CONTEXT:  SQL statement "CREATE INDEX planet_osm_ways_nodes ON planet_osm_ways USING gin (nodes) WITH (fastupdate=off)"      
PL/pgSQL function inline_code_block line 12 at SQL statement
$ sudo -u postgres psql -Xqd gis -f sql/names.sql
$ sudo -u postgres psql -d gis -c 'select populate_admin();'
ERROR:  Geometry type (Point) does not match column type (LineString)                                                        
CONTEXT:  PL/pgSQL function insert_admin_row(admin,geometry) line 3 at assignment                                            
SQL statement "SELECT insert_admin_row(theRow, ST_GeometryN(geom, i))"                                                       
PL/pgSQL function insert_admin_rows(admin,geometry,boolean) line 10 at PERFORM                                               
SQL statement "SELECT insert_admin_rows(row, result, TRUE)"
PL/pgSQL function populate_admin() line 26 at PERFORM

Change 468257 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: restrict OSM sync check to maps1001

https://gerrit.wikimedia.org/r/468257

Change 468257 merged by Gehel:
[operations/puppet@production] maps: restrict OSM sync check to maps1001

https://gerrit.wikimedia.org/r/468257

Change 482648 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Restore settings for maps1004

https://gerrit.wikimedia.org/r/482648

Change 482648 merged by Gehel:
[operations/puppet@production] Restore settings for maps1004

https://gerrit.wikimedia.org/r/482648

@Gehel, @Jhernandez and @Mholloway the instance maps1004 is ready to go and the updated report is available here https://www.mediawiki.org/wiki/Wikimedia_Maps/Tile_generation_report

MSantos updated the task description. (Show Details)Jan 11 2019, 4:03 PM

@Gehel, @Jhernandez and @Mholloway the instance maps1004 is ready to go and the updated report is available here https://www.mediawiki.org/wiki/Wikimedia_Maps/Tile_generation_report

\o/

Gehel closed this task as Resolved.Feb 7 2019, 3:53 PM

Change 464441 abandoned by MSantos:
Skipping download if PBF file exists

Reason:
Change is rebased and without conflicts at I6a6ea30949620ebc6ac7eb24e68b754ba76f8d01

https://gerrit.wikimedia.org/r/464441