Page MenuHomePhabricator

Verify loading instructions for existing styles on Stretch
Closed, ResolvedPublic

Description

Roughly follow https://wikitech.wikimedia.org/wiki/Maps#Setup, but with some modifications per @Pnorman:

When it comes to the maps team/RI team testing the existing instructions and cleaning them up, the following parts of the new instructions might be useful, as well as the following notes.

  • Any instructions that reference the "ct" database will need to run on the "gis" database, but should not be followed blindly. Different extensions and tables are present in the two setups.
  • Anything using process-osm-data should not be run. That script runs the commands in the readmes of cleartables, meddo, brighmed, and kartotherian. You could also run them by hand. A script with the instructions for osm-bright.tm2source and kartotherian is needed instead. Because there are fewer components, this is simpler, and a script might not be necessary.

Of the instructions on https://wikitech.wikimedia.org/wiki/Maps/Loading, these sections could be applied to osm-bright.tm2source and osm-bright.tm2:

  • Downloading sections 1-2
  • Database checks (The first occurance of checks)
  • Database setup
  • Database checks (2nd occurance) rendering tables, slim tables, and rendering read access

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Discovery. · View Herald TranscriptJun 29 2018, 2:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Pnorman @Gehel

First issue: I've dowloaded and checksummed the OSM full planet PBF for 6/25. Following the https://wikitech.wikimedia.org/wiki/Maps#Importing_database instructions (since process_osm_data is unavailable), I run the osm2pgsql command, with the following result:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL: no pg_hba.conf entry for host "10.192.16.34", user "osmimporter", database "gis", SSL off

Looking at pg_hba.conf, there is an entry for user osmimporter, but for the IP range 127.0.0.1/32. Should something be updated in Puppet to fix this? Can I just update the file manually for now?


Separately, I notice there's an osm-initial-import shell script. Should I be using this instead?

Mholloway renamed this task from Verify loading instructions for existing styles on Stretch to Verify loading instructions for existing styles on Stretch on maps-test2003.Jun 29 2018, 2:36 PM
Gehel added a comment.Jun 29 2018, 2:39 PM

using localhost instead of maps-test2003 should resolve the issue. It makes some sense to only allow osmupdater on 127.0.0.1. Ideally, we should probably move to unix sockets, but not for right now.

Import is now in progress.

Change 443668 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[operations/puppet@production] Use backports version of osm2pgsql on Stretch for improved memory handling

https://gerrit.wikimedia.org/r/443668

Mholloway edited projects, added Maps-Sprint; removed Discovery.Jul 3 2018, 7:05 PM
Mholloway moved this task from Backlog to In progress on the Maps-Sprint board.

Per convo with @Pnorman earlier, trying again with a smaller cache size. Separately, the patch above will give us a newer version of osm2pgsql on Stretch including memory management improvements, which should also help.

Change 443668 merged by Gehel:
[operations/puppet@production] Use backports version of osm2pgsql on Stretch for improved memory handling

https://gerrit.wikimedia.org/r/443668

Mholloway changed the task status from Open to Stalled.Jul 10 2018, 6:17 PM
Mholloway moved this task from In progress to Stalled/Waiting on the Maps-Sprint board.

Pausing this for now in favor of getting a Stretch testing environment set up in Cloud VPS. To be continued there.

LGoto added a subscriber: LGoto.Jul 18 2018, 3:46 PM

@Mholloway can you add the Stretch task as a child of this ticket? Thanks!

Mholloway renamed this task from Verify loading instructions for existing styles on Stretch on maps-test2003 to Verify loading instructions for existing styles on Stretch.Jul 23 2018, 7:31 PM
Mholloway changed the task status from Stalled to Open.
Mholloway moved this task from Stalled/Waiting to In progress on the Maps-Sprint board.
Mholloway closed this task as Resolved.Aug 2 2018, 7:08 PM

Data loaded on deployment-maps04, running Stretch. See new instructions at https://wikitech.wikimedia.org/wiki/Maps/Beta_Cluster_setup.