Page MenuHomePhabricator
Create Task
Maniphest T198485

Verify loading instructions for existing styles on Stretch
Closed, ResolvedPublic
Actions

Description

Roughly follow https://wikitech.wikimedia.org/wiki/Maps#Setup, but with some modifications per @Pnorman:

When it comes to the maps team/RI team testing the existing instructions and cleaning them up, the following parts of the new instructions might be useful, as well as the following notes.

  • Any instructions that reference the "ct" database will need to run on the "gis" database, but should not be followed blindly. Different extensions and tables are present in the two setups.
  • Anything using process-osm-data should not be run. That script runs the commands in the readmes of cleartables, meddo, brighmed, and kartotherian. You could also run them by hand. A script with the instructions for osm-bright.tm2source and kartotherian is needed instead. Because there are fewer components, this is simpler, and a script might not be necessary.

Of the instructions on https://wikitech.wikimedia.org/wiki/Maps/Loading, these sections could be applied to osm-bright.tm2source and osm-bright.tm2:

  • Downloading sections 1-2
  • Database checks (The first occurance of checks)
  • Database setup
  • Database checks (2nd occurance) rendering tables, slim tables, and rendering read access

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Use backports version of osm2pgsql on Stretch for improved memory handlingoperations/puppetproduction+7 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Mholloway created this task.Jun 29 2018, 2:19 PM
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptJun 29 2018, 2:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Mholloway triaged this task as High priority.Jun 29 2018, 2:20 PM
• Mholloway added a project: Product-Infrastructure-Team-Backlog-Deprecated (Kanban).
• Mholloway moved this task from To Do to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.
• Mholloway added a subscriber: Gehel.Jun 29 2018, 2:36 PM

@Pnorman @Gehel

First issue: I've dowloaded and checksummed the OSM full planet PBF for 6/25. Following the https://wikitech.wikimedia.org/wiki/Maps#Importing_database instructions (since process_osm_data is unavailable), I run the osm2pgsql command, with the following result:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL: no pg_hba.conf entry for host "10.192.16.34", user "osmimporter", database "gis", SSL off

Looking at pg_hba.conf, there is an entry for user osmimporter, but for the IP range 127.0.0.1/32. Should something be updated in Puppet to fix this? Can I just update the file manually for now?

Separately, I notice there's an osm-initial-import shell script. Should I be using this instead?

• Mholloway renamed this task from Verify loading instructions for existing styles on Stretch to Verify loading instructions for existing styles on Stretch on maps-test2003.Jun 29 2018, 2:36 PM
Gehel added a comment.Jun 29 2018, 2:39 PM

using localhost instead of maps-test2003 should resolve the issue. It makes some sense to only allow osmupdater on 127.0.0.1. Ideally, we should probably move to unix sockets, but not for right now.

• Mholloway added a comment.Jun 29 2018, 3:18 PM

Import is now in progress.

• Mholloway added a comment.Jun 30 2018, 5:44 PM

Import failed: output here

gerritbot subscribed.Jul 3 2018, 6:14 PM

Change 443668 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[operations/puppet@production] Use backports version of osm2pgsql on Stretch for improved memory handling

https://gerrit.wikimedia.org/r/443668

gerritbot added a project: Patch-For-Review.Jul 3 2018, 6:14 PM
• Mholloway edited projects, added Maps-Sprint; removed Discovery-ARCHIVED.Jul 3 2018, 7:05 PM
• Mholloway moved this task from Backlog to In progress on the Maps-Sprint board.
• Mholloway added a comment.Jul 3 2018, 7:41 PM

Per convo with @Pnorman earlier, trying again with a smaller cache size. Separately, the patch above will give us a newer version of osm2pgsql on Stretch including memory management improvements, which should also help.

gerritbot added a comment.Jul 4 2018, 9:16 AM

Change 443668 merged by Gehel:
[operations/puppet@production] Use backports version of osm2pgsql on Stretch for improved memory handling

https://gerrit.wikimedia.org/r/443668

• Mholloway changed the task status from Open to Stalled.Jul 10 2018, 6:17 PM
• Mholloway moved this task from In progress to Stalled/Waiting on the Maps-Sprint board.

Pausing this for now in favor of getting a Stretch testing environment set up in Cloud VPS. To be continued there.

• Mholloway moved this task from Doing to Blocked on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Jul 10 2018, 6:19 PM
LGoto subscribed.Jul 18 2018, 3:46 PM

@Mholloway can you add the Stretch task as a child of this ticket? Thanks!

• Mholloway added a subtask: T172090: move maps-test cluster to cloud infrastructure.Jul 18 2018, 3:49 PM
• Mholloway renamed this task from Verify loading instructions for existing styles on Stretch on maps-test2003 to Verify loading instructions for existing styles on Stretch.Jul 23 2018, 7:31 PM
• Mholloway changed the task status from Stalled to Open.
• Mholloway added a parent task: T198622: migrate maps servers to stretch with the current style.
• Mholloway moved this task from Stalled/Waiting to In progress on the Maps-Sprint board.
• Mholloway moved this task from Blocked to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Jul 23 2018, 7:33 PM
• Mholloway closed this task as Resolved.Aug 2 2018, 7:08 PM

Data loaded on deployment-maps04, running Stretch. See new instructions at https://wikitech.wikimedia.org/wiki/Maps/Beta_Cluster_setup.

• Jhernandez closed subtask T172090: move maps-test cluster to cloud infrastructure as Resolved.Aug 23 2018, 4:54 PM
• Mholloway mentioned this in T202888: tilerator / tileratorui crashed on maps-test2003.Aug 27 2018, 2:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits