Page MenuHomePhabricator

Write new import script
Closed, ResolvedPublic

Description

A new script needs to be written to download and import OSM data

This should also prepare the files for T157611: Add functionality to keep up to date planet and run osmborder to puppet

Updates will also handling. TBD: have one script do both with different options, or independent scripts

Event Timeline

Pnorman created this task.Feb 8 2017, 8:11 PM
Pnorman updated the task description. (Show Details)Feb 8 2017, 8:13 PM

@Gehel what Postgres user will the import/updates be done as, and what user does the rendering?

Also, what's the current update script?

Deskana moved this task from Backlog to In progress on the Maps-Sprint board.Mar 3 2017, 6:52 PM
Pnorman reassigned this task from Pnorman to Gehel.Mar 3 2017, 6:52 PM

The script that was written for this seems to be working fine at generating borders and loading them into the database. We're still testing this on the full planet.

Gehel added a comment.Mar 3 2017, 6:55 PM

@Pnorman has a script written and tested (@Pnorman could you link that script from here?). Time now for @Gehel to integrate that into our puppet modules.

The script I've been working on is https://gist.github.com/pnorman/17b17e2d76505c7365278cbc1c7af80f

The puppet changes needed will be

  • Locations of things
  • DB creation can be removed
  • osm2pgsql memory settings

Also, post-update stuff with tile expiry might need some adjustment

Worth noting that the verbosity and logging locations will need to be changed so we have logging, notification on errors, and no undesired emails to ops.

Change 345573 had a related patch set uploaded (by Gehel):
[operations/puppet@production] postgresql - simplify creation of databases

https://gerrit.wikimedia.org/r/345573

Change 345573 merged by Gehel:
[operations/puppet@production] postgresql - simplify creation of databases

https://gerrit.wikimedia.org/r/345573

debt moved this task from In progress to Stalled/Waiting on the Maps-Sprint board.May 30 2017, 7:36 PM

We're waiting on a few things before this is done.

Change 378245 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] [WIP] maps: move to vector tiles and cleartables

https://gerrit.wikimedia.org/r/378245

TheDJ added a subscriber: TheDJ.Sep 15 2017, 2:32 PM

Change 383841 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: introduce the base requirements for vector tiles and cleartables

https://gerrit.wikimedia.org/r/383841

Change 378245 abandoned by Gehel:
[WIP] maps: move to vector tiles and cleartables

https://gerrit.wikimedia.org/r/378245

Change 383841 merged by Gehel:
[operations/puppet@production] maps: introduce the base requirements for vector tiles and cleartables

https://gerrit.wikimedia.org/r/383841

Change 341563 abandoned by Gehel:
maps - cleartables osm replication

https://gerrit.wikimedia.org/r/341563