Page MenuHomePhabricator

Install / configure new maps servers in codfw
Closed, ResolvedPublic

Details

Related Gerrit Patches:

Related Objects

Event Timeline

Gehel created this task.May 10 2016, 4:14 PM
Restricted Application added a subscriber: Zppix. · View Herald TranscriptMay 10 2016, 4:14 PM
Gehel added a project: Maps-Sprint.
Gehel moved this task from Backlog to In progress on the Maps-Sprint board.

Change 287992 had a related patch set uploaded (by Gehel):
WIP - Preparing configuration for new maps servers

https://gerrit.wikimedia.org/r/287992

Change 287992 merged by Gehel:
Preparing configuration for new maps servers

https://gerrit.wikimedia.org/r/287992

Change 288106 had a related patch set uploaded (by Gehel):
WIP - Allow host specific private configuration

https://gerrit.wikimedia.org/r/288106

Change 288106 abandoned by Gehel:
WIP - Allow host specific private configuration

Reason:
Not actually needed. Now that I understand a bit more how our postgresql module works, let's go for a simpler solution.

https://gerrit.wikimedia.org/r/288106

Change 288215 had a related patch set uploaded (by Gehel):
Create necessary folders for Postgresql and Cassandra

https://gerrit.wikimedia.org/r/288215

Yurik moved this task from All map-related tasks to Tracking on the Maps board.May 12 2016, 10:56 PM
Gehel added a comment.May 19 2016, 2:51 PM

base osm import completed. Postgres slaves are still catching up. This was done with much trial and error (I'm still learning how Postgres works). Puppet is not yet updated and documentation is not much better than when I started.

Log show errors:

2016-05-19 14:50:01 GMT ERROR:  function bail_out(unknown, text) does not exist at character 564

so probably something is still missing in the base import / config.

Yurik added a comment.May 19 2016, 3:00 PM

I think you did not import custom functions as described in https://github.com/kartotherian/osm-bright.tm2source

Gehel added a comment.May 19 2016, 4:45 PM

Custom function imported, no more errors in postgres logs. But tilerator, tileratorui and karthotherian are still not reporting OK. I'm investigating logs, but there does not seem to be all that much information there...

Change 289829 had a related patch set uploaded (by Gehel):
Maps - make redis server configureable

https://gerrit.wikimedia.org/r/289829

Change 289829 merged by Gehel:
Maps - make redis server configureable

https://gerrit.wikimedia.org/r/289829

Gehel added a comment.May 20 2016, 1:06 PM

Log of osm2pgsql run:

Import was done according to documentation.

Change 290472 had a related patch set uploaded (by Gehel):
Specific configuration for new maps cluster.

https://gerrit.wikimedia.org/r/290472

Change 290472 merged by Gehel:
Specific configuration for new maps cluster.

https://gerrit.wikimedia.org/r/290472

Gehel added a comment.May 25 2016, 8:14 AM

replication-osm is now running (and taking time catching up). postgresql access has been fixed, initial data import created a nodes.bin file that had to be moved in the correct place (/srv/osmosis).

Gehel added a comment.Jun 7 2016, 12:28 PM

Tile generation is completed.

Now trying to get initial import and OSM replication to work reliably. Initial import was started yesterday, but failed, most probably because replicate-osm started running from cron in the middle of the import.

Initial import restarted and in progress.

Change 293331 had a related patch set uploaded (by Gehel):
Maps tables should all be owned by osmimporter

https://gerrit.wikimedia.org/r/293331

Change 293477 had a related patch set uploaded (by Gehel):
Configure proxy for HTTPS as well as HTTP in replicate-osm.

https://gerrit.wikimedia.org/r/293477

Change 293331 merged by Gehel:
Maps tables should all be owned by osmimporter

https://gerrit.wikimedia.org/r/293331

Change 293477 merged by Gehel:
Configure proxy for HTTPS as well as HTTP in replicate-osm.

https://gerrit.wikimedia.org/r/293477

MaxSem closed this task as Resolved.Jun 14 2016, 10:16 PM
MaxSem added a subscriber: MaxSem.

These servers are now serving traffic.

Change 288215 abandoned by Gehel:
WIP - Create necessary folders for Postgresql and Cassandra

Reason:
Replaced by multiple more atomic changes

https://gerrit.wikimedia.org/r/288215