Page MenuHomePhabricator
Create Task
Maniphest T254014

Reimport OSM data on eqiad
Closed, ResolvedPublic
Actions

Description

As a fallout of T243609, we need to reset our Postgres DB to a sane state.

The process:

  • 1) disable tilerator
  • 2) depool maps1004
  • 3) isolate maps1004 from the rest of the cluster [MrG]
  • 4) delete postgres data, recreate the empty DB [MrG]
  • 5) run script to reimport data from OSM, needs to run from root [MrG]
osm-initial-import \
   -d 201019 \ latest dump
   -s  https://planet.openstreetmap.org/replication/hour/000/071/112.state.txt \ hourly replication
   -x webproxy.eqiad.wmnet:8080
  • 6) enable replication and check at least one cycle of OSM replication
  • 7) repool maps1004
  • 8) re-init all slaves (cookbook)
  • 9) enable tilerator

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -2tilerator: enable in eqiad
operations/puppetproduction+0 -3Isolate eqiad master maps1004 from cluster
operations/puppetproduction+1 -1Enable replication in eqiad
operations/puppetproduction+2 -0Temporarily disable tilerator in eqiad
operations/puppetproduction+1 -1maps: reenable eqiad OSM replication
Customize query in gerrit

Related Objects
Search...

Event Timeline

MSantos created this task.May 29 2020, 2:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2020, 2:35 PM
MSantos added projects: Maps, Discovery-Search, Product-Infrastructure-Team-Backlog-Deprecated.May 29 2020, 2:36 PM
MSantos moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.
MSantos mentioned this in T249086: Reimport OSM data on codfw.
MSantos added subscribers: Ainali, TheDJ, Atomsmith, Gehel.
MSantos updated the task description. (Show Details)May 29 2020, 2:40 PM
Gehel updated the task description. (Show Details)Jun 22 2020, 7:55 PM
Gehel added a parent task: T243609: Maps master servers running out of space.Jun 26 2020, 7:09 AM
Gehel mentioned this in T243609: Maps master servers running out of space.
Gehel triaged this task as High priority.Jun 26 2020, 8:02 AM
gerritbot added a comment.Jun 29 2020, 8:45 PM

Change 608459 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/puppet@production] Temporarily disable tilerator

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608459

gerritbot added a project: Patch-For-Review.Jun 29 2020, 8:45 PM
gerritbot added a comment.Jun 30 2020, 8:43 PM

Change 608726 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/puppet@production] Enable replication in eqiad

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608726

gerritbot added a comment.Jun 30 2020, 8:59 PM

Change 608729 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/puppet@production] Isolate eqiad master maps1004 from cluster

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608729

wolfgang8741 added a subscriber: wolfgang8741.Jul 13 2020, 2:36 PM
mxn added a subscriber: mxn.Jul 16 2020, 9:11 PM
CBogen moved this task from needs triage to Ops / SRE on the Discovery-Search board.Jul 23 2020, 7:29 PM
jhsoby added a subscriber: jhsoby.Aug 11 2020, 12:33 PM
MSantos added a parent task: T260456: [Maps] Reduce Map Sync Latency with OpenStreetMaps (OSM).Aug 17 2020, 2:03 PM
MSantos moved this task from All map-related tasks to Production Infrastructure on the Maps board.Sep 24 2020, 10:44 AM
MSantos mentioned this in T137939: Increase frequency of OSM replication.Sep 30 2020, 3:10 PM
hnowlan added a subscriber: hnowlan.Oct 1 2020, 2:31 PM
ScottDavis added a subscriber: ScottDavis.Oct 5 2020, 2:34 PM
WDoranWMF added a project: Platform Team Workboards (Green).Oct 7 2020, 6:04 AM
hnowlan added a comment.Oct 12 2020, 2:59 PM

Just a few queries to get me clued in:

  • Is there advance notice required for these changes or a safe period within which to do it?
  • For deleting the postgres data, is it a matter of just doing a dropdb?
  • I assume the cookbook required is sre.postgresql.postgres-init?

While I'm here - are there any plans to move maps servers to buster? I know it would further slow this rollout but if the whole DC is out of commission at the moment now could be a good time to attempt moving so as to save further disruption down the road?

MSantos added a comment.Oct 13 2020, 11:33 AM

I know it would further slow this rollout but if the whole DC is out of commission at the moment now could be a good time to attempt moving so as to save further disruption down the road?

In order to the upgrade to buster, here are a few things to take in consideration:

  • Tile regeneration is the pain point of this work, see last report: https://www.mediawiki.org/wiki/Wikimedia_Maps/Tile_generation_report
  • We need a spike to make sure mapnik (and the whole stack) will work okay with buster, last time we had to backport mapnik and pin old versions of some node packages to make it possible
  • I would prefer to fix OSM in eqiad straight away and put the efforts on k8s migration now that we have capacity for it in the Product Infrastructure team
Gehel added a comment.Oct 14 2020, 6:36 PM
In T254014#6536833, @hnowlan wrote:

Just a few queries to get me clued in:

  • Is there advance notice required for these changes or a safe period within which to do it?

This should have no impact on users (reimaging the primary first, re-syncing the replicas one after the other). So no advanced notice should be required and no specific communication.

  • For deleting the postgres data, is it a matter of just doing a dropdb?

Or even rm -rf /srv/postgresql/9.6/main/, puppet should recreate everything.

  • I assume the cookbook required is sre.postgresql.postgres-init?

This is actually badly named, this cookbook only re-init the replicas, not the primary. Also, this cookbook has only been used a few times, a long time ago. It *probably* works. The re-import procedure is documented on wiki.

While I'm here - are there any plans to move maps servers to buster? I know it would further slow this rollout but if the whole DC is out of commission at the moment now could be a good time to attempt moving so as to save further disruption down the road?

There are no plans, except that we know that this is something that must be done at some point. Migration to Buster is likely to be non trivial as some dependencies (mapnik) are not available on Buster.

gerritbot added a comment.Oct 28 2020, 11:33 AM

Change 636909 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] maps: reenable eqiad OSM replication

https://gerrit.wikimedia.org/r/636909

gerritbot added a comment.Oct 28 2020, 11:35 AM

Change 636909 abandoned by Hnowlan:
[operations/puppet@production] maps: reenable eqiad OSM replication

Reason:

https://gerrit.wikimedia.org/r/636909

gerritbot added a comment.Oct 28 2020, 4:15 PM

Change 608459 merged by Hnowlan:
[operations/puppet@production] Temporarily disable tilerator in eqiad

https://gerrit.wikimedia.org/r/608459

gerritbot added a comment.Oct 28 2020, 4:31 PM

Change 608729 merged by Hnowlan:
[operations/puppet@production] Isolate eqiad master maps1004 from cluster

https://gerrit.wikimedia.org/r/608729

MSantos updated the task description. (Show Details)Oct 28 2020, 5:13 PM
WDoranWMF moved this task from Backlog to Doing on the Platform Team Workboards (Green) board.Oct 28 2020, 9:54 PM
gerritbot added a comment.Nov 2 2020, 1:03 PM

Change 608726 merged by Hnowlan:
[operations/puppet@production] Enable replication in eqiad

https://gerrit.wikimedia.org/r/608726

MSantos updated the task description. (Show Details)Nov 3 2020, 3:33 PM
gerritbot added a comment.Nov 5 2020, 7:01 PM

Change 639608 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] tilerator: enable in eqiad

https://gerrit.wikimedia.org/r/639608

hnowlan updated the task description. (Show Details)Nov 5 2020, 7:08 PM

OSM import complete, all replicas up to date. Tilerator is ready to be reenabled

gerritbot added a comment.Nov 5 2020, 8:35 PM

Change 639608 merged by Hnowlan:
[operations/puppet@production] tilerator: enable in eqiad

https://gerrit.wikimedia.org/r/639608

hnowlan updated the task description. (Show Details)Nov 5 2020, 9:06 PM

This is theoretically done. I'll keep an eye on this and resolve tomorrow if all is okay

hnowlan added a comment.Nov 6 2020, 4:58 PM

Seems okay to me, resolving.

hnowlan closed this task as Resolved.Nov 6 2020, 4:58 PM
edwardbetts added a subscriber: edwardbetts.Nov 21 2020, 10:41 AM
jhsoby added a comment.Nov 23 2020, 3:07 PM

@hnowlan Sorry if this is the wrong place to ask, but do you know why lower zoom levels haven't been updated yet? On January 1st, 2020, a lot of counties and municipalities in Norway merged. At zoom levels 10 and higher, they appear correctly:

Skjermdump fra 2020-11-23 16-04-15.png (594×490 px, 288 KB)

But at levels 9 and lower, the old borders are still visible:
Skjermdump fra 2020-11-23 16-02-39.png (573×543 px, 369 KB)

Any idea as to why?

Bstorm mentioned this in T285668: Labs OSMdb is outdated/outofsync.Aug 30 2021, 10:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL