Page MenuHomePhabricator

Can't read from OSM replicas when connecting from Stretch bastion
Closed, ResolvedPublic

Description

tools.osmlint@tools-sgebastion-06:~$ psql -h osmdb.eqiad.wmnet -U osm -d gis
psql (9.6.10)
Type "help" for help.

gis=> select * from planet_osm_polygon limit 1;
ERROR:  permission denied for relation planet_osm_polygon

Event Timeline

MaxSem created this task.Feb 7 2019, 10:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2019, 10:12 PM
bd808 renamed this task from Can't read from OSM replicas to Can't read from OSM replicas when connecting from Stretch bastion.Feb 8 2019, 6:37 AM
bd808 added subscribers: TheDJ, bd808.Feb 8 2019, 6:59 AM

Seems to match this comment from @TheDJ:

maps-tiles1 now has access to the pgsql server.
When running the renderer I see SELECT ST_SRID("way") AS srid FROM planet_osm_polygon WHERE "way" IS NOT NULL LIMIT 1; ERROR: permission denied for relation planet_osm_polygon but the old tiles servers seem to have that problem as well.
Now i need a way to verify if the new renderer is creating proper tiles. I hope to be able to get to that this weekend.

@akosiaris and @Bstorm, can one of you take a look at the gis database and osm to see if there are CIDR based restrictions on access that would explain this?

Bstorm added a comment.Feb 9 2019, 5:08 AM

There are, and I'd fixed them for other maps issues during neutron migrations, but there might be more things to change (and puppet did some things since then).

Bstorm added a comment.Feb 9 2019, 4:50 PM

The configs look identical for both CIDRs...

Bstorm added a comment.Feb 9 2019, 4:51 PM
This comment was removed by Bstorm.
Bstorm added a comment.Feb 9 2019, 4:53 PM

I'm going to look around on the server a little. I want to see if there's something else at play there.

Bstorm added a comment.Feb 9 2019, 4:56 PM

2018-12-09 06:29:47 GMT ERROR: permission denied for relation planet_osm_polygon <--- These errors go back ages. That not new. That might go back to the re-image, perhaps or some other time, but that one is less likely to relate to moving the jobs.

Bstorm added a comment.Feb 9 2019, 4:59 PM

@MaxSem Can you confirm you were able to actually read it on trusty? The logs suggest at least somebody couldn't before we even completed the stretch bastion. There might be some local permissions lost during the re-image months ago that we need to re-do or something more toward the end of 2018 when the puppet changes broke things for a bit. My suspicion is that we are just noticing something has been broken for a long time, which is a good opportunity to fix it!

Bstorm added a comment.Feb 9 2019, 5:17 PM

Yup works now. Permissions have been broken for ages, and I am so glad we are noticing now. Setting perms on the other tables.

Bstorm added a comment.Feb 9 2019, 5:18 PM

These basically just supposed to be SELECTable, right?

Bstorm added a comment.EditedFeb 9 2019, 5:27 PM

Also set it so osm can execute functions.

Bstorm added a comment.Feb 9 2019, 5:30 PM

If the permissions go away, then something is overwriting them, and we'll need to track that down. I think this should fix the problem as long as nobody has an expectation of writing directly to these tables (which would surprise me since these are syncs from osm).

Based on the logs, I am entirely convinced that this was broken for ages and somehow nobody noticed or raised the issue. I'm so glad you did @MaxSem!

As I indicated in the other ticket, the old nodes indeed were also throwing this error when I was trying to validate if the new node was working. This might mean that tile updates were disabled for many months, and tiles were being served from disk cache instead or something...

I can confirm access now works, and it seems the tile servers of the map project are now catching up with tile freshness

Bstorm closed this task as Resolved.Feb 11 2019, 11:01 PM
Bstorm claimed this task.

Well, that's good :) I'll close this for now, and we can reopen if there is something else that pops up with them.