tools.osmlint@tools-sgebastion-06:~$ psql -h osmdb.eqiad.wmnet -U osm -d gis psql (9.6.10) Type "help" for help. gis=> select * from planet_osm_polygon limit 1; ERROR: permission denied for relation planet_osm_polygon
2018-12-09 06:29:47 GMT ERROR: permission denied for relation planet_osm_polygon <--- These errors go back ages. That not new. That might go back to the re-image, perhaps or some other time, but that one is less likely to relate to moving the jobs.
@MaxSem Can you confirm you were able to actually read it on trusty? The logs suggest at least somebody couldn't before we even completed the stretch bastion. There might be some local permissions lost during the re-image months ago that we need to re-do or something more toward the end of 2018 when the puppet changes broke things for a bit. My suspicion is that we are just noticing something has been broken for a long time, which is a good opportunity to fix it!
If the permissions go away, then something is overwriting them, and we'll need to track that down. I think this should fix the problem as long as nobody has an expectation of writing directly to these tables (which would surprise me since these are syncs from osm).
Based on the logs, I am entirely convinced that this was broken for ages and somehow nobody noticed or raised the issue. I'm so glad you did @MaxSem!
As I indicated in the other ticket, the old nodes indeed were also throwing this error when I was trying to validate if the new node was working. This might mean that tile updates were disabled for many months, and tiles were being served from disk cache instead or something...
I can confirm access now works, and it seems the tile servers of the map project are now catching up with tile freshness