During an incident it has been pointed out that wdqs-admins, a group which has sudo privileges to restart various wdqs-related services and nginx, can not read nginx logs in /var/log/nginx/error.log.
This is a follow-up task to make sure admins who can restart nginx can also read its error logs.
It should be noted that the group already has privileges for journalctl. ALL = NOPASSWD: /bin/journalctl *.
sudo journalctl -u nginx should already work but it does not contain the same information that is in the error.log (checked on wdqs2007).
Change 615818 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: let wdqs-admins view nginx logs
Change 615821 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: let wdqs-admins run jstack as root
@dcausse As John pointed out on Gerrit the (only) java process running on wdqs machines is running as the blazegraph user and wdqs-admins already can run any commands as the user blazegraph.
So this works for me:
sudo -u blazegraph jstack 46484
where 46484 is one of the java processes. Does that work for you?
I could not use this command last time and it did not work I think because the jvm was too busy. I need root to use jstack -F which I think has more chances to attach the jvm.
We discussed this during this week's SRE meeting and resolved to enable full root access for wdqs admins, rather than granularly expanding access one file at a time. This also lines up better with how we currently manage Elasticsearch, where admins have root access.
Change 616564 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: turn all wdqs-admins into wdqs-roots
Here's a new patch that moves all wdqs-admins to wdqs-roots and then also removes the wdqs-admins group entirely and changes which group is used for the deployment_server access that comes with it. Please review above.
Change 616593 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: add all members of wdqs-admins to wdqs-roots
Change 616593 merged by Herron:
[operations/puppet@production] admins: add all members of wdqs-admins to wdqs-roots
https://gerrit.wikimedia.org/r/616593 has been merged, and I've ran puppet on the wdqs* hosts.
@dcausse @RKemper could you have a look and confirm that's working as expected for you?
@herron thanks for the deploy. It works well for me.
For jstack I need an extra package for it to function properly (https://gerrit.wikimedia.org/r/c/operations/puppet/+/617074).
Ack, and I see that patch has been merged now, great. I'll transition this to resolved, but please don't hesitate to re-open if needed. Thanks!
Change 615818 merged by Dzahn:
[operations/puppet@production] admins: let wdqs-admins view nginx logs
Change 615821 merged by Dzahn:
[operations/puppet@production] admins: let wdqs-admins run jstack as root
Change 616564 abandoned by Dzahn:
[operations/puppet@production] admins: turn all wdqs-admins into wdqs-roots
Reason:
replaced by multiple steps that are already merged, ticket is resolved