Page MenuHomePhabricator

wdqs admins should have access to nginx logs, jstack on wdqs machines
Closed, ResolvedPublic

Description

During an incident it has been pointed out that wdqs-admins, a group which has sudo privileges to restart various wdqs-related services and nginx, can not read nginx logs in /var/log/nginx/error.log.

This is a follow-up task to make sure admins who can restart nginx can also read its error logs.

It should be noted that the group already has privileges for journalctl. ALL = NOPASSWD: /bin/journalctl *.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

sudo journalctl -u nginx should already work but it does not contain the same information that is in the error.log (checked on wdqs2007).

Change 615818 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: let wdqs-admins view nginx logs

https://gerrit.wikimedia.org/r/615818

Change 615821 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: let wdqs-admins run jstack as root

https://gerrit.wikimedia.org/r/615821

RKemper renamed this task from wdqs admins should have access to nginx logs on wdqs machines to wdqs admins should have access to nginx logs, jstack on wdqs machines.Jul 23 2020, 9:14 PM

@dcausse As John pointed out on Gerrit the (only) java process running on wdqs machines is running as the blazegraph user and wdqs-admins already can run any commands as the user blazegraph.

So this works for me:

sudo -u blazegraph jstack 46484

where 46484 is one of the java processes. Does that work for you?

I could not use this command last time and it did not work I think because the jvm was too busy. I need root to use jstack -F which I think has more chances to attach the jvm.

Joe triaged this task as High priority.Jul 27 2020, 9:14 AM

We discussed this during this week's SRE meeting and resolved to enable full root access for wdqs admins, rather than granularly expanding access one file at a time. This also lines up better with how we currently manage Elasticsearch, where admins have root access.

Change 616564 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: turn all wdqs-admins into wdqs-roots

https://gerrit.wikimedia.org/r/616564

Here's a new patch that moves all wdqs-admins to wdqs-roots and then also removes the wdqs-admins group entirely and changes which group is used for the deployment_server access that comes with it. Please review above.

Change 616593 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: add all members of wdqs-admins to wdqs-roots

https://gerrit.wikimedia.org/r/616593

Change 616593 merged by Herron:
[operations/puppet@production] admins: add all members of wdqs-admins to wdqs-roots

https://gerrit.wikimedia.org/r/616593

https://gerrit.wikimedia.org/r/616593 has been merged, and I've ran puppet on the wdqs* hosts.

@dcausse @RKemper could you have a look and confirm that's working as expected for you?

@herron thanks for the deploy. It works well for me.
For jstack I need an extra package for it to function properly (https://gerrit.wikimedia.org/r/c/operations/puppet/+/617074).

@herron thanks for the deploy. It works well for me.
For jstack I need an extra package for it to function properly (https://gerrit.wikimedia.org/r/c/operations/puppet/+/617074).

Ack, and I see that patch has been merged now, great. I'll transition this to resolved, but please don't hesitate to re-open if needed. Thanks!

Change 615818 merged by Dzahn:
[operations/puppet@production] admins: let wdqs-admins view nginx logs

https://gerrit.wikimedia.org/r/615818

Change 615821 merged by Dzahn:
[operations/puppet@production] admins: let wdqs-admins run jstack as root

https://gerrit.wikimedia.org/r/615821

Change 616564 abandoned by Dzahn:
[operations/puppet@production] admins: turn all wdqs-admins into wdqs-roots

Reason:
replaced by multiple steps that are already merged, ticket is resolved

https://gerrit.wikimedia.org/r/616564