Page MenuHomePhabricator
Create Task
Maniphest T137878

Allow *-admin groups to see systemd logs for their units
Closed, ResolvedPublic
Actions

Description

With the switch to Jessie, services using service::node have started using SystemD, which keeps its own log (containing its start/stop actions and the service's stderr). Alas, these logs are not visible to people in the respective *-admin groups:

mobrovac@scb1001:~$ sudo journalctl -u changeprop

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for mobrovac:

This request extends to the following groups: citoid-admin, changeprop-admin, cxserver-admin, mathoid-admin, maps-admins, graphoid-admin, mobileapps-admin, parsoid-admin (the switch to Jessie will happen soon enough that it makes sense in this request), apertium-admins (idem), ocg-render-admins and sc-admins.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
service::node: expliticly set syslog identifieroperations/puppetproduction+1 -0
service::node: Output std out/err to a fileoperations/puppetproduction+101 -15
Customize query in gerrit

Related Objects

Event Timeline

• mobrovac created this task.Jun 15 2016, 12:23 PM
Restricted Application added a project: SRE. · View Herald TranscriptJun 15 2016, 12:23 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
Krenair subscribed.Jun 15 2016, 2:32 PM

I looked through some of the other -admin groups. Are any of these also relevant?

gerrit-admin
ocg-render-admins
analytics-admins
chromium-admin
zotero-admin
apertium-admins
RobH changed the task status from Open to Stalled.Jun 15 2016, 4:00 PM
RobH claimed this task.
RobH subscribed.

The scope of this is fairly large, but the request/change itself is a single permission (ability to read that log file) for all the admin groups.

I'm not sure if it should cover just what @mobrovac lists, or also include @Krenair's additions. It seems this should likely be added to all admin groups for services that will run on jessie. (It is likely easier to just add to all admin groups if doing so wouldn't break things.)

Since this is a proposed change to the operation of admin/sudo groups, this will require ops meeting review. I'm on clinic duty this and next week, so I'll claim this task (and stall it) until the meeting next Monday.

RobH moved this task from Untriaged to Group Manager Approval Pending on the SRE-Access-Requests board.Jun 15 2016, 4:00 PM
• mobrovac updated the task description. (Show Details)Jun 15 2016, 5:17 PM

I added apertium and OCG to the list. Zotero is not relevant as we'll never switch it to Jessie. For the rest, I really don't know if that would be useful.

• mobrovac updated the task description. (Show Details)Jun 15 2016, 5:18 PM
RobH added a comment.Jun 15 2016, 6:56 PM

@mobrovac: Thank you for the feedback! We'll go with your appended list unless others can point out where the other admin services will migrate to jessie and need it.

I'll be certain to mention the other potential groups when relaying this into the ops meeting.

RobH reassigned this task from RobH to Joe.Jun 20 2016, 5:53 PM
RobH added a subscriber: Joe.

This request has been denied in the operations meeting, in its currently proposed use of sudo rights to view the syslog.

Rather, its been suggested we parse syslog and output the affected/required service lines into their own file, rather than granting sudo.

@Joe will provide feedback on this task (so I am assigning it to him), however this week is immediately preceding Wikimania, and response times will be delayed accordingly.

fgiunchedi triaged this task as Medium priority.Jul 11 2016, 10:48 AM
Joe changed the task status from Stalled to Open.Jul 13 2016, 12:34 PM
Joe added a comment.Jul 13 2016, 12:37 PM

So my point is basically that for every service (at least the standard ones that we define via service::node, we can:

  1. ensure that journald forwards the interesting logs to syslog
  2. Make syslog write to a logfile with mode 0540 owned by the appropriate -admins group
  3. Prepare the corresponding logrotate rules

This should be relatively easy, clean, and avoid adding sudo rules we don't really need.

• mobrovac added a comment.Jul 14 2016, 7:55 AM

Wouldn't it be simpler just to fwd the logs to syslog and allow admins to read /var/log/syslog (which they currently can't) ?

gerritbot subscribed.Jul 14 2016, 4:22 PM

Change 299000 had a related patch set uploaded (by Mobrovac):
service::node: Output std out/err to a file

https://gerrit.wikimedia.org/r/299000

gerritbot added a project: Patch-For-Review.Jul 14 2016, 4:22 PM
• mobrovac mentioned this in rOPUPf029fecd774a: service::node: Output std out/err to a file.Jul 14 2016, 4:28 PM
• mobrovac mentioned this in rOPUP428d86c36359: service::node: Output std out/err to a file.Jul 14 2016, 4:32 PM
• mobrovac added a project: User-mobrovac.Jul 21 2016, 12:14 PM
• mobrovac mentioned this in rOPUP31a56be5960c: service::node: Output std out/err to a file.Jul 22 2016, 10:54 AM
Joe mentioned this in rOPUP1f1d44d4cd14: service::node: Output std out/err to a file.Jul 27 2016, 8:27 AM
Joe mentioned this in rOPUPd3a9bd06a41c: service::node: Output std out/err to a file.Jul 27 2016, 10:39 AM
Joe mentioned this in rOPUP3dac54bc6a94: service::node: Output std out/err to a file.Jul 27 2016, 10:44 AM
Joe mentioned this in rOPUPa80c6aa712b2: service::node: Output std out/err to a file.Jul 27 2016, 10:55 AM
Joe mentioned this in rOPUP18568c5c7144: service::node: Output std out/err to a file.
gerritbot added a comment.Jul 27 2016, 11:04 AM

Change 299000 merged by Giuseppe Lavagetto:
service::node: Output std out/err to a file

https://gerrit.wikimedia.org/r/299000

Joe mentioned this in rOPUP76dcd9e34b10: service::node: Output std out/err to a file.Jul 27 2016, 11:05 AM
Joe mentioned this in rOPUPce9e25762e21: service::node: Output std out/err to a file.
Joe closed this task as Resolved.Jul 27 2016, 11:23 AM
• mobrovac reopened this task as Open.Aug 8 2016, 5:46 PM

Hm, this patch doesn't actually seem to work. All of the services' syslog.log files are empty, but there definitely are entries for them. Perhaps an explicit mention of syslog in the SystemD service files is needed after all?

Joe added a comment.Aug 10 2016, 12:49 PM

@mobrovac I already know what to do in order to fix it. At the moment all logs are registered as "firejail".

• mobrovac added a comment.Aug 10 2016, 12:50 PM

Duh! Good point. So obvious ...

gerritbot added a comment.Aug 10 2016, 1:48 PM

Change 304028 had a related patch set uploaded (by Giuseppe Lavagetto):
service::node: expliticly set syslog identifier

https://gerrit.wikimedia.org/r/304028

gerritbot added a comment.Aug 10 2016, 1:52 PM

Change 304028 merged by Giuseppe Lavagetto:
service::node: expliticly set syslog identifier

https://gerrit.wikimedia.org/r/304028

Joe mentioned this in rOPUP9dc01f1c2e4b: service::node: expliticly set syslog identifier.Aug 10 2016, 1:52 PM
• mobrovac closed this task as Resolved.Aug 10 2016, 2:02 PM
Stashbot subscribed.Aug 10 2016, 3:50 PM

Mentioned in SAL [2016-08-10T15:50:32Z] <_joe_> restarting parsoid for T137878

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits