Page MenuHomePhabricator

Get puppet runs into logstash
Closed, ResolvedPublic

Revisions and Commits

Related Objects

StatusSubtypeAssignedTask
OpenFeatureNone
Resolvedbd808

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Diffusion mentioned this in Unknown Object (Diffusion Commit).Apr 24 2015, 5:04 PM
bd808 changed the task status from Open to Stalled.Aug 14 2015, 11:23 PM

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

emailbot mentioned this in Unknown Object (Task).Dec 18 2015, 6:37 PM
This comment was removed by Legoktm.

Change 274054 had a related patch set uploaded (by Xqt):
Remove imagetransfer.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

This comment was removed by Legoktm.

Change 291859 had a related patch set uploaded (by Xqt):
Remove casechecker .py from "T001/T003" testing (print function)

https://gerrit.wikimedia.org/r/291859

Change 274054 abandoned by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 303408 had a related patch set uploaded (by Xqt):
Remove casechecker.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/303408

Change 291859 abandoned by Xqt:
Remove casechecker .py from "T001/T003" testing (print function)

Reason:
duplicate of I1d23ab55e

https://gerrit.wikimedia.org/r/291859

Change 303408 merged by jenkins-bot:
Remove casechecker.py from ignoring T001/T003 (print function)

https://gerrit.wikimedia.org/r/303408

Change 274054 restored by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 308753 had a related patch set (by Paladox) published:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Change 308753 merged by Dzahn:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Change 330972 had a related patch set (by Paladox) published:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Change 330972 abandoned by Paladox:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Change 274054 merged by jenkins-bot:
Remove imagetransfer.py from T001/T003 testing

https://gerrit.wikimedia.org/r/274054

Change 407865 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Modify access rules+test+test

https://gerrit.wikimedia.org/r/407865

Change 434719 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

https://gerrit.wikimedia.org/r/434719

This is strange. Even the first task is still unresolved. Can someone please guide me how we go about in the phabricator about backlog clearing?

Aklapper changed the task status from Stalled to Open.Nov 3 2020, 10:38 AM
In T1#1541803, @bd808 wrote:

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

Five years later, is that still the take / situation? (If it is, feel free to update the task status again.)

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

jbond closed this task as Resolved.EditedNov 4 2021, 8:51 PM
jbond assigned this task to bd808.
jbond updated Other Assignee, added: jbond.

after the last change above, we now have puppet report data going into logstash. I haven't played with it much and im a elastic noob but the following is an example showing when a hosts where running a specific puppet git sha1 https://logstash.wikimedia.org/goto/473ad9da22eb1255d4168658f42a7ed5

As said i have not played with the data much but here is an example of what data gets sent on each run.

In T1#6599767, @Volans wrote:

@Aklapper TL;DR yes, as it depends on T213902

i think this comment referees to puppet diff's however the implementation doesn't add the diffs and im not sure if we would want them. As such i think we can finally close this task down and will boldly do so