Get puppet runs into logstash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bd808
	Feb 5 2014, 11:17 PM

Details

Other Assignee: jbond
Reference: fl4

Project	Branch	Lines +/-	Subject
operations/puppet	production	+2 -1	P:rsyslog: ship puppetmaster logs to kafka
operations/puppet	production	+6 -0	puppetmaster: enable logstash reports
operations/puppet	production	+0 -1	puppet-agent: remove --show_diff from scheduled puppet-run script
mediawiki/core	master	+0 -0	Test: Do not merge

Customize query in gerrit

	Reference	Source Branch	Dest Branch	Author	Title
	repos/releng/release!10	tarball-echo-plus	master	jforrester	Add Echo, LoginNotfiy, and Thanks to the default MediaWiki bundle

Customize query in GitLab

Revisions and Commits

Unknown Object (Diffusion Commit)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open	Feature	None	T63779 Add system logs to logstash (tracking)
		Resolved		• bd808	T1 Get puppet runs into logstash

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Krenair awarded a token.Aug 13 2015, 11:57 PM

Krenair rescinded a token.

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

• MZMcBride added a subscriber: • MZMcBride.Aug 30 2015, 12:41 AM

• emailbot mentioned this in Unknown Object (Task).Dec 18 2015, 6:37 PM

• ZOKIDIN added a subscriber: • ZOKIDIN.Jan 30 2016, 5:38 AM

This comment was removed by Legoktm.

santhosh mentioned this in T128162: Resuming a re-translated deleted translation will cause restoring the sections from old deleted translation.Feb 26 2016, 5:10 AM

Change 274054 had a related patch set uploaded (by Xqt):
Remove imagetransfer.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

gerritbot added a project: Patch-For-Review.Mar 1 2016, 4:42 AM

• ZOKIDIN added a comment.Mar 1 2016, 1:33 PM

This comment was removed by Legoktm.

Change 291859 had a related patch set uploaded (by Xqt):
Remove casechecker .py from "T001/T003" testing (print function)

https://gerrit.wikimedia.org/r/291859

valhallasw mentioned this in T136633: Support multi-task commits.May 31 2016, 6:31 PM

valhallasw mentioned this in T137311: ReleaseTaggerBot broke on 2 June.Jun 8 2016, 2:33 PM

mmodell mentioned this in rOPUP28a86d08f542: Avoid breaking full phabricator URLs.Jun 17 2016, 6:05 PM

Paladox mentioned this in T85002: Gerrit bug search not (naively) working with Phabricator tasks.Jul 25 2016, 4:44 PM

Paladox mentioned this in rOPUPb68924b2979b: Avoid breaking full phabricator URLs.Jul 28 2016, 12:11 PM

Paladox mentioned this in rOPUPfcaae0d53d72: Gerrit: Avoid breaking full phabricator URLs.Jul 31 2016, 6:17 PM

Paladox mentioned this in rOPUPe99ffb93b141: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 10:07 AM

Paladox mentioned this in rOPUP1b010e47fcfd: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:51 PM

Paladox mentioned this in rOPUPc0f0b85eac3f: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:56 PM

Change 274054 abandoned by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 303408 had a related patch set uploaded (by Xqt):
Remove casechecker.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/303408

Change 291859 abandoned by Xqt:
Remove casechecker .py from "T001/T003" testing (print function)

Reason:
duplicate of I1d23ab55e

https://gerrit.wikimedia.org/r/291859

Change 303408 merged by jenkins-bot:
Remove casechecker.py from ignoring T001/T003 (print function)

https://gerrit.wikimedia.org/r/303408

Paladox mentioned this in T76459: Allow links to phabricator comments from gerrit messages..Aug 12 2016, 5:32 PM

Paladox mentioned this in T75997: Full Phabricator links are being expanded weirdly in gerrit.Aug 24 2016, 10:44 AM

Paladox mentioned this in rOPUP54c73ebb696b: Fix phabricator expanding links.Aug 24 2016, 11:02 AM

Paladox mentioned this in rOPUP27a89db35696: Fix phabricator expanding links.Aug 24 2016, 11:08 AM

Paladox mentioned this in T144098: Create a bot similar to gerritbot for phabricator differential.Aug 27 2016, 6:22 PM

Change 274054 restored by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 308753 had a related patch set (by Paladox) published:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Paladox mentioned this in D397: Testing some task changes.Oct 1 2016, 11:55 PM

Paladox mentioned this in T146843: Re-evaluate how we implement phabricator's search engine.Oct 2 2016, 11:07 AM

Change 308753 merged by Dzahn:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Change 330972 had a related patch set (by Paladox) published:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Change 330972 abandoned by Paladox:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Tgr added a parent task: T63779: Add system logs to logstash (tracking).Feb 11 2017, 1:37 AM

Change 274054 merged by jenkins-bot:
Remove imagetransfer.py from T001/T003 testing

https://gerrit.wikimedia.org/r/274054

greg moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Feb 14 2017, 6:52 PM

Fz-29 mentioned this in T164623: Weekly Reports : Add a "hierarchy" type to the Cargo extension [GSoC-2017].Jun 24 2017, 9:56 PM

Awesome_Aasim added a subscriber: Awesome_Aasim.Sep 17 2017, 5:03 AM

Awesome_Aasim removed a subscriber: Awesome_Aasim.

• IZoid added a subtask: T49578: Score should output SVG.Dec 9 2017, 6:34 PM

Ebe123 removed a subtask: T49578: Score should output SVG.Dec 9 2017, 6:37 PM

• Gilles mentioned this in T165272: Review research on performance perception.Jan 3 2018, 2:15 PM

Change 407865 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Modify access rules+test+test

https://gerrit.wikimedia.org/r/407865

Change 434719 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

https://gerrit.wikimedia.org/r/434719

Diffusion mentioned this in rGERRITDEPLOY0e03b79ba4a7: Fix NPE in ChangeDetailFactory for loadMessages isPatchVisible.Jun 7 2018, 6:55 PM

Diffusion mentioned this in rGERRITDEPLOY0ec0d19c468c: ListTags: Fix ref in TagInfo for signed/annotated tags.Jun 7 2018, 7:13 PM

ClemFlip mentioned this in T143326: Invent automatic segmentation in translation units without <translate> tags.Dec 13 2018, 1:13 PM

Ssomansh mentioned this in T219442: Google Summer of Code(2019): Design and Development of a tool to track developer's activity.Mar 27 2019, 7:36 PM

This is strange. Even the first task is still unresolved. Can someone please guide me how we go about in the phabricator about backlog clearing?

@Capankajsmilyo: Please move off-topic general questions to mw:Talk:Phabricator/Help and read mw:Bug management/Development prioritization

Legoktm added a subscriber: Legoktm.Jun 25 2019, 7:27 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:34 PM

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 20 2020, 1:29 PM

DVrandecic mentioned this in T261474: First set of builtin functions.Aug 27 2020, 10:16 PM

In T1#1541803, @bd808 wrote:

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

Five years later, is that still the take / situation? (If it is, feel free to update the task status again.)

@Aklapper TL;DR yes, as it depends on T213902

Thanks. Added subtask.

IN mentioned this in T270021: 404 when attempting phabricator tasks with lowercase t.Dec 13 2020, 9:27 AM

jbond added a subscriber: jbond.Jan 25 2021, 10:33 AM

lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

Maintenance_bot added a project: observability.Jul 12 2021, 2:49 AM

• bd808 mentioned this in Blog Post: Shrinking the tasks backlog.Jul 14 2021, 6:29 PM

lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:10 AM

BrandonXLF mentioned this in T288367: More menu items are too large in modern Vector skin.Aug 6 2021, 6:48 PM

lmata edited projects, added Observability-Logging; removed observability, SRE Observability.Aug 9 2021, 12:18 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 12:45 AM

lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 1:00 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 1:46 AM

lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:53 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:49 AM

DVrandecic mentioned this in T292892: Function model: Introduce a Union generic type.Oct 9 2021, 3:01 AM

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

after the last change above, we now have puppet report data going into logstash. I haven't played with it much and im a elastic noob but the following is an example showing when a hosts where running a specific puppet git sha1 https://logstash.wikimedia.org/goto/473ad9da22eb1255d4168658f42a7ed5

As said i have not played with the data much but here is an example of what data gets sent on each run.

In T1#6599767, @Volans wrote:

@Aklapper TL;DR yes, as it depends on T213902

i think this comment referees to puppet diff's however the implementation doesn't add the diffs and im not sure if we would want them. As such i think we can finally close this task down and will boldly do so

Restricted Application added a project: User-bd808. · View Herald TranscriptNov 4 2021, 8:51 PM

Aklapper removed a subtask: T213902: Implement sensitive logstash access control.Nov 5 2021, 2:59 PM

herron awarded a token.Nov 5 2021, 3:34 PM

SunAfterRain mentioned this in T296128: Wikipedia app process interwiki link(s) error..Nov 20 2021, 12:29 PM

CactiStaccingCrane awarded a token.Dec 13 2021, 11:48 AM

MarcoAurelio mentioned this in T298041: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured".Dec 20 2021, 5:42 PM

• bd808 mentioned this in T325381: stashbot: timestamps in SAL messages can be interpreted as ticket links.Dec 16 2022, 6:29 PM

Change 875401 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] phabricator: update pattern to support old tickets like T1

https://gerrit.wikimedia.org/r/875401

Change 875401 merged by Jbond:

[operations/puppet@production] phabricator: update pattern to support old tickets like T1