Get puppet runs into logstash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Feb 5 2014, 11:17 PM

Details

Other Assignee: jbond
Reference: fl4

Subject	Repo	Branch	Lines +/-
puppet-agent: remove --show_diff from scheduled puppet-run script	operations/puppet	production	+0 -1
P:rsyslog: ship puppetmaster logs to kafka	operations/puppet	production	+2 -1
puppetmaster: enable logstash reports	operations/puppet	production	+6 -0
Test: Do not merge	mediawiki/core	master	+0 -0

Customize query in gerrit

Title	Reference	Author	Source Branch	Dest Branch
deploy: Add http://www.mediawiki.org/ontology/ontology.owl httpbb test	repos/releng/train-dev!56	dancy	main-Ide3492d2ca4c558c0eff54a8459b48942031b536	main
Use static.php to serve www.mediawiki.org/ontology/ontology.owl (v2)	repos/releng/train-dev!54	dancy	main-I991d969dd9343126661d3d6e9159588cb00855fa	main
Remove unused createPrivateSubtask()	repos/phabricator/extensions!30	aklapper	rmCreatePrivateSubtask	wmf/stable
Ignore more phorge transaction types	toolforge-repos/wikibugs2!21	bd808	work/bd808/ignore-more-phorge-events	main
Draft: MergeMessageFileLists.php: Use a realm-specific extension-list	repos/releng/scap!259	thcipriani	work/thcipriani/T125678	master
Remove obsolete defaults for git_server	repos/releng/scap!256	hashar	cfg_git_server	master
Use static.php to serve www.mediawiki.org/ontology/ontology.owl	repos/releng/train-dev!50	dancy	main-I9724e771081b18122c66db329bc2cc6bc31ebec3	main
Enhance Phorge event rendering	toolforge-repos/wikibugs2!17	bd808	work/bd808/irc-formatting	main
phorge: Check for "core:create" transaction type to flag new tasks	toolforge-repos/wikibugs2!11	bd808	work/bd808/T127506-new	main
Handle logstash timeouts separately from spikes in errors reported by logstash	repos/releng/scap!221	dancy	master-Ifd410026dfea14cdd5f90a7ddad9000e2ef4db05	master
maintain-kubeusers: bump to 0.0.122-20240221143623-84a40909	repos/cloud/toolforge/toolforge-deploy!203	project_1317_bot_df3177307bed93c3f34e421e26c86e38	bump_maintain-kubeusers	main
user: fix bad permission for skel-copied files	repos/cloud/toolforge/maintain-kubeusers!14	dcaro	fix_permissions	main
maintain-kubeusers: bump to 0.0.121-20240221131329-d6f9cd11	repos/cloud/toolforge/toolforge-deploy!201	project_1317_bot_df3177307bed93c3f34e421e26c86e38	bump_maintain-kubeusers	main
canary_checks: Improve format of Waiting for canary traffic message	repos/releng/scap!217	dancy	master-I7240ab57c5461e5805562bfcf42a29b2d3680b83	master
user: copy stuff from /etc/skel if there's anything	repos/cloud/toolforge/maintain-kubeusers!13	dcaro	copy_skel	main
Mention how long we're waiting for canary traffic	repos/releng/scap!215	dancy	master-I3745ed1b83962bc72c9e0874060cc9efbf61dea4	master
Provide link when user without permissions tries to create project	repos/phabricator/deployment!32	aklapper	T165192verboseProjCreationPermError	wmf/stable
Update mathjax-node and texvcjs	repos/mediawiki/services/mathoid!3	physikerwelt	T137787	main
Call git submodule sync before git submodule update	repos/releng/scap!130	dancy	master-c5d7	master
Phabricator: add override for the browser time zone conflict message	repos/phabricator/deployment!6	dannys712	dannys712-wmf/stable-patch-89495	wmf/stable

Revisions and Commits

Unknown Object (Diffusion Commit)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open	Feature	None	T63779 Add system logs to logstash (tracking)
		Resolved		bd808	T1 Get puppet runs into logstash

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

• emailbot mentioned this in Unknown Object (Task).Dec 18 2015, 6:37 PM

• ZOKIDIN subscribed.Jan 30 2016, 5:38 AM

This comment was removed by Legoktm.

• santhosh mentioned this in T128162: Resuming a re-translated deleted translation will cause restoring the sections from old deleted translation.Feb 26 2016, 5:10 AM

Change 274054 had a related patch set uploaded (by Xqt):
Remove imagetransfer.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

gerritbot added a project: Patch-For-Review.Mar 1 2016, 4:42 AM

• ZOKIDIN added a comment.Mar 1 2016, 1:33 PM

This comment was removed by Legoktm.

Change 291859 had a related patch set uploaded (by Xqt):
Remove casechecker .py from "T001/T003" testing (print function)

https://gerrit.wikimedia.org/r/291859

valhallasw mentioned this in T136633: Support multi-task commits.May 31 2016, 6:31 PM

valhallasw mentioned this in T137311: ReleaseTaggerBot broke on 2 June.Jun 8 2016, 2:33 PM

• mmodell mentioned this in rOPUP28a86d08f542: Avoid breaking full phabricator URLs.Jun 17 2016, 6:05 PM

Paladox mentioned this in T85002: Gerrit bug search not (naively) working with Phabricator tasks.Jul 25 2016, 4:44 PM

Paladox mentioned this in rOPUPb68924b2979b: Avoid breaking full phabricator URLs.Jul 28 2016, 12:11 PM

Paladox mentioned this in rOPUPfcaae0d53d72: Gerrit: Avoid breaking full phabricator URLs.Jul 31 2016, 6:17 PM

Paladox mentioned this in rOPUPe99ffb93b141: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 10:07 AM

Paladox mentioned this in rOPUP1b010e47fcfd: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:51 PM

Paladox mentioned this in rOPUPc0f0b85eac3f: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:56 PM

Change 274054 abandoned by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 303408 had a related patch set uploaded (by Xqt):
Remove casechecker.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/303408

Change 291859 abandoned by Xqt:
Remove casechecker .py from "T001/T003" testing (print function)

Reason:
duplicate of I1d23ab55e

https://gerrit.wikimedia.org/r/291859

Change 303408 merged by jenkins-bot:
Remove casechecker.py from ignoring T001/T003 (print function)

https://gerrit.wikimedia.org/r/303408

Paladox mentioned this in T76459: Allow links to phabricator comments from gerrit messages..Aug 12 2016, 5:32 PM

Paladox mentioned this in T75997: Full Phabricator links are being expanded weirdly in gerrit.Aug 24 2016, 10:44 AM

Paladox mentioned this in rOPUP54c73ebb696b: Fix phabricator expanding links.Aug 24 2016, 11:02 AM

Paladox mentioned this in rOPUP27a89db35696: Fix phabricator expanding links.Aug 24 2016, 11:08 AM

Paladox mentioned this in T144098: Create a bot similar to gerritbot for phabricator differential.Aug 27 2016, 6:22 PM

Change 274054 restored by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 308753 had a related patch set (by Paladox) published:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Paladox mentioned this in T146843: Re-evaluate how we implement phabricator's search engine.Oct 2 2016, 11:07 AM

Change 308753 merged by Dzahn:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Change 330972 had a related patch set (by Paladox) published:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Change 330972 abandoned by Paladox:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Tgr added a parent task: T63779: Add system logs to logstash (tracking).Feb 11 2017, 1:37 AM

Change 274054 merged by jenkins-bot:
Remove imagetransfer.py from T001/T003 testing

https://gerrit.wikimedia.org/r/274054

greg moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Feb 14 2017, 6:52 PM

Fz-29 mentioned this in T164623: Weekly Reports : Add a "hierarchy" type to the Cargo extension [GSoC-2017].Jun 24 2017, 9:56 PM

Awesome_Aasim subscribed.Sep 17 2017, 5:03 AM

Awesome_Aasim unsubscribed.

• IZoid added a subtask: T49578: Score should output SVG.Dec 9 2017, 6:34 PM

Ebe123 removed a subtask: T49578: Score should output SVG.Dec 9 2017, 6:37 PM

• Gilles mentioned this in T165272: Review research on performance perception.Jan 3 2018, 2:15 PM

Change 407865 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Modify access rules+test+test

https://gerrit.wikimedia.org/r/407865

Change 434719 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

https://gerrit.wikimedia.org/r/434719

Diffusion mentioned this in rGERRITDEPLOY0e03b79ba4a7: Fix NPE in ChangeDetailFactory for loadMessages isPatchVisible.Jun 7 2018, 6:55 PM

Diffusion mentioned this in rGERRITDEPLOY0ec0d19c468c: ListTags: Fix ref in TagInfo for signed/annotated tags.Jun 7 2018, 7:13 PM

ClemFlip mentioned this in T143326: Invent automatic segmentation in translation units without <translate> tags.Dec 13 2018, 1:13 PM

Ssomansh mentioned this in T219442: Google Summer of Code(2019): Design and Development of a tool to track developer's activity.Mar 27 2019, 7:36 PM

This is strange. Even the first task is still unresolved. Can someone please guide me how we go about in the phabricator about backlog clearing?

@Capankajsmilyo: Please move off-topic general questions to mw:Talk:Phabricator/Help and read mw:Bug management/Development prioritization

Legoktm subscribed.Jun 25 2019, 7:27 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:34 PM

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 20 2020, 1:29 PM

DVrandecic mentioned this in T261474: First set of builtin functions.Aug 27 2020, 10:16 PM

In T1#1541803, @bd808 wrote:

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

Five years later, is that still the take / situation? (If it is, feel free to update the task status again.)

@Aklapper TL;DR yes, as it depends on T213902

Thanks. Added subtask.

• IN mentioned this in T270021: 404 when attempting phabricator tasks with lowercase t.Dec 13 2020, 9:27 AM

jbond subscribed.Jan 25 2021, 10:33 AM

lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

Maintenance_bot added a project: observability.Jul 12 2021, 2:49 AM

bd808 mentioned this in Blog Post: Shrinking the tasks backlog.Jul 14 2021, 6:29 PM

lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:10 AM

BrandonXLF mentioned this in T288367: More menu items are too large in modern Vector skin.Aug 6 2021, 6:48 PM

lmata edited projects, added Observability-Logging; removed observability, SRE Observability.Aug 9 2021, 12:18 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 12:45 AM

lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 1:00 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 1:46 AM

lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:53 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:49 AM

DVrandecic mentioned this in T292892: Function model: Introduce a Union generic type.Oct 9 2021, 3:01 AM

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

after the last change above, we now have puppet report data going into logstash. I haven't played with it much and im a elastic noob but the following is an example showing when a hosts where running a specific puppet git sha1 https://logstash.wikimedia.org/goto/473ad9da22eb1255d4168658f42a7ed5

As said i have not played with the data much but here is an example of what data gets sent on each run.

In T1#6599767, @Volans wrote:

@Aklapper TL;DR yes, as it depends on T213902

i think this comment referees to puppet diff's however the implementation doesn't add the diffs and im not sure if we would want them. As such i think we can finally close this task down and will boldly do so

Restricted Application added a project: User-bd808. · View Herald TranscriptNov 4 2021, 8:51 PM

Aklapper removed a subtask: T213902: Implement sensitive logstash access control.Nov 5 2021, 2:59 PM

herron awarded a token.Nov 5 2021, 3:34 PM

SunAfterRain mentioned this in T296128: Wikipedia app process interwiki link(s) error..Nov 20 2021, 12:29 PM

CactiStaccingCrane awarded a token.Dec 13 2021, 11:48 AM

MarcoAurelio mentioned this in T298041: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured".Dec 20 2021, 5:42 PM

bd808 mentioned this in T325381: stashbot: timestamps in SAL messages can be interpreted as ticket links.Dec 16 2022, 6:29 PM

Change 875401 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] phabricator: update pattern to support old tickets like T1

https://gerrit.wikimedia.org/r/875401

Change 875401 merged by Jbond:

[operations/puppet@production] phabricator: update pattern to support old tickets like T1