Get puppet runs into logstash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bd808
	Feb 5 2014, 11:17 PM

Details

Other Assignee: jbond
Reference: fl4

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
puppet-agent: remove --show_diff from scheduled puppet-run script	operations/puppet	production	+0 -1
P:rsyslog: ship puppetmaster logs to kafka	operations/puppet	production	+2 -1
puppetmaster: enable logstash reports	operations/puppet	production	+6 -0
Test: Do not merge	mediawiki/core	master	+0 -0

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
lint: stop finding '*.inc' files	repos/releng/scap!1049	hashar	php83-linter	master
api-gateway: display logs-api openapi spec too	repos/cloud/toolforge/toolforge-deploy!1035	dcaro	enable_logs_api	main
d/changelog: bump to 16.1.24	repos/cloud/toolforge/jobs-cli!136	dcaro	bump_jobs-cli	main
settings: allow to configure the loki url per-deployment	repos/cloud/toolforge/logs-api!2	dcaro	configure_loki_url	main
api-gateway: bump to 0.0.81-20251016082112-1c4f5a64	repos/cloud/toolforge/toolforge-deploy!1001	group_203_bot_f4d95069bb2675e4ce1fff090c1c1620	bump_api-gateway	main
[jobs-api, jobs-cli] test for job logs	repos/cloud/toolforge/toolforge-deploy!924	raymond-ndibe	test_for_jobs_logs	main
logs: use logs-api for logs	repos/cloud/toolforge/jobs-cli!121	dcaro	use_logs_api	main
logs_api: add the option to enable logs-api	repos/cloud/toolforge/api-gateway!75	dcaro	add_logs_api	main
global: first commit	repos/cloud/toolforge/logs-api!1	dcaro	first_commit	main
Parsoid: Add trailing slash; ensures patched Parsoid is used	repos/test-platform/catalyst/ci-charts!88	thcipriani	main-Icd9d06bc89d394e082c85afab4b600120700ba83	main
Add trailing slash to ensure patched version of Parsoid is used	repos/test-platform/catalyst/patchdemo!184	cscott	fix-parsoid	main
Add DAGs for video metric aggregations + update test var JSONs	repos/data-engineering/airflow-dags!1488	andrewtavis-wmde	wlb-video-metric-dags	main
Update namespace of TemplateParser	repos/design-systems/CodexExample!13	cscott	fix-namespace	main
Update function-schemata sub-module to HEAD (5d98a1f)	repos/abstract-wiki/wikifunctions/function-orchestrator!256	jforrester	sync-function-schemata	main
Update function-schemata sub-module to HEAD (5d98a1f)	repos/abstract-wiki/wikifunctions/function-evaluator!276	jforrester	sync-function-schemata	main
Update function-schemata sub-module to HEAD (5d98a1f)	repos/abstract-wiki/wikifunctions/wikilambda-cli!56	jforrester	sync-function-schemata	main
definitions: Add Z1952/bax-bamu, Z1953/xon, Z1954/cdo-hant and Z1955/cdo-latn ZNaturalLanguages	repos/abstract-wiki/wikifunctions/function-schemata!177	jforrester	languages-bax-bamu-xon	main
Disable publishing notifications for new repositories by default	repos/phabricator/phabricator!75	aklapper	T143162disablePublishing	wmf/stable
jenkins-rel: fix templating for Jinja 2.x	repos/releng/jenkins-deploy!74	hashar	fix-template	master
jenkins-rel: do not set keepUndefinedParameters	repos/releng/jenkins-deploy!73	hashar	no-keepUndefinedParameters	master

Revisions and Commits

Unknown Object (Diffusion Commit)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open	Feature	None	T63779 Add system logs to logstash (tracking)
		Resolved		• bd808	T1 Get puppet runs into logstash

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

gerritbot added a project: Patch-For-Review.Mar 1 2016, 4:42 AM

• ZOKIDIN added a comment.Mar 1 2016, 1:33 PM

This comment was removed by Legoktm.

Change 291859 had a related patch set uploaded (by Xqt):
Remove casechecker .py from "T001/T003" testing (print function)

https://gerrit.wikimedia.org/r/291859

valhallasw mentioned this in T136633: Support multi-task commits.May 31 2016, 6:31 PM

valhallasw mentioned this in T137311: ReleaseTaggerBot broke on 2 June.Jun 8 2016, 2:33 PM

• mmodell mentioned this in rOPUP28a86d08f542: Avoid breaking full phabricator URLs.Jun 17 2016, 6:05 PM

Paladox mentioned this in T85002: Gerrit bug search not (naively) working with Phabricator tasks.Jul 25 2016, 4:44 PM

Paladox mentioned this in rOPUPb68924b2979b: Avoid breaking full phabricator URLs.Jul 28 2016, 12:11 PM

Paladox mentioned this in rOPUPfcaae0d53d72: Gerrit: Avoid breaking full phabricator URLs.Jul 31 2016, 6:17 PM

Paladox mentioned this in rOPUPe99ffb93b141: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 10:07 AM

Paladox mentioned this in rOPUP1b010e47fcfd: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:51 PM

Paladox mentioned this in rOPUPc0f0b85eac3f: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:56 PM

Change 274054 abandoned by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 303408 had a related patch set uploaded (by Xqt):
Remove casechecker.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/303408

Change 291859 abandoned by Xqt:
Remove casechecker .py from "T001/T003" testing (print function)

Reason:
duplicate of I1d23ab55e

https://gerrit.wikimedia.org/r/291859

Change 303408 merged by jenkins-bot:
Remove casechecker.py from ignoring T001/T003 (print function)

https://gerrit.wikimedia.org/r/303408

Paladox mentioned this in T76459: Allow links to phabricator comments from gerrit messages..Aug 12 2016, 5:32 PM

Paladox mentioned this in T75997: Full Phabricator links are being expanded weirdly in gerrit.Aug 24 2016, 10:44 AM

Paladox mentioned this in rOPUP54c73ebb696b: Fix phabricator expanding links.Aug 24 2016, 11:02 AM

Paladox mentioned this in rOPUP27a89db35696: Fix phabricator expanding links.Aug 24 2016, 11:08 AM

Paladox mentioned this in T144098: Create a bot similar to gerritbot for phabricator differential.Aug 27 2016, 6:22 PM

Change 274054 restored by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

Change 308753 had a related patch set (by Paladox) published:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Paladox mentioned this in T146843: Re-evaluate how we implement phabricator's search engine.Oct 2 2016, 11:07 AM

Change 308753 merged by Dzahn:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Change 330972 had a related patch set (by Paladox) published:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Change 330972 abandoned by Paladox:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Tgr added a parent task: T63779: Add system logs to logstash (tracking).Feb 11 2017, 1:37 AM

Change 274054 merged by jenkins-bot:
Remove imagetransfer.py from T001/T003 testing

https://gerrit.wikimedia.org/r/274054

greg moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Feb 14 2017, 6:52 PM

Fz-29 mentioned this in T164623: Weekly Reports : Add a "hierarchy" type to the Cargo extension [GSoC-2017].Jun 24 2017, 9:56 PM

Awesome_Aasim subscribed.Sep 17 2017, 5:03 AM

Awesome_Aasim unsubscribed.

• IZoid added a subtask: T49578: Score should output SVG.Dec 9 2017, 6:34 PM

Ebe123 removed a subtask: T49578: Score should output SVG.Dec 9 2017, 6:37 PM

• Gilles mentioned this in T165272: Review research on performance perception.Jan 3 2018, 2:15 PM

Change 407865 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Modify access rules+test+test

https://gerrit.wikimedia.org/r/407865

Change 434719 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

https://gerrit.wikimedia.org/r/434719

Diffusion mentioned this in rGERRITDEPLOY0e03b79ba4a7: Fix NPE in ChangeDetailFactory for loadMessages isPatchVisible.Jun 7 2018, 6:55 PM

Diffusion mentioned this in rGERRITDEPLOY0ec0d19c468c: ListTags: Fix ref in TagInfo for signed/annotated tags.Jun 7 2018, 7:13 PM

ClemFlip mentioned this in T143326: Invent automatic segmentation in translation units without <translate> tags.Dec 13 2018, 1:13 PM

Ssomansh mentioned this in T219442: Google Summer of Code(2019): Design and Development of a tool to track developer's activity.Mar 27 2019, 7:36 PM

This is strange. Even the first task is still unresolved. Can someone please guide me how we go about in the phabricator about backlog clearing?

@Capankajsmilyo: Please move off-topic general questions to mw:Talk:Phabricator/Help and read mw:Bug management/Development prioritization

Legoktm subscribed.Jun 25 2019, 7:27 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:34 PM

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 20 2020, 1:29 PM

DVrandecic mentioned this in T261474: First set of builtin functions.Aug 27 2020, 10:16 PM

In T1#1541803, @bd808 wrote:

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

Five years later, is that still the take / situation? (If it is, feel free to update the task status again.)

@Aklapper TL;DR yes, as it depends on T213902

Thanks. Added subtask.

• IN mentioned this in T270021: 404 when attempting phabricator tasks with lowercase t.Dec 13 2020, 9:27 AM

jbond subscribed.Jan 25 2021, 10:33 AM

• lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

Maintenance_bot added a project: observability.Jul 12 2021, 2:49 AM

• bd808 mentioned this in Blog Post: Shrinking the tasks backlog.Jul 14 2021, 6:29 PM

• lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:10 AM

BrandonXLF mentioned this in T288367: More menu items are too large in modern Vector skin.Aug 6 2021, 6:48 PM

• lmata edited projects, added Observability-Logging; removed observability, SRE Observability.Aug 9 2021, 12:18 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 12:45 AM

• lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 1:00 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 1:46 AM

• lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:53 AM

Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:49 AM

DVrandecic mentioned this in T292892: Function model: Introduce a Union generic type.Oct 9 2021, 3:01 AM

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

after the last change above, we now have puppet report data going into logstash. I haven't played with it much and im a elastic noob but the following is an example showing when a hosts where running a specific puppet git sha1 https://logstash.wikimedia.org/goto/473ad9da22eb1255d4168658f42a7ed5

As said i have not played with the data much but here is an example of what data gets sent on each run.

In T1#6599767, @Volans wrote:

@Aklapper TL;DR yes, as it depends on T213902

i think this comment referees to puppet diff's however the implementation doesn't add the diffs and im not sure if we would want them. As such i think we can finally close this task down and will boldly do so

Restricted Application added a project: User-bd808. · View Herald TranscriptNov 4 2021, 8:51 PM

Aklapper removed a subtask: T213902: Implement sensitive logstash access control.Nov 5 2021, 2:59 PM

herron awarded a token.Nov 5 2021, 3:34 PM

SunAfterRain mentioned this in T296128: Wikipedia app process interwiki link(s) error..Nov 20 2021, 12:29 PM

CactiStaccingCrane awarded a token.Dec 13 2021, 11:48 AM

MarcoAurelio mentioned this in T298041: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured".Dec 20 2021, 5:42 PM

• bd808 mentioned this in T325381: stashbot: timestamps in SAL messages can be interpreted as ticket links.Dec 16 2022, 6:29 PM

Change 875401 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] phabricator: update pattern to support old tickets like T1

https://gerrit.wikimedia.org/r/875401

Change 875401 merged by Jbond:

[operations/puppet@production] phabricator: update pattern to support old tickets like T1