Page MenuHomePhabricator
Create Task
Maniphest T1

Get puppet runs into logstash
Closed, ResolvedPublic
Actions

Details

Other Assignee
jbond
Reference
fl4
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
puppet-agent: remove --show_diff from scheduled puppet-run scriptHerronoperations/puppetproduction+0 -1
P:rsyslog: ship puppetmaster logs to kafkaJbondoperations/puppetproduction+2 -1
puppetmaster: enable logstash reportsJbondoperations/puppetproduction+6 -0
Test: Do not mergePaladoxmediawiki/coremaster+0 -0
Customize query in gerrit

Revisions and Commits

Unknown Object (Diffusion Commit)

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenFeatureNoneT63779 Add system logs to logstash (tracking)
 Resolvedbd808T1 Get puppet runs into logstash

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Paladox mentioned this in rOPUPb68924b2979b: Avoid breaking full phabricator URLs.Jul 28 2016, 12:11 PM
Paladox mentioned this in rOPUPfcaae0d53d72: Gerrit: Avoid breaking full phabricator URLs.Jul 31 2016, 6:17 PM
Paladox mentioned this in rOPUPe99ffb93b141: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 10:07 AM
Paladox mentioned this in rOPUP1b010e47fcfd: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:51 PM
Paladox mentioned this in rOPUPc0f0b85eac3f: Gerrit: Avoid breaking full phabricator URLs.Aug 1 2016, 5:56 PM
gerritbot added a comment.Aug 7 2016, 8:20 AM

Change 274054 abandoned by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

gerritbot added a comment.Aug 7 2016, 8:21 AM

Change 303408 had a related patch set uploaded (by Xqt):
Remove casechecker.py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/303408

gerritbot added a comment.Aug 7 2016, 9:53 AM

Change 291859 abandoned by Xqt:
Remove casechecker .py from "T001/T003" testing (print function)

Reason:
duplicate of I1d23ab

https://gerrit.wikimedia.org/r/291859

gerritbot added a comment.Aug 7 2016, 3:32 PM

Change 303408 merged by jenkins-bot:
Remove casechecker.py from ignoring T001/T003 (print function)

https://gerrit.wikimedia.org/r/303408

Paladox mentioned this in T76459: Allow links to phabricator comments from gerrit messages..Aug 12 2016, 5:32 PM
Paladox mentioned this in T75997: Full Phabricator links are being expanded weirdly in gerrit.Aug 24 2016, 10:44 AM
Paladox mentioned this in rOPUP54c73ebb696b: Fix phabricator expanding links.Aug 24 2016, 11:02 AM
Paladox mentioned this in rOPUP27a89db35696: Fix phabricator expanding links.Aug 24 2016, 11:08 AM
Paladox mentioned this in T144098: Create a bot similar to gerritbot for phabricator differential.Aug 27 2016, 6:22 PM
gerritbot added a comment.Sep 1 2016, 7:51 AM

Change 274054 restored by Xqt:
Remove casechecker .py from T001/T003 testing (print function)

https://gerrit.wikimedia.org/r/274054

gerritbot added a comment.Sep 6 2016, 3:34 PM

Change 308753 had a related patch set (by Paladox) published:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

Paladox mentioned this in T146843: Re-evaluate how we implement phabricator's search engine.Oct 2 2016, 11:07 AM
gerritbot added a comment.Jan 6 2017, 6:52 PM

Change 308753 merged by Dzahn:
Add support for searching gerrit using bug:T1

https://gerrit.wikimedia.org/r/308753

gerritbot added a comment.Jan 6 2017, 7:00 PM

Change 330972 had a related patch set (by Paladox) published:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

gerritbot added a comment.Jan 6 2017, 7:01 PM

Change 330972 abandoned by Paladox:
Test: Do not merge

https://gerrit.wikimedia.org/r/330972

Tgr added a parent task: T63779: Add system logs to logstash (tracking).Feb 11 2017, 1:37 AM
gerritbot added a comment.Feb 13 2017, 5:11 AM

Change 274054 merged by jenkins-bot:
Remove imagetransfer.py from T001/T003 testing

https://gerrit.wikimedia.org/r/274054

greg moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Feb 14 2017, 6:52 PM
Fz-29 mentioned this in T164623: Weekly Reports : Add a "hierarchy" type to the Cargo extension [GSoC-2017].Jun 24 2017, 9:56 PM
Awesome_Aasim subscribed.Sep 17 2017, 5:03 AM
Awesome_Aasim unsubscribed.
• IZoid added a subtask: T49578: Score should output SVG.Dec 9 2017, 6:34 PM
Ebe123 removed a subtask: T49578: Score should output SVG.Dec 9 2017, 6:37 PM
• Gilles mentioned this in T165272: Review research on performance perception.Jan 3 2018, 2:15 PM
gerritbot added a comment.Feb 3 2018, 12:48 AM

Change 407865 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Modify access rules+test+test

https://gerrit.wikimedia.org/r/407865

gerritbot added a comment.May 23 2018, 3:15 PM

Change 434719 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

https://gerrit.wikimedia.org/r/434719

Diffusion mentioned this in rGERRITDEPLOY0e03b79ba4a7: Fix NPE in ChangeDetailFactory for loadMessages isPatchVisible.Jun 7 2018, 6:55 PM
Diffusion mentioned this in rGERRITDEPLOY0ec0d19c468c: ListTags: Fix ref in TagInfo for signed/annotated tags.Jun 7 2018, 7:13 PM
ClemFlip mentioned this in T143326: Invent automatic segmentation in translation units without <translate> tags.Dec 13 2018, 1:13 PM
Ssomansh mentioned this in T219442: Google Summer of Code(2019): Design and Development of a tool to track developer's activity.Mar 27 2019, 7:36 PM
Capankajsmilyo subscribed.Apr 10 2019, 10:09 AM

This is strange. Even the first task is still unresolved. Can someone please guide me how we go about in the phabricator about backlog clearing?

Aklapper added a comment.Apr 10 2019, 12:32 PM

@Capankajsmilyo: Please move off-topic general questions to mw:Talk:Phabricator/Help and read mw:Bug management/Development prioritization

Legoktm subscribed.Jun 25 2019, 7:27 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:34 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 20 2020, 1:29 PM
DVrandecic mentioned this in T261474: First set of builtin functions.Aug 27 2020, 10:16 PM
Aklapper changed the task status from Stalled to Open.Nov 3 2020, 10:38 AM
In T1#1541803, @bd808 wrote:

We have the technology to implement this, but WMF TechOps has valid concerns about sensitive data that may be contained in the puppet reports themselves. As one example, file diffs can show passwords that are generally protected via file permissions from reading by all users with shell access to the host.

Five years later, is that still the take / situation? (If it is, feel free to update the task status again.)

Volans subscribed.Nov 3 2020, 11:07 AM

@Aklapper TL;DR yes, as it depends on T213902

Aklapper added a subtask: T213902: Implement sensitive logstash access control.Nov 3 2020, 12:27 PM

Thanks. Added subtask.

• IN mentioned this in T270021: 404 when attempting phabricator tasks with lowercase t.Dec 13 2020, 9:27 AM
jbond subscribed.Jan 25 2021, 10:33 AM
• lmata edited projects, added: SRE Observability; removed: observability.Jul 12 2021, 2:21 AM
Maintenance_bot added a project: observability.Jul 12 2021, 2:49 AM
bd808 mentioned this in Blog Post: Shrinking the tasks backlog.Jul 14 2021, 6:29 PM
• lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:10 AM
BrandonXLF mentioned this in T288367: More menu items are too large in modern Vector skin.Aug 6 2021, 6:48 PM
• lmata edited projects, added: Observability-Logging; removed: observability, SRE Observability.Aug 9 2021, 12:18 AM
Maintenance_bot edited projects, added: SRE Observability; removed: Observability-Logging.Aug 9 2021, 12:45 AM
• lmata edited projects, added: Observability-Logging; removed: SRE Observability.Aug 9 2021, 1:00 AM
Maintenance_bot edited projects, added: SRE Observability; removed: Observability-Logging.Aug 9 2021, 1:46 AM
• lmata edited projects, added: Observability-Logging; removed: SRE Observability.Aug 9 2021, 2:53 AM
Maintenance_bot edited projects, added: SRE Observability; removed: Observability-Logging.Aug 9 2021, 3:49 AM
DVrandecic mentioned this in T292892: Function model: Introduce a Union generic type.Oct 9 2021, 3:01 AM
gerritbot added a comment.Oct 27 2021, 1:07 PM

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

gerritbot added a comment.Oct 27 2021, 1:17 PM

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

gerritbot added a comment.Nov 2 2021, 2:19 PM

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

gerritbot added a comment.Nov 4 2021, 5:00 PM

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

jbond closed this task as Resolved.EditedNov 4 2021, 8:51 PM
jbond assigned this task to bd808.
jbond updated Other Assignee, added: jbond.

after the last change above, we now have puppet report data going into logstash. I haven't played with it much and im a elastic noob but the following is an example showing when a hosts where running a specific puppet git sha1 https://logstash.wikimedia.org/goto/473ad9da22eb1255d4168658f42a7ed5

As said i have not played with the data much but here is an example of what data gets sent on each run.

In T1#6599767, @Volans wrote:

@Aklapper TL;DR yes, as it depends on T213902

i think this comment referees to puppet diff's however the implementation doesn't add the diffs and im not sure if we would want them. As such i think we can finally close this task down and will boldly do so

Restricted Application added a project: User-bd808. · View Herald TranscriptNov 4 2021, 8:51 PM
Aklapper removed a subtask: T213902: Implement sensitive logstash access control.Nov 5 2021, 2:59 PM
• herron awarded a token.Nov 5 2021, 3:34 PM
SunAfterRain mentioned this in T296128: Wikipedia app process interwiki link(s) error..Nov 20 2021, 12:29 PM
CactiStaccingCrane awarded a token.Dec 13 2021, 11:48 AM
MarcoAurelio mentioned this in T298041: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured".Dec 20 2021, 5:42 PM
bd808 mentioned this in T325381: stashbot: timestamps in SAL messages can be interpreted as ticket links.Dec 16 2022, 6:29 PM
gerritbot added a comment.Jan 4 2023, 6:01 PM

Change 875401 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] phabricator: update pattern to support old tickets like T1

https://gerrit.wikimedia.org/r/875401

gerritbot added a comment.Jan 4 2023, 6:04 PM

Change 875401 merged by Jbond:

[operations/puppet@production] phabricator: update pattern to support old tickets like T1

https://gerrit.wikimedia.org/r/875401

gerritbot added a comment.May 8 2023, 6:38 AM
This comment was removed by brennen.
bd808 mentioned this in rLTSTd154406f2042: Lookup and echo Phabricator objects named in messages.May 25 2023, 10:45 PM
CodeReviewBot added a comment.Jun 21 2023, 3:36 PM
This comment was removed by brennen.
brennen subscribed.Jun 23 2023, 6:42 PM
gerritbot added a comment.Apr 2 2024, 2:09 PM

Change #434719 abandoned by Herron:

[operations/puppet@production] puppet-agent: remove --show_diff from scheduled puppet-run script

Reason:

spring cleaning -- stale patch

https://gerrit.wikimedia.org/r/434719

Pppery mentioned this in T382958: History merge should support timestamps on both ends.Jan 3 2025, 5:37 PM
Pppery removed a project: Patch-For-Review.
dcaro mentioned this in T400223: Investigate daily disconnections of IRC bots hosted in Toolforge.Aug 5 2025, 12:07 PM
Bugreporter mentioned this in T406125: Add substitute button.Oct 6 2025, 4:03 PM
CodeReviewBot added a project: Patch-For-Review.Nov 20 2025, 2:56 PM

deerbee opened https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/2

Extend matching pattern

• UUMFOfffce reopened this task as Open.Jul 4 2026, 4:00 AM
• UUMFOfffce raised the priority of this task from Medium to Unbreak Now!.
• UUMFOfffce edited subscribers, added: • UUMFOfffce; removed: Volans.
This comment was removed by Aklapper.
Johannnes89 closed this task as Resolved.Jul 4 2026, 6:28 AM
Johannnes89 lowered the priority of this task from Unbreak Now! to Medium.
Johannnes89 edited subscribers, added: Volans; removed: • UUMFOfffce.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits