Deploy phatality into kibana
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• mmodell
	Aug 19 2019, 8:53 PM

Description

Phatality is a kibana plugin that streamlines the workflow that Release-Engineering-Team (and others) use for tracking errors in Phabricator. Specifically, the plugin adds a link from an error log entry in Kibana directly to the Wikimedia-production-error form with error details pre-filled.

Here is an example URL for the submit feature:

https://phabricator.wikimedia.org/maniphest/task/edit/form/46/?title=Call+to+a+member+function+getName()+on+a+non-object+(null)&Description=Error+Details+Here

And the UI is currently just the bare-minimum, it looks like this:

@fgiunchedi : It appears that kibana plugins can be deployed with scap. Is this something that you would be willing to help with or perhaps just be around in case I screw something up? Are there any special deployment considerations for kibana that I should be aware of? Also are there any plans for upgrading kibana in the near future? It looks like I will have to keep the version of phatality in sync precisely with the version of kibana or kibana will refuse to load the plugin.

Details

Subject	Repo	Branch	Lines +/-
kibana::phatality: fix sudo line	operations/puppet	production	+1 -1
hieradata: add phatality dsh groups	operations/puppet	production	+3 -0
Set up scap target for deploying the phatality plugin into kibana	operations/puppet	production	+26 -1
Phatality plugin for kibana	releng/phatality	master	+3 K -0
Set up scap target for deploying the phatality plugin into kibana	operations/puppet	production	+24 -0

Customize query in gerrit

Revisions and Commits

rPHAT Phatality
	rPHAT05d8a7d4b901 Phatality plugin for kibana

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Feature	None	T70820 Automatically log fatals/exceptions into Phabricator with stack traces
Resolved		• mmodell	T185155 Improve error reporting / integration between Kibana and Phabricator
Resolved		• mmodell	T230752 Deploy phatality into kibana

Event Timeline

• mmodell created this task.Aug 19 2019, 8:53 PM

• mmodell updated the task description. (Show Details)Aug 19 2019, 10:22 PM

@mmodell for sure, me or @herron or @colewhite can help with that, what timeline were you looking at for the deployment ? I did a quick search on gerrit/phab for phatality but couldn't find the source, do you have a pointer to that? Yes scap could be an option but I don't think we have "prior art" for scap + kibana plugins. re: upgrades yes there are plans to move to the logging pipeline to Elastic stack 7 and above here: T183281: [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) although no timeline yet

@fgiunchedi WIP commit: https://gerrit.wikimedia.org/r/#/c/releng/phatality/+/531047/

It's not ready for deployment yet but I intend to have it ready by next week, perhaps the week after. It's not quite ready for code review yet but I'll let you know when it is in a more presentable state.

• mmodell triaged this task as Medium priority.Aug 20 2019, 10:24 AM

fgiunchedi awarded a token.Aug 20 2019, 1:26 PM

greg moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201908) board.Aug 21 2019, 3:44 PM

Change 531047 had a related patch set uploaded (by 20after4; owner: 20after4):
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

gerritbot added a project: Patch-For-Review.Aug 29 2019, 1:38 AM

@fgiunchedi: This is ready to go whenever you have some time to look it over.

• mmodell moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201909) board.Sep 4 2019, 5:11 AM

@mmodell ok! I hope to be able to take a look this week

Awesome, thank you @fgiunchedi!

• mmodell updated the task description. (Show Details)Sep 4 2019, 4:54 PM

• mmodell added a project: User-MModell.Sep 4 2019, 11:58 PM

• mmodell moved this task from Backlog to Working on it on the User-MModell board.

• mmodell edited projects, added Release-Engineering-Team-TODO (201908); removed Release-Engineering-Team-TODO (201909).

• mmodell added a commit: rPHAT05d8a7d4b901: Phatality plugin for kibana.Sep 9 2019, 5:11 PM

Phatality is now deployed in beta, as can be seen: https://logstash-beta.wmflabs.org/app/kibana#/dashboard/Fatal-Monitor

Screenshot from 2019-09-10 12-23-42.png (316×414 px, 18 KB)

• mmodell edited projects, added Release-Engineering-Team-TODO (201909); removed Release-Engineering-Team-TODO (201908).Sep 11 2019, 5:12 PM

hashar awarded a token.Sep 11 2019, 5:13 PM

hashar subscribed.

Very nice @mmodell !

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip

So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

In T230752#5487593, @mmodell wrote:

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip

So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

Sounds good to me! I take it we'll also need some sudo changes to let scap run kibana-plugin

@fgiunchedi Indeed, either sudo or maybe we can deploy to a directory owned by the scap deploy user and symlink to the deployment directory from kibana's plugins directory?

Change 537240 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

@fgiunchedi ^ this might be one way to avoid needing sudo.

Thanks @mmodell ! I think on balance I'm leaning towards sudo + kibana-plugin install, reason being is that kibana-plugin is the recommended method upstream. cc @colewhite @herron

Change 537240 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

fgiunchedi moved this task from Inbox to Up next on the observability board.Sep 24 2019, 9:57 AM

Reverted the above at https://gerrit.wikimedia.org/r/c/operations/puppet/+/538848 because a couple of things were not working yet, see comments in the review

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

In T230752#5519026, @mmodell wrote:

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

Yes you can use conftool data for scap::dsh::groups, it'd be cluster logstash and service kibana

Change 531047 merged by 20after4:
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

Change 538858 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

• mmodell mentioned this in rPHAT6d07e46c360f: Phatality plugin for kibana.Sep 24 2019, 11:53 AM

Change 538858 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

Change 539054 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Change 539054 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2019, 8:10 AM

Change 539104 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

gerritbot added a project: Patch-For-Review.Sep 25 2019, 12:14 PM

Change 539104 merged by Jbond:
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2019, 1:10 PM

Should be all deployed now, ready for another round of plugin-install

• mmodell closed this task as Resolved.Sep 26 2019, 7:32 AM

• mmodell moved this task from Ready to Completed on the Release-Engineering-Team-TODO (201909) board.

Lovely thank you :-]

antoine-approve

Thank you very much @fgiunchedi for helping with this! The plugin is working!! The only thing that remains is this: I found out that kibana-plugin install will not overwrite the plugin - it requires kibana-plugin remove phatality first followed by kibana-plugin install the new version. So in order to deploy future updates, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539191/ is needed. Otherwise this is all done! Thanks again.

Aklapper awarded a token.Sep 26 2019, 8:34 AM

So the latest changes to the sudo rules apparently don't work...

	F30290939: Screenshot from 2019-09-10 12-23-42.png
	Sep 10 2019, 5:24 PM

Deploy phatality into kibanaClosed, ResolvedPublicActions