Page MenuHomePhabricator

Deploy phatality into kibana
Closed, ResolvedPublic

Description

Phatality is a kibana plugin that streamlines the workflow that Release-Engineering-Team (and others) use for tracking errors in Phabricator. Specifically, the plugin adds a link from an error log entry in Kibana directly to the Wikimedia-production-error form with error details pre-filled.

Here is an example URL for the submit feature:

https://phabricator.wikimedia.org/maniphest/task/edit/form/46/?title=Call+to+a+member+function+getName()+on+a+non-object+(null)&Description=Error+Details+Here

And the UI is currently just the bare-minimum, it looks like this:

@fgiunchedi : It appears that kibana plugins can be deployed with scap. Is this something that you would be willing to help with or perhaps just be around in case I screw something up? Are there any special deployment considerations for kibana that I should be aware of? Also are there any plans for upgrading kibana in the near future? It looks like I will have to keep the version of phatality in sync precisely with the version of kibana or kibana will refuse to load the plugin.

Revisions and Commits

Event Timeline

mmodell created this task.Aug 19 2019, 8:53 PM
mmodell updated the task description. (Show Details)Aug 19 2019, 10:22 PM

@mmodell for sure, me or @herron or @colewhite can help with that, what timeline were you looking at for the deployment ? I did a quick search on gerrit/phab for phatality but couldn't find the source, do you have a pointer to that? Yes scap could be an option but I don't think we have "prior art" for scap + kibana plugins. re: upgrades yes there are plans to move to the logging pipeline to Elastic stack 7 and above here: T183281: [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) although no timeline yet

@fgiunchedi WIP commit: https://gerrit.wikimedia.org/r/#/c/releng/phatality/+/531047/

It's not ready for deployment yet but I intend to have it ready by next week, perhaps the week after. It's not quite ready for code review yet but I'll let you know when it is in a more presentable state.

mmodell triaged this task as Medium priority.Aug 20 2019, 10:24 AM

Change 531047 had a related patch set uploaded (by 20after4; owner: 20after4):
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

@fgiunchedi: This is ready to go whenever you have some time to look it over.

@mmodell ok! I hope to be able to take a look this week

Awesome, thank you @fgiunchedi!

mmodell updated the task description. (Show Details)Sep 4 2019, 4:54 PM
hashar added a subscriber: hashar.

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip

So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip

So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

Sounds good to me! I take it we'll also need some sudo changes to let scap run kibana-plugin

@fgiunchedi Indeed, either sudo or maybe we can deploy to a directory owned by the scap deploy user and symlink to the deployment directory from kibana's plugins directory?

Change 537240 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

@fgiunchedi ^ this might be one way to avoid needing sudo.

Thanks @mmodell ! I think on balance I'm leaning towards sudo + kibana-plugin install, reason being is that kibana-plugin is the recommended method upstream. cc @colewhite @herron

Change 537240 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

fgiunchedi moved this task from Inbox to Up next on the observability board.Sep 24 2019, 9:57 AM

Reverted the above at https://gerrit.wikimedia.org/r/c/operations/puppet/+/538848 because a couple of things were not working yet, see comments in the review

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

Yes you can use conftool data for scap::dsh::groups, it'd be cluster logstash and service kibana

Change 531047 merged by 20after4:
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

Change 538858 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

Change 538858 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

Change 539054 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Change 539054 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Change 539104 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

Change 539104 merged by Jbond:
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

Should be all deployed now, ready for another round of plugin-install

mmodell closed this task as Resolved.Sep 26 2019, 7:32 AM
mmodell moved this task from Ready to Completed on the Release-Engineering-Team-TODO (201909) board.

Lovely thank you :-]

antoine-approve

Thank you very much @fgiunchedi for helping with this! The plugin is working!! The only thing that remains is this: I found out that kibana-plugin install will not overwrite the plugin - it requires kibana-plugin remove phatality first followed by kibana-plugin install the new version. So in order to deploy future updates, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539191/ is needed. Otherwise this is all done! Thanks again.

So the latest changes to the sudo rules apparently don't work...