Page MenuHomePhabricator

Deploy phatality into kibana
Closed, ResolvedPublic

Description

Phatality is a kibana plugin that streamlines the workflow that Release-Engineering-Team (and others) use for tracking errors in Phabricator. Specifically, the plugin adds a link from an error log entry in Kibana directly to the Wikimedia-production-error form with error details pre-filled.

Here is an example URL for the submit feature:

https://phabricator.wikimedia.org/maniphest/task/edit/form/46/?title=Call+to+a+member+function+getName()+on+a+non-object+(null)&Description=Error+Details+Here

And the UI is currently just the bare-minimum, it looks like this:

@fgiunchedi : It appears that kibana plugins can be deployed with scap. Is this something that you would be willing to help with or perhaps just be around in case I screw something up? Are there any special deployment considerations for kibana that I should be aware of? Also are there any plans for upgrading kibana in the near future? It looks like I will have to keep the version of phatality in sync precisely with the version of kibana or kibana will refuse to load the plugin.

Details

Related Gerrit Patches:

Revisions and Commits

Event Timeline

mmodell created this task.Aug 19 2019, 8:53 PM
mmodell updated the task description. (Show Details)Aug 19 2019, 10:22 PM

@mmodell for sure, me or @herron or @colewhite can help with that, what timeline were you looking at for the deployment ? I did a quick search on gerrit/phab for phatality but couldn't find the source, do you have a pointer to that? Yes scap could be an option but I don't think we have "prior art" for scap + kibana plugins. re: upgrades yes there are plans to move to the logging pipeline to Elastic stack 7 and above here: T183281: [epic] ELK upgrade to 6.x (elasticsearch, kibana, logstash) although no timeline yet

@fgiunchedi WIP commit: https://gerrit.wikimedia.org/r/#/c/releng/phatality/+/531047/

It's not ready for deployment yet but I intend to have it ready by next week, perhaps the week after. It's not quite ready for code review yet but I'll let you know when it is in a more presentable state.

mmodell triaged this task as Medium priority.Aug 20 2019, 10:24 AM

Change 531047 had a related patch set uploaded (by 20after4; owner: 20after4):
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

@fgiunchedi: This is ready to go whenever you have some time to look it over.

@mmodell ok! I hope to be able to take a look this week

Awesome, thank you @fgiunchedi!

mmodell updated the task description. (Show Details)Sep 4 2019, 4:54 PM
hashar added a subscriber: hashar.

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip

So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

The way I deployed it in beta is fairly straightforward - I copied the zip file with the plugin code over (with scp) and just ran kibana-plugin install file:///path/to/phatality.zip
So I'm thinking to deploy with scap we can just scap the zip file over and use a promote script in scap to install the zip into kibana. Does that sound ok to you @fgiunchedi?

Sounds good to me! I take it we'll also need some sudo changes to let scap run kibana-plugin

@fgiunchedi Indeed, either sudo or maybe we can deploy to a directory owned by the scap deploy user and symlink to the deployment directory from kibana's plugins directory?

Change 537240 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

@fgiunchedi ^ this might be one way to avoid needing sudo.

Thanks @mmodell ! I think on balance I'm leaning towards sudo + kibana-plugin install, reason being is that kibana-plugin is the recommended method upstream. cc @colewhite @herron

Change 537240 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/537240

fgiunchedi moved this task from Inbox to Up next on the observability board.Sep 24 2019, 9:57 AM

Reverted the above at https://gerrit.wikimedia.org/r/c/operations/puppet/+/538848 because a couple of things were not working yet, see comments in the review

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

@fgiunchedi: do you know what hosts are running kibana? I need to set up a scap target list for the deployment repo.

Yes you can use conftool data for scap::dsh::groups, it'd be cluster logstash and service kibana

Change 531047 merged by 20after4:
[releng/phatality@master] Phatality plugin for kibana

https://gerrit.wikimedia.org/r/531047

Change 538858 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

Change 538858 merged by Filippo Giunchedi:
[operations/puppet@production] Set up scap target for deploying the phatality plugin into kibana

https://gerrit.wikimedia.org/r/538858

Change 539054 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Change 539054 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add phatality dsh groups

https://gerrit.wikimedia.org/r/539054

Change 539104 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

Change 539104 merged by Jbond:
[operations/puppet@production] kibana::phatality: fix sudo line

https://gerrit.wikimedia.org/r/539104

Should be all deployed now, ready for another round of plugin-install

mmodell closed this task as Resolved.Sep 26 2019, 7:32 AM
mmodell moved this task from Ready to Completed on the Release-Engineering-Team-TODO (201909) board.

Lovely thank you :-]

antoine-approve

Thank you very much @fgiunchedi for helping with this! The plugin is working!! The only thing that remains is this: I found out that kibana-plugin install will not overwrite the plugin - it requires kibana-plugin remove phatality first followed by kibana-plugin install the new version. So in order to deploy future updates, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539191/ is needed. Otherwise this is all done! Thanks again.

So the latest changes to the sudo rules apparently don't work...