Page MenuHomePhabricator

Empty kibana dashboards after logstash upgrade
Closed, ResolvedPublic

Description

After upgrading of kibana (T154473), my dashboads are empty. I see that the cause is that there are some columns that are no longer available- in particular, *.raw ones. I fixed what I could and now I have this: https://logstash.wikimedia.org/app/kibana#/dashboard/DBQuery

However, I was not able to fix the other "top" reports because the *.raw filelds are empty and I cannot use the non-raw version, for example:

raw.png (991×1 px, 78 KB)

Is there a way of getting a "top" for the wiki, dbhost, fname fields? This is not just purely cosmetic- if there is a db host, or a wiki, or a function failing more than the others this helps a lot finding the core issue and reducing the outage time.

Event Timeline

I'm having the same problem trying to build a dashboard for T149451: Get 5xx logs into kibana/logstash. Namely none of the fields I'm trying to select e.g. uri_path are available to get further breakdowns, e.g. top 20 urls

It seems the *.raw was unintentionally renamed to *.keyword. Looking into a fix for the templates so we go back to the previous .*raw. Unfortunately this will only fix days going forward, the time between when the server update happened until i fix it will continue to have *.keyword

for me I don't need archives, and I could change things to .keyword, but maybe putting it back is better for others- for me either way works.

Change 353150 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Logstash match_mapping_type still uses string, not text

https://gerrit.wikimedia.org/r/353150

I've manually updated the template in the logstash cluster, first testing on deployment-prep to verify, then updating prod as well. The patch needs to be merged so logstash doesn't decide to re-install the old template on a restart. The new index for may 11th should have the fix, restoring *.raw fields so all the dashboards continue working as they did before the upgrade.

for me I don't need archives, and I could change things to .keyword, but maybe putting it back is better for others- for me either way works.

There are probably numerous dashboards that still refer to the raw fields, I think easiest is to continue using that name.

Change 353150 merged by Gehel:
[operations/puppet@production] Logstash match_mapping_type still uses string, not text

https://gerrit.wikimedia.org/r/353150

Gehel claimed this task.
Gehel added a subscriber: Gehel.

The permanent fix is merged, this should be fixed. Feel free to reopen if it isn't the case.