First impressions of the new Logstash/Kibana based on using Firefox 74 for macOS on an idle high-end MacBook Pro using a fast WiFi connection.

• It is even slower to load. Just to have the UI appear initially at all now takes 7-8 seconds on logstash-next compared to ~ 1s second on logstash (this is while loading the domain and seeing the "Loading" animation).

• All interface links and buttons are unresposive. When hovering any link or button (e.g. on any dashboard the "Close", "Show dates", "Lucene" or "Refresh" buttons) they are without a pointer cursor for the first 1-2 seconds before they can be clicked.
  • This also applies to modal interfaces such as the "edit filter" overlay, and the date inputs.
  • This is actually really difficult to screw up in a modern browser, so I'm kind of impressed they managed to make the UI this bad.

• As a silver lining, they seem to have finally fixed the autocomplete widget for "Edit filter". It no longer tries to preload all 90 days of Logstash indexes client-side and iterate over every unique field on every keystroke (which is what led to T189333). Instead, this data is now lazy-loaded in chunks and filtering is debounced properly, resulting in an input field that is now actually usable, in all browsers I tried. Yay!

The codfw cluster is currently yellow, from explain I see a lot of "explanation" : "node does not match index setting [index.routing.allocation.require] filters [disktype:\"hdd\"]"

I acked the alerts since the notifications were turned off.

there was some work to rotate old indexes to spinning disks but the cluster knew of no nodes with the "hdd" disktype attribute. it looks like the configuration was stale and restarting logstash[2021-2022] allowed the indexes to be assigned.

Since Elastic stack 7.7 has been released I think it'd make sense we upgrade to that before the switch, supposedly there have been improvements to memory usage!

Change 609397 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: decom check_procs

https://gerrit.wikimedia.org/r/609397

Change 609397 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: decom check_procs

https://gerrit.wikimedia.org/r/609397

Change 610079 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] add thirdparty/elastic78 component

https://gerrit.wikimedia.org/r/610079

Change 610079 merged by Herron:
[operations/puppet@production] add thirdparty/elastic78 component

https://gerrit.wikimedia.org/r/610079

Change 610135 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set v7 cluster to version 7.8

https://gerrit.wikimedia.org/r/610135

Change 610135 merged by Herron:
[operations/puppet@production] logstash: set v7 cluster to version 7.8

https://gerrit.wikimedia.org/r/610135

Mentioned in SAL (#wikimedia-operations) [2020-07-09T19:16:27Z] <herron> upgraded eqiad elk7 cluster from 7.4.2 to 7.8.0 T234854

I am getting a lot of 500 internal server errors on logstash-next instance. I am guessing that is expected/WIP?

In T234854#6344456, @jcrespo wrote:

I am getting a lot of 500 internal server errors on logstash-next instance. I am guessing that is expected/WIP?

Not necessarily expected, but is being tracked in T259219 and a fix has been put in place

Mentioned in SAL (#wikimedia-operations) [2020-08-26T18:08:44Z] <herron> upgraded eqiad elk v7 cluster from 7.8.0 to 7.9.0 T234854

Another thing I tracked is that dashboards doesn't seem to do proper filtering. Do dashboards have to be fully redone for 7? I am assuming dashborad definitions have been automatically imported.

If I compare:

• https://logstash-next.wikimedia.org/app/dashboards#/view/DBQuery

with the original

• https://logstash.wikimedia.org/app/kibana#/dashboard/DBQuery

We can see than on the original, channel DBQuery logs are rare (queries rarely fail unless a very bad thing is happening), however at the time of the comment, I am getting 1 result for the old install and dozens of thousands on the new one. Something's wrong because on the host list there are hosts like "kubernetes2013" and "gerrit1001" which wouldn't be doing mediawiki queries. Are the original dashboards wrong? Did the syntax to create them changed, or there is another reason? This is not the only dashboard affected (do all dashboards need redo?). I can create a task if you believe this is a real issue.

@jcrespo Thanks for bringing this to our attention. The filters on that dashboard indicate they are broken because the filter pattern logstash-* cannot be found on logstash-next.

Trying to load: https://logstash-next.wikimedia.org/app/dashboards#/view/6bcd2a10-7d21-11e7-86fb-51c84229aeb7

My laptop fan starts spinning very hard, everything times out and I get this popup:

To compare with https://logstash.wikimedia.org/app/kibana#/dashboard/6bcd2a10-7d21-11e7-86fb-51c84229aeb7

In T234854#6584418, @ayounsi wrote:

Trying to load: https://logstash-next.wikimedia.org/app/dashboards#/view/6bcd2a10-7d21-11e7-86fb-51c84229aeb7

My laptop fan starts spinning very hard, everything times out and I get this popup:

Hmm, I'm not able to reproduce this consistently. On occasion the slow query popup does appear, but have not been able to produce any full timeouts. Is this occurring for you with a 15m time selection?

I was able to load it successfully this time on both Chrome and Firefox, it's very slow though. This is all with the default 15min.

All the filters have a red "error" after them.

Surprisingly, the content of the dashboard visualizations are different between the 2 instances (looks like different sources, -next is about mediawiki, not networks).

Is there a easy to read changelog for end-users like me?

In T234854#6440649, @colewhite wrote:

@jcrespo Thanks for bringing this to our attention. The filters on that dashboard indicate they are broken because the filter pattern logstash-* cannot be found on logstash-next.

Could you elaborate on this- does this mean "don't worry, it will be right when the cut-over happens" or "you should do X actionable beforehand to fix it"? Sorry, I don't have lot of experience about kibana internals, but this dashboard is important to us. CC @Marostegui

In T234854#6016076, @Krinkle wrote:

First impressions of the new Logstash/Kibana based on using Firefox 74 for macOS on an idle high-end MacBook Pro using a fast WiFi connection.

• It is even slower to load. Just to have the UI appear initially at all now takes 7-8 seconds on logstash-next compared to ~ 1s second on logstash (this is while loading the domain and seeing the "Loading" animation).

• All interface links and buttons are unresposive. When hovering any link or button (e.g. on any dashboard the "Close", "Show dates", "Lucene" or "Refresh" buttons) they are without a pointer cursor for the first 1-2 seconds before they can be clicked.
  • This also applies to modal interfaces such as the "edit filter" overlay, and the date inputs.
  • This is actually really difficult to screw up in a modern browser, so I'm kind of impressed they managed to make the UI this bad.

• As a silver lining, they seem to have finally fixed the autocomplete widget for "Edit filter". It no longer tries to preload all 90 days of Logstash indexes client-side and iterate over every unique field on every keystroke (which is what led to T189333). […]

The above is still the case. I think to the extent possible we should remain on Kibana 6 until and unless these are addressed by upstream, or for us to invest in a basic replacement that can do the minimum visualisations and permalink access we actually want/need in a way that doesn't routinely crash browsers or cost an hour to perform a simple triage task.

In T234854#6599335, @jcrespo wrote:

Could you elaborate on this- does this mean "don't worry, it will be right when the cut-over happens" or "you should do X actionable beforehand to fix it"? Sorry, I don't have lot of experience about kibana internals, but this dashboard is important to us. CC @Marostegui

@jcrespo This issue should be resolved at this point as I now see the logstash-* filter pattern on logstash-next. Please let us know if it is still broken.

In T234854#6016076, @Krinkle wrote:

• It is even slower to load. Just to have the UI appear initially at all now takes 7-8 seconds on logstash-next compared to ~ 1s second on logstash (this is while loading the domain and seeing the "Loading" animation).

I can confirm the UI itself has a longer preload time than Kibana 5. Wall clock from login to rendered home dashboard:

Timing the dashboards implies there is a difference in how Visualizations are rendered and the render is indeed much slower. The "mediawiki-errors" dashboard for example:

7.10 should drop any day now. It's worth seeing what dropping the call to calculateObjectHash() will do for Visualization performance.

@jcrespo This issue should be resolved at this point as I now see the logstash-* filter pattern on logstash-next. Please let us know if it is still broken.

If you click at the links I sent at T234854#6439791 you can see that I get (as of this writing) no errors on DBQuery on the original dashboard and several hundred of thousands of messages on "next" as filtering is not working- I believe it is displaying all logs ("Disabled Index pattern logstash-* not found"). This error is happening on all DB* dashboards: https://logstash-next.wikimedia.org/app/dashboards Maybe I am doing something wrong? Please advice.

@colewhite I am still seeing lots of different results between:
https://logstash-next.wikimedia.org/goto/690aac80b9993d9af88216d0e8103e74 and https://logstash.wikimedia.org/goto/9ac0b716462a5627846cdc060b521c38 as Jaime points out.

Same goes for another crucial dashboard for us:
https://logstash-next.wikimedia.org/goto/2d23f98699ac3555f2f0e0f5af14b1a7 vs https://logstash.wikimedia.org/goto/eab0261f0dffc7f385252ae71b013060 vs

In T234854#6605499, @jcrespo wrote:

If you click at the links I sent at T234854#6439791 you can see that I get (as of this writing) no errors on DBQuery on the original dashboard and several hundred of thousands of messages on "next" as filtering is not working- I believe it is displaying all logs ("Disabled Index pattern logstash-* not found"). This error is happening on all DB* dashboards: https://logstash-next.wikimedia.org/app/dashboards Maybe I am doing something wrong? Please advice.

Thank you for following up on this issue. It seems there is a problem in the migration somewhere. I have amended the dashboards for now; please have a look.

If they are working as expected, we see the problem and will ensure it is fixed prior to the cut-over.

I have amended the dashboards for now; please have a look.

Thanks, they work now.

As a minor issue, one thing I noticed is that page load in the old ones takes 3 seconds, 26 seconds on the new one. This is on the "DBConnection" dashboard (which just filters by type and channel), at a moment where only 2 events are loaded, so it shouldn't be a memory/browser issue. Could it be some indexes are missing on the new setup or something else?

In T234854#6608216, @jcrespo wrote:

As a minor issue, one thing I noticed is that page load in the old ones takes 3 seconds, 26 seconds on the new one. This is on the "DBConnection" dashboard (which just filters by type and channel), at a moment where only 2 events are loaded, so it shouldn't be a memory/browser issue. Could it be some indexes are missing on the new setup or something else?

Unfortunately, it is the browser causing the delay. Kibana 7.10 is purported to have the fix that should eliminate this unnecessary client-side work.

In T234854#6609248, @colewhite wrote:

Unfortunately, it is the browser causing the delay. Kibana 7.10 is purported to have the fix that should eliminate this unnecessary client-side work.

Good news! Thank you for the work!

In T234854#6601662, @Krinkle wrote:

I think to the extent possible we should remain on Kibana 6 until and unless these are addressed by upstream

Thanks for the feedback, it is much appreciated. Observability discussed this concern as a team this morning, and yes we are in agreement that we should hold off until addressed upstream. We'll postpone cutover, and will be contacting elastic to get a better understanding of when 7.10 will be released, as a related fix was merged back in Sept (https://github.com/elastic/kibana/pull/77646) but it is not yet clear when updated packages will be made available.

Kibana 7.10 is live. A quick check of performance indicates performance is back to parity with Kibana 5.

Home dashboard: ~7s
The "mediawiki-errors" dashboard: ~13s

In T234854#6646842, @colewhite wrote:

Kibana 7.10 is live. A quick check of performance indicates performance is back to parity with Kibana 5.

Home dashboard: ~7s
The "mediawiki-errors" dashboard: ~13s

This might be unrepresentative, as it seems none of the filters currently work on Kibana 7.1.

On the various dashboards I checked, all of them have Error for their filter bubbles. This wasn't obvious to me at first since at least one of them actually uses the word "Error" as the value, so it saying "channel: Error" didn't immediately look out of place. But is is actually telling us that the search queries have failed:

When attempting to edit a few of these, it seems they have all turned into blank boxes with a rejected JSON blob as custom DSL (even the ones authored as regular UI-made filter bubble on Kibana 5).

As such, it seems all the dashboards are querying the all source data without any filters (all hosts, all applications/services/programs etc.). The mediawiki-errors dashboard, for example, includes apache2 messages from phab1001, and syslog messages from cp4031, restbase, wdqs2006, etc.

Is there some sort of migration script that exists and needs to be run? Or was there a breaking change in the release that removed support for filters from dashboards on previous versions?

It might also have something to do with the logstash-* index names, which seems to be the first field that shows an error when editing those filter bubbles, in which case might just be a configuration issue.

@Krinkle That looks very similar to the problems I found initially on DB* dashboards, and then they did something to fix it- people here will know more. This was my initial, no longer happening report: T234854#6439791

For additional feedback on my side, I have been using logstash-next for the most part in the last 2 weeks and I am happy with it so far, after the couple of bumps I ran into were solved. Thank you! :-) This stresses the need for beta-testing. Thank you again!

Change 654294 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana: change backend naming from kibana-next to kibana7

https://gerrit.wikimedia.org/r/654294

Change 654436 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana7: add kibana7 conftool entries

https://gerrit.wikimedia.org/r/654436

Change 654437 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana7: repoint (rename) kibana-next services to kibana7

https://gerrit.wikimedia.org/r/654437

Change 654438 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana7: remove kibana-next conftool entries

https://gerrit.wikimedia.org/r/654438

In T234854#6652368, @Krinkle wrote:

It might also have something to do with the logstash-* index names, which seems to be the first field that shows an error when editing those filter bubbles, in which case might just be a configuration issue.

Yes this looks to have been largely the issue. Essentially the existing filters were configured for an index pattern with pattern id logstash-*. At some point along the way the index pattern named logstash-* was re-created in logstash-next where the default behavior is to assign a random index pattern id like acba6310-f6d3-11ea-b848-090a7444f26c. After that happened some dashboards were then updated to use the new index pattern id, and some weren't.

What I've done to address this is re-create the index pattern named logstash-*, and manually speficy an index pattern id of logstash-*. I've also created an index pattern named logstash* with pattern id acba6310-f6d3-11ea-b848-090a7444f26c.

There will be another import from ELK5 to ELK7 in the next few days, and in that process we should be able to converge on a single index pattern. And along with that dashboards will be checked individually for this problem (and general functionality) as well.

Change 655696 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: add kibana7.svc record

https://gerrit.wikimedia.org/r/655696

Change 655696 merged by Herron:
[operations/dns@master] dns: add kibana7.svc record

https://gerrit.wikimedia.org/r/655696

Change 654436 merged by Herron:
[operations/puppet@production] kibana7: add kibana7 conftool entries

https://gerrit.wikimedia.org/r/654436

Change 654437 merged by Herron:
[operations/puppet@production] kibana7: repoint (rename) kibana-next services to kibana7

https://gerrit.wikimedia.org/r/654437

Change 655754 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ELK: promote logstash-next to logstash.wikimedia.org

https://gerrit.wikimedia.org/r/655754

Change 655802 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana7: change vhost from logstash-next to logstash.wikimedia.org

https://gerrit.wikimedia.org/r/655802

Mentioned in SAL (#wikimedia-operations) [2021-01-13T17:11:20Z] <herron> beginning cutover of https://logstash.wikimedia.org frontend to ELK7 T234854

Change 655802 merged by Herron:
[operations/puppet@production] kibana7: change vhost from logstash-next to logstash.wikimedia.org

https://gerrit.wikimedia.org/r/655802

Change 655754 merged by Herron:
[operations/puppet@production] ELK: promote logstash-next to logstash.wikimedia.org

https://gerrit.wikimedia.org/r/655754

Change 655951 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] elk7: enable icinga notifications

https://gerrit.wikimedia.org/r/655951

Change 655951 merged by Herron:
[operations/puppet@production] elk7: enable icinga notifications

https://gerrit.wikimedia.org/r/655951

Change 655957 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] elk7: change kibana7 monitoring to critical

https://gerrit.wikimedia.org/r/655957

Change 655958 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] elk7: remove logstash-next cache setting

https://gerrit.wikimedia.org/r/655958

Change 655959 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: remove logstash-next.wikimedia.org record

https://gerrit.wikimedia.org/r/655959

Change 654294 abandoned by Herron:
[operations/puppet@production] kibana: change backend naming from kibana-next to kibana7

Reason:
replaced by patches beginning with I40047b8824b9d44bf30d29bace4d7fd276d18e62

https://gerrit.wikimedia.org/r/654294

Change 663697 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: remove logstash inputs on legacy cluster

https://gerrit.wikimedia.org/r/663697

Change 663697 merged by Cwhite:
[operations/puppet@production] profile: remove logstash inputs on legacy cluster

https://gerrit.wikimedia.org/r/663697

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:30:52Z] <shdubsh> completed removal of kafka logging inputs to legacy logstash cluster - T234854

Change 676477 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove logstash output on legacy cluster

https://gerrit.wikimedia.org/r/676477

Change 676477 merged by Cwhite:

[operations/puppet@production] logstash: remove logstash output on legacy cluster

https://gerrit.wikimedia.org/r/676477

Change 655957 merged by Herron:

[operations/puppet@production] elk7: change kibana7 monitoring to critical

https://gerrit.wikimedia.org/r/655957

Change 655959 merged by Herron:

[operations/dns@master] dns: remove logstash-next.wikimedia.org record

https://gerrit.wikimedia.org/r/655959

Change 655958 merged by Herron:

[operations/puppet@production] elk7: remove logstash-next cache setting

https://gerrit.wikimedia.org/r/655958

Change 654438 merged by Herron:

[operations/puppet@production] kibana7: remove kibana-next conftool entries

https://gerrit.wikimedia.org/r/654438