Page MenuHomePhabricator

Annotations in Grafana only showing subject
Open, Needs TriagePublic

Assigned To
None
Authored By
Peter
Mar 10 2022, 7:35 AM
Referenced Files
F34996740: Screenshot 2022-03-10 at 14.42.14.png
Mar 10 2022, 1:43 PM
F34995532: Screenshot 2022-03-10 at 08.58.34.png
Mar 10 2022, 7:59 AM
F34995456: Screenshot 2022-03-10 at 08.33.26.png
Mar 10 2022, 7:35 AM
F34995442: Screenshot 2022-03-10 at 08.20.21.png
Mar 10 2022, 7:35 AM
F34995447: Screenshot 2022-03-10 at 08.28.51.png
Mar 10 2022, 7:35 AM

Description

There's some strange things going on with the annotations that we use in Grafana 8.3.6 (annotations from Graphite) where the subject ("what") is displayed but not the data part:

Screenshot 2022-03-10 at 08.20.21.png (340×652 px, 146 KB)

It's easiest to spot by going to https://grafana.wikimedia.org/d/IvAfnmLMk/page-drilldown?orgId=1&viewPanel=80 and choosing "Show each test" and hover on each line.

I have a Grafana instance running 8.3.3 and there it looks like this:

Screenshot 2022-03-10 at 08.28.51.png (672×792 px, 527 KB)

I get some errors in the console:

Screenshot 2022-03-10 at 08.33.26.png (144×1 px, 65 KB)

Where the websocket is https://github.com/grafana/grafana/issues/36929 I guess.

I'm gonna try update my instance to the same version and see if I get the same problem +. then upgrade to latest released Grafana.

Event Timeline

Peter renamed this task from Annotations in Grafana 8.3.6 only showing subject to Annotations in Grafana only showing subject .Mar 10 2022, 7:54 AM
Peter updated the task description. (Show Details)

I upgraded my instance to 8.3.6 and there it works so no version issue.

Could it be something with the database, since I started to collect a lot more data when I started with T203060 ?

Let me look into the logs on the Graphite instance and If there problem could be there.

This is really strange. Zooming out 30 days on the data instead we get the data part and not the subject:

Screenshot 2022-03-10 at 08.58.34.png (1×1 px, 1 MB)

I can see that the data is delivered from Graphite:

Screenshot 2022-03-10 at 14.42.14.png (300×2 px, 411 KB)

Hi @colewhite do you know if some configuration for showing annotations that includes HTML was changed when you upgraded to Grafana 8 in T282863

Hi @colewhite do you know if some configuration for showing annotations that includes HTML was changed when you upgraded to Grafana 8 in T282863

T300997 was the only config change we did, and it applied only to the instance at grafana-next.wm.o.

The plugins we use were upgraded, but AFAIK these should not effect annotations.

This is really strange. Zooming out 30 days on the data instead we get the data part and not the subject:

I noticed the same behavior. Seems to affect Firefox as well as Chrome.

Interestingly, the behavior changes on 2022-02-21. The Grafana 8 upgrade was completed on March 1st.

I think there's a bug in how annotations are rendered when combined with other annotations or annotation sources.

  1. Go to the panel on 2022-02-21
  2. Enable Show sync-wikiversions and Show each tests
  3. Observe behavior of each annotation by hovering over it
  4. Disable Show sync-wikiversions
  5. Observe inconsistent annotation rendering
    1. It may take a few tries turning the annotation switches on and off to see a difference.
    2. Each time an annotation switch is toggled, annotation data is refreshed and the ui components rebuilt.
  6. The inconsistent rendering is even more erratic when mixed with the Changing name server on bare metal annotation on 2022-02-21 14:21:10 (link)

Ah you are right @colewhite thank you so much! I could reproduce by turning on/off different annotations. This is a really annoying bug but with turning off the other annotations I can at least get the info I need.

Hmm 2022-02-18 I added a lot more tests in T203060 so that the database that holds the Graphite annotations for "Show each tests" started to increase. Normally we keep data for 34 days in that and purge older entries every Sunday. I did a purge yesterday to decrease the data just see if it had any effect but couldn't see anything. However you seeing it started a couple of days after I turned that on sounds suspicious? However inspecting the data transferred it looked like everything was sent from Graphite to Grafana.

I can create an issue in Grafana and ask them.

This has magically been fixed, after I created the issue and when someone from the Grafana team tried to reproduce, it started to work again. I tried now on multiple browsers, both logged in and logged out.

Thanks, I re-opened the issue.

Peter removed Peter as the assignee of this task.Mar 14 2022, 6:36 PM
Peter moved this task from Doing: Prio Interrupt to Radar on the Performance-Team board.
Peter edited projects, added Performance-Team (Radar); removed Performance-Team.