Removing tag, because IIUC this is a subtask about adding metrics, and hence doesn't need a separate announcement. Please undo if I've misunderstood. :)

I would add a comparison point with Notifications given by the icons:

• Talk pages notifications
• Talk pages subscription notifications.

As brought in a conversation by @Michael, this sounds like a good use case for using Metrics Platform as the experimentation/analytics tool, calculating a click through rate should be fairly easy and an associated Superset dashboard could be shared more publicly than grafana. What do you think @KStoller-WMF?

I just found this dashboard already exists in grafana: Unseen flyout. Instrumentation code in server Echo/Hooks.php#1126 for the impressions and client ext.echo.init.js#256 for clicks.

My prior weight on how easy would be to use MP instruments here also depends on T381853, so maybe grafana is just fine.

Thanks, @Sgs! Perfect, that data is already open and accessible to anyone interested. 🎉

Hmmm, what do you think about me re-scoping this task to focus on creating a Grafana dashboard that effectively displays the relevant data in a more meaningful way. Some ideas:

• Dashboard Terminology: Rename ambiguous terms for better clarity. For example, the term "Unseen flyout" is unclear (at least to me!); renaming it to something like "Talk Page Notification Alert Bar" would make its purpose and connection to the data more understandable.
• Key Metrics Display: Include a dedicated panel for the Click Through Rate (CTR), as this is a critical metric for understanding engagement.
• Filtering Options:Enable filtering by individual wikis, making the data more meaningful to individual communities.
• Comparison of User Types: If possible, provide comparisons of impressions and CTR for Temporary Accounts versus IP Editors. I unclear if this level of granularity is achievable given how the data is currently collected.

Change #1102786 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/Echo@master] [WIP] Add wiki and user type to flyout metric

https://gerrit.wikimedia.org/r/1102786

I started the changes to enable filtering by wikis and user types. Since one of the relevant metrics is captured in the server (and has already been migrated to Prometheus) and the click is captured on the server which still lacks Prometheus support, it's been brought during code review that changing the filters for the current chart which still consumes Graphite seems not worth. Marking this as blocked until T355837 is resolved and we can make the appropriate changes.

Change #1102786 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] statslib: migrate unseen.click metric

https://gerrit.wikimedia.org/r/1102786

Moving back to doing until the new Grafana dashboard is finalized.

I'm failing to find how to create a variable based on values for the wiki label, similar to what we have in graphite dashboards for per-wiki metrics. This is the variable setting form for each:

Graphite	Promtheus

Any hints? cc @colewhite @Michael

In T381607#10500524, @Sgs wrote:

I'm failing to find how to create a variable based on values for the wiki label, similar to what we have in graphite dashboards for per-wiki metrics. This is the variable setting form for each:

Graphite	Promtheus

Any hints? cc @colewhite @Michael

The Prometheus screenshot looks correct to me. I tried the same on a test dashboard and the values showed up in the Preview of Values section after clicking "Run Query."

Thank you @colewhite, for some reason the query was not loading at the time I tried. I'm trying to build a panel ( Echo notifications ) that allows to filter by wiki and "user types" and I'm failing to make a query that would work also for the "All" option. I've customized the all option to be of value .* and used the regex operator =~ in the query, eg: sum(increase(mediawiki_Echo_unseen_impression_total{wiki=~"$wiki",user_type=~"$user_type"}[1h])). Is there any Prometheus dashboard that achieves per-wiki filtering, maybe in a different way? Ty!

In T381607#10531933, @Sgs wrote:

I'm trying to build a panel ( Echo notifications ) that allows to filter by wiki and "user types" and I'm failing to make a query that would work also for the "All" option. I've customized the all option to be of value .* and used the regex operator =~

I've updated that dashboard to substitute the all option with .*. Does the dashboard now behave as you expect?

In T381607#10532517, @colewhite wrote:

I've updated that dashboard to substitute the all option with .*. Does the dashboard now behave as you expect?

Thank you @colewhite, that helped!

I found why we aren't getting impressions or clicks of IP users even if temporary accounts are not broadly available. The reason is the "talk page unseen message" exists for anon users but it is handled by core rather than Echo, hence it is not tracked by the echo instrumentation. I believe we can instrument the impressions and clicks there but it is irrelevant for the Echo color decision making in T341305.

I've moved a new dashboard in Growth Team's folder: Echo talk page notification CTR. We can try to add additional logging for the message anon users get but I'm not sure where that should be charted. Or maybe mixing with Echo is just fine if we think about it as a general mw kpi. What do you think @KStoller-WMF ?