@olga , I have published the initial version of sticky header AB test analysis report at: https://jenniferwang-wmf.github.io/Web_sticky_header/ . Let me know if you have any questions.

@jwang T304366 wouldn't explain the low click rate as the error impacts both menus, not just the sticky header.

Most of the user drop down menus in my opinion are not links you need to click while reading. It would be useful to look closely at the timestamps compared to the init event. I imagine most clicks to the user menu (not sticky menu) items happen within the first one minute of a session.

@Jdlrobson

Most of the user drop down menus in my opinion are not links you need to click while reading

I agree. Language down, history and talk tab might be pageview based links to click. The others are more user based links. Therefore, I defined the metric as scroll-back per session and clicks per session instead of scroll-back per pageview.

It would be useful to look closely at the timestamps compared to the init event.

What's the purpose? I looked at the number of inits, pageviews, sessions in analysis notebook. Data shows number of daily inits > number of daily pageviews > number of daily sessions. . Clicks/sessions gave us the highest CTR number.

What's the purpose? I looked at the number of inits, pageviews, sessions in analysis notebook. Data shows number of daily inits > number of daily pageviews > number of daily sessions. . Clicks/sessions gave us the highest CTR number.

I think it would be useful to know the time duration between the session initializing and a menu item being clicked. This is important thing to consider IMO as the sticky header is hidden at page init, but every other button is not.

My hypothesis is that if you limited your analysis to say clicks that occurred after say 2 minutes, you'd see that the majority of clicks were to the sticky header icons not the main header. Happy to chat on Slack/hangout if it's still not clear.

587 sessions in stikyHeaderEnabled group has old skin version (skinversion=1). They will not see the new feature. Our analysis exclude them in treatment group (stikyHeaderEnabled).

@jwang did you check the skin field for these events? If skinversion=1 and skin=vector-2022 then the new skin is being used and they should be included in analysis. The A/B test code only runs for new Vector [1] but confusingly our data reports different versions of skin version due to the recently completed migration so these were almost definitely valid.

If excluding them please can you check that you excluded any events in the stickyHeaderDisabled group with skinversion set to 1 as well.

[1] (Technical) The event is trigged in resources/skins.vector.es6/AB.js which is only called in resources/skins.vector.js/skin.js

Do you mean we only need to group users by experiment group tag (stickyHeaderDisabled , stickyHeaderEnabled)? Please clarify for below combination, which feature will user see.

Our metrics are ratio to number of sessions in each category . As along as there are no mis-categorized users in treatment and control groups, the analysis result is still valid.

You can ignore the skinversion field for the purpose of this experiment. It's irrelevant and misleading. It's impossible to log events for anything other than new Vector.

stickyHeaderDisabled = do not see sticky header
stickyHeaderEnabled = see sticky header

Thanks for confirming that. If so, the analysis will be much much easier than current version. Tag @ovasileva @cjming so that we are all on the same page.

The current analysis result is still valid, as there is no mis-categorization. The sessions we excluded ( skinverion=1 & test_group=stickyHeaderEnabled) is just 0.29% of total sessions in stickyHeaderEnabled group.