There will likely be little or no observable difference in user behavior from the "sticky headers" / expanded sections feature on pages that are very short (requiring little to no scrolling).
We may need to limit the pages where this feature is active (or at least where we collect data about its usage) to exclude very short articles. This is similar to what we did for image browsing, where we only enabled the feature on articles with a certain number of images.
Some additional points from @jwang that we might want to consider;
I assume that for very short or empty pages (0–1000 bytes), users are unlikely to use our feature at all. I’d like to exclude those cases so we can focus on measuring the impact on users who are likely to use the feature. Any method that helps identify users who wouldn’t use the feature is sufficient for our measurement needs.
In terms of priority and importance, the number of clicks on section headers and scroll-to-top actions are more of a curiosity question for us. So any simple method engineers can implement that helps us narrow down the user group a bit more is good enough. Whether click behavior differs between pages with, for example, 50,000 vs. 70,000 bytes, that’s outside the scope of our current measurement—though it is a great research question.
So the key decision point is determining how to roughly identify scenarios where users no longer need to click the section header or tap to scroll back. Page length could be a factor. However, from the discussion in this thread, I learned that it might not be only about page length, so feel free to propose other options, such as “pixel height,” as Matthias suggested.
Open questions
- What criteria will we use to determine whether a given page is too short?
- Do we want to exclude the sticky headers feature from activating on "short pages" all together? Or is it sufficient to simply not collect usage data in these cases?