@jonkatz ping pang pong

@Dbrant or @Deskana what did you do to measure for read-more in apps? We probably want to implement similarly with similar principles.

^ @dgarry is actually @Deskana I believe

^thanks, just edited.

We looked at a few different metrics.

Reach

• number of uniques who saw at least one panel in a month / install base
• we had no monthly uniques data at that stage; I'd recommend using MAUs instead of install base

Clickthrough

• number of clicks on a result / number of times the panel was seen

Number of page views from feature

• number of clicks

Since we didn't have a baseline to compare the feature to, we opted just to record these metrics and report them as-is without comparing them to anything.

@JKatzWMF In the app, we do the following:

• Send an event when the Read More section is first scrolled into view.
• If the user taps one of the items, send an event with the index of the item that was tapped.
• These events come with a unique session id, to tell us what the user decides to do after first seeing the Read More section (click an item or not).
• (sampled at 1:100, per user)

thanks @Dbrant and @Deskana! are the results somewhere where I can grab them? I want to set goals for the web that use your metrics as an anchoring point.

@JKatzWMF There's historical data in this spreadsheet: https://docs.google.com/spreadsheets/d/1iyNqWgC8lNWk3ckSQJS_ACrfeW1UrBHOf58L97oeNqM/edit#gid=0

@Deskana awesome, thanks!

In addition to the above measures we could anonymously record the dwell time after a user clicks on a read-more link. That way we know that the link was not clicked by mistake. For this we'd need to set an average threshold time that is enough to read a lead section of an article and send an event logging event to record that the user has actually stayed at least that long after the page has finished rendering.

I disagree-unless we are worried that our ux is deceptive or nonintuitive
or overly prone to unintentional clicks, we need to be able to trust that
people's fingers are acting under their command.

Sorry for the confusion. I didn't mean a mistake because of UI confusion, but mistake because the user thought that the article they clicked on is the one they are looking for. So if the user finds out that the article is not what they want, it doesn't mean that the 'read more' link proved useful, although we record a click count. It just means the user clicked on a link "by mistake" and didn't find what they were looking for. This situation doesn't add value to anyone and should not be counted towards the success of 'read more'.

@bmansurov I see now. this is something the discovery team, specifically @Deskana, is trying to figure out for search and it actually very hard to measure using dwell time alone. I think that until Discovery, who has a lead time and much larger stake in this, figures out a way to measure user satisfaction, we should assume what we assume for every other navigational feature (such as blue links)-->that users clicking over time represent user value and that clicks would diminish over time if they were not generating value.

One could measure dwell time on clicks from read more and compare the dwell time distributions between links navigated to from other sources such as search, which is what our data records. That would tell you whether there's a difference; whether a longer or shorter dwell time is better or worse is harder to figure out and there is room for legitimate debate there.

@Jdlrobson, I suggest we do the following, given that we do not have uniques on web:

Minimum
Send an event when the Read More section is first scrolled into view.
If the user taps one of the items, send an event with the index of the item that was tapped.
(sampled at 1:100, per user)

Ideal
Ideally we could show what the apps could not, which is that the read more clicks represent new engagement rather than cannibalism. This could be accomplished via redirect or appending something to the url when read more is active (X% of the time) and then looking at pageview referrals. You would want to see that pageview's coming from pages with read more were higher than when the pages did not have read more.

We would probably need to sample more aggressively. Given this is beta only to start with I think let' start with maybe a 5% sample there just to be conservative (we can adjust it later as needed)

In terms of cannabilism:
Not to clear on how appending a query string url to links will show that.

Are you expecting to be able to prove the following?
Total page views with referrer from read more * sampling rate = Total page views with referrer in mobile beta

E.g. if 20 links are clicked by users in read more and there are 100 links in total clicked in beta (including those with read more) then we would expect that we were using a 20% sampling rate?

@Jdlrobson I don't understand your example, but the idea around appending
urls on a url that has read more enabled would be that you could show read
more on 10% of all page views for all articles, and append those urls with
'rd_more'. Then someone could query hadoop for referers like '%rd_more%'
or URI like '%rd_more%'

Then you could say something like the following:

*Tuesday, Dec 16th*
Total articles --> 1000 pageviews, 400 links clicked (blue links + read
more), 40%
Total with read more--> 100 pageviews, 45 links clicked (blue links + read
more), 45%

Therefore, readmore leads to 12.5% more overall clicks.

Talked to Jon Katz.
We agreed testing cannabilisation of links can wait for the time being.

To summarise:
We will this do the following (copying apps):

• Enable EventLogging on a 1% sample of users using read more (note read more is in beta so this will be 1% of users in beta).
• Sample rate will be configurable by global config variable. We will tweak it as we go.
• Send an event when the Read More section is first scrolled into view.
• If the user taps one of the items inside read more, send an event with the index of the item that was tapped.
• It should be possible to link the above two events to answer "if user saw read more what % engaged with it"
• We hope that 12.5% of users that see read more engage with read more.

Once you are all okay with this I will add this to T114303

Sounds like a plan.

We'll also need to include the notion of where the related articles came from: editor-curated via the relatedarticles parser function or the API.

@Jdlrobson

We hope that 12.5% of users that see read more engage with read more.

Let's say "pageviews" instead of users, since we can't track users. Per out discussion, I also think we should set the bar at 5%. We saw 16% with app, but that is a more motivated user-group.

Otherwise, all good--thanks for summarizing :)