Page MenuHomePhabricator

[EPIC] Read more related articles after end of current article
Closed, ResolvedPublic

Description

https://m.mediawiki.org/w/index.php?title=Reading/Web/Projects/Read_more

Summary

A user who gets to the bottom of an article on either mobile or desktop web is shown a list of other articles that are related to the one they just finished. The notion is that if the reader has gotten to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for. This has been release on apps and saw a 16% click-through (For users who saw it). Additionally, 25% of the users who saw read more results clicked through at least 1x in a __ period (I can't tell from the query here: https://docs.google.com/spreadsheets/d/1iyNqWgC8lNWk3ckSQJS_ACrfeW1UrBHOf58L97oeNqM/edit?ts=560ae622#gid=1184183009)

Outcome

A/B tests in mobile-web beta show an increased CTR from pages enabled with read more. Read more launched on mobile and desktop web. Editors who have 'see also' sections, appreciate the new format and do not feel that this is an unwarranted step in the wrong direction.

Goal Visibility

This is part of our external goal for the quarter

Rationale

Launching on Android and iOS has shown a high level of success[1]. Though, obviously, results will be different on the web, we have no reason to think it will be better or worse.

Success Metrics

%CTR (click through rate, clicks/views) is >10%.
Ideally, we will be able to ensure that these clicks are not-cannabilistic (increasing overall clicks as opposed to taking clicks from blue links)--this might be possible to measure using the 'referer' field in page logs and doing a before/after analysis or, ideally, a/b test.

External Dependencies

Community Liaisons are instrumental for this rollout.

Unknowns

Not sure if the new feature will be welcomed on desktop, which we, as a team, haven't touched yet. We hope the added configurability offered to editors (over the app editions), and the commitment it shows to honoring editor control in this regard, will be well received.
Not sure if we will be able to a/b test referrals from page (would likely require url appends..)

Product Plan

Prototyping

This will be tested in beta first.

MVP

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished.
We are able to measure the engagement clicks/impressions with this feature.

User Stories

  • A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished so that they can continue reading about that topic.
  • Someone editing a wikipedia article can manually change the article suggestions using wikitext so that they can correct any erroneous or sub-optimal suggestions.
  • An project stakeholder (such as PM or data analysit) is able to measure the engagement clicks/impressions with this feature so they can determine if it is adding user value.

Metrics Implementation

We will want to track:

  • impressions of read more suggestions (the lump--not each item suggested)
  • clicks to read a suggested article
    • the position of article (1,2,3)
    • ideally: whether or not article was edited manually
  • Ideally:
    • overall referrals from a page with read more
    • overall referrals from same page without read more

Timeline Estimate

  • build read more according to mobile web specs
  • launch on mobile web beta
  • measure impact using event logs CTR (referral data won't be helpful at these numbers)
  • discuss with community
  • launch on mobile
  • build for desktop
  • launch on desktop (open discussion about whether we launch in beta first and/or do progressive rollout)

Delivery Estimate

End of December on desktop?

[1]
https://docs.google.com/spreadsheets/d/1iyNqWgC8lNWk3ckSQJS_ACrfeW1UrBHOf58L97oeNqM/edit?ts=560ae622#gid=1184183009)

Related Objects

StatusSubtypeAssignedTask
ResolvedJdlrobson
ResolvedJdlrobson
DuplicateNone
ResolvedJdlrobson
Resolved bmansurov
Resolvedphuedx
DuplicateNone
Resolved bmansurov
InvalidNone
ResolvedJdlrobson
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedJdlrobson
Resolved bmansurov
Resolvedphuedx
ResolvedJdlrobson
Duplicate KHammerstein
Resolvedphuedx
Resolvedphuedx
ResolvedSumit
Resolved bmansurov
Resolved bmansurov
Resolved Nirzar
Resolved Moushira
ResolvedJdlrobson
Resolvedphuedx
Resolvedphuedx
Resolvedphuedx
ResolvedJdlrobson
DeclinedEsanders
ResolvedJdlrobson

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
phuedx subscribed.

@KHammerstein: I've moved this to Should Have as I'm unsure of where this has come from. Is this a Must Have?

@phuedx Should have is fine for now. We definitely want to give readers access to similar articles eventually. We should wrap up header, sticky footer and menu styling updates first?

Jdlrobson renamed this task from Read more related articles after end of current article to [EPIC] Read more related articles after end of current article.Sep 18 2015, 11:45 PM
Jdlrobson moved this task from To classify to Goals on the Reading-Web-Planning board.

@JKatzWMF can you weigh in/clarify the numerical outcome for this and testing/rollout plan?

JKatzWMF added a subscriber: Jdlrobson.

@Jdlrobson I edited the EPIC description to fit Reading's new EPIC standards. I included metrics and measurement goals. One question: I know we had talked about over-writable read-more. Is that still the case? If not, let me know and I will edit accordingly.

@JKatzWMF: Yes. My first pass at this uses the related articles available via RelatedArticles. I'll soon have it falling back to the "morelike" API if there aren't any set.

Jdlrobson claimed this task.

I'm declaring victory on this goal. Let's look at the data over the next month and work out the next steps - whether that be pushing feature to stable / killing it / iterating on it.

I'm declaring victory on this goal. Let's look at the data over the next month and work out the next steps - whether that be pushing feature to stable / killing it / iterating on it.

I assume "victory" refers to deployment of the feature (on mobile web beta only, right?), not yet the associated quarterly goal which also stipulated a "5% increase in links clicked". Which by the way does not match the success metric specified above in this task: "%CTR (click through rate, clicks/views) is >10%." Which one is it?

Regarding the third user story, what is the schema for this instrumentation?

@Tbayer, I think you are right regarding vicotry.

We can't measure the associated quarterly goal so we should use the 10% click through number*.

*Actually, on mobile we could look at referrals in the web logs between beta and stable and compare the ratio, before and after the related articles roadmap...let me know if you're up for that.

The schema for measuring the third user story is: https://meta.wikimedia.org/wiki/Schema:RelatedArticles

(I happened upon these questions, but generally only see things I am tagged in)

Just figured out why I suddenly started seeing Related Articles below the category line. I love it - great feature - but would like to have more control over what articles pop-up. Also, can it be alternating with each load? Thanks for the new toy!! ~~~~

Atsme: so interesting! If we keep it generated by a machine, than rotating the articles is more achievable (and an interesting idea we hadn't considered). However, we have also made it so that editors can select the articles they would like to show up by adding this tag to the page:

{{#related:new page title1}}
{{#related:new page title2}}
{{#related:new page title3}}

Jdlrobson renamed this task from [EPIC] Read more related articles after end of current article to [EPIC] Push Read more related articles to stable.Dec 22 2015, 10:31 PM
Jdlrobson reopened this task as Open.

Hi, I created a new ticket T122260 to track the move from related articles in beta to stable
Are you guys okay with using that as a separate task? I think we should recognize that this goal was accomplished in Q2

JKatzWMF renamed this task from [EPIC] Push Read more related articles to stable to [EPIC] Read more related articles after end of current article.Dec 23 2015, 4:16 AM
JKatzWMF closed this task as Resolved.

Hi, I created a new ticket T122260 to track the move from related articles in beta to stable
Are you guys okay with using that as a separate task? I think we should recognize that this goal was accomplished in Q2

Sure. This task tracked the Q2 goal of releasing this feature as an MFβ/beta feature, which we've done.

This has to be deactivated for the German Wikipedia ASAP (and never should have been activated) since the German WP does have editoral rules with which this tool does not comply, namely w:de:Wikipedia:Themenring. Why you never ask before you invent such a tool (which in measures of sense does not deliver any useful content as well)

Why do you not accept, that Wikipedia is an encyclopaedic project and not a playground for developpers?????

This has to be deactivated for the German Wikipedia ASAP (and never should have been activated) since the German WP does have editoral rules with which this tool does not comply, namely w:de:Wikipedia:Themenring.

That's nonsense. Speaking as one of the authors of the page that you quote, it does not apply to this situation at all.
A more suitable comparison would be "Siehe auch" (see also) links, which have been an established practice on the German Wikipedia since at least 2003 - see https://de.wikipedia.org/wiki/Wikipedia:Assoziative_Verweise .

That's nonsense. Speaking as one of the authors of the page that you quote, it does not apply to this situation at all.

Off course this is the case. This is a classical Themenring. It does no matter wether such an unwanted navigation element is created by a temlate or by software.

A more suitable comparison would be "Siehe auch" (see also) links, which have been an established practice on the German Wikipedia since at least 2003 - see https://de.wikipedia.org/wiki/Wikipedia:Assoziative_Verweise .

Also under this rule the extension would be unwanted, since Assoziative Verweise are only allowed if they link to a continuative artile, e.g. in the article on the river Rhine to the List of rivers in Germany. A link to let's say Mannheim or Kölner Dom (which both are located directly on its banks) would be unwanted. And the latter is the principle this extension works.

Also Wikipedia:Assoziative Verweise and/or common sense are discouraging the inclusion of links to articles already linked earlier in the text.