[EPIC] Read more related articles after end of current article
Closed, ResolvedPublic

Description

https://m.mediawiki.org/w/index.php?title=Reading/Web/Projects/Read_more

Summary

A user who gets to the bottom of an article on either mobile or desktop web is shown a list of other articles that are related to the one they just finished. The notion is that if the reader has gotten to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for. This has been release on apps and saw a 16% click-through (For users who saw it). Additionally, 25% of the users who saw read more results clicked through at least 1x in a __ period (I can't tell from the query here: https://docs.google.com/spreadsheets/d/1iyNqWgC8lNWk3ckSQJS_ACrfeW1UrBHOf58L97oeNqM/edit?ts=560ae622#gid=1184183009)

Outcome

A/B tests in mobile-web beta show an increased CTR from pages enabled with read more. Read more launched on mobile and desktop web. Editors who have 'see also' sections, appreciate the new format and do not feel that this is an unwarranted step in the wrong direction.

Goal Visibility

This is part of our external goal for the quarter

Rationale

Launching on Android and iOS has shown a high level of success[1]. Though, obviously, results will be different on the web, we have no reason to think it will be better or worse.

Success Metrics

%CTR (click through rate, clicks/views) is >10%.
Ideally, we will be able to ensure that these clicks are not-cannabilistic (increasing overall clicks as opposed to taking clicks from blue links)--this might be possible to measure using the 'referer' field in page logs and doing a before/after analysis or, ideally, a/b test.

External Dependencies

Community Liaisons are instrumental for this rollout.

Unknowns

Not sure if the new feature will be welcomed on desktop, which we, as a team, haven't touched yet. We hope the added configurability offered to editors (over the app editions), and the commitment it shows to honoring editor control in this regard, will be well received.
Not sure if we will be able to a/b test referrals from page (would likely require url appends..)

Product Plan

Prototyping

This will be tested in beta first.

MVP

A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished.
We are able to measure the engagement clicks/impressions with this feature.

User Stories

  • A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished so that they can continue reading about that topic.
  • Someone editing a wikipedia article can manually change the article suggestions using wikitext so that they can correct any erroneous or sub-optimal suggestions.
  • An project stakeholder (such as PM or data analysit) is able to measure the engagement clicks/impressions with this feature so they can determine if it is adding user value.

Metrics Implementation

We will want to track:

  • impressions of read more suggestions (the lump--not each item suggested)
  • clicks to read a suggested article
    • the position of article (1,2,3)
    • ideally: whether or not article was edited manually
  • Ideally:
    • overall referrals from a page with read more
    • overall referrals from same page without read more

Timeline Estimate

  • build read more according to mobile web specs
  • launch on mobile web beta
  • measure impact using event logs CTR (referral data won't be helpful at these numbers)
  • discuss with community
  • launch on mobile
  • build for desktop
  • launch on desktop (open discussion about whether we launch in beta first and/or do progressive rollout)

Delivery Estimate

End of December on desktop?

[1]
https://docs.google.com/spreadsheets/d/1iyNqWgC8lNWk3ckSQJS_ACrfeW1UrBHOf58L97oeNqM/edit?ts=560ae622#gid=1184183009)

Related Objects

StatusAssignedTask
ResolvedJdlrobson
ResolvedJdlrobson
DuplicateNone
ResolvedJdlrobson
Resolvedbmansurov
Resolvedphuedx
DuplicateNone
Resolvedbmansurov
InvalidNone
ResolvedJdlrobson
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedJdlrobson
Resolvedbmansurov
Resolvedphuedx
ResolvedJdlrobson
Duplicate KHammerstein
Resolvedphuedx
Resolvedphuedx
ResolvedSumit
Resolvedbmansurov
Resolvedbmansurov
ResolvedNirzar
Resolved Moushira
ResolvedJdlrobson
Resolvedphuedx
Resolvedphuedx
Resolvedphuedx
ResolvedJdlrobson
DeclinedEsanders
ResolvedJdlrobson
There are a very large number of changes, so older changes are hidden. Show Older Changes
phuedx moved this task from To Triage to Tracking on the Reading-Web-Backlog board.May 4 2015, 3:49 PM
phuedx added a subscriber: phuedx.

@KHammerstein: I've moved this to Should Have as I'm unsure of where this has come from. Is this a Must Have?

@phuedx Should have is fine for now. We definitely want to give readers access to similar articles eventually. We should wrap up header, sticky footer and menu styling updates first?

Jhernandez triaged this task as Normal priority.Aug 17 2015, 11:19 AM
Jdlrobson renamed this task from Read more related articles after end of current article to [EPIC] Read more related articles after end of current article.Sep 18 2015, 11:45 PM
Jdlrobson moved this task from To classify to Goals on the Reading-Web-Planning board.
KLans_WMF updated the task description. (Show Details)Sep 21 2015, 4:29 PM
KLans_WMF added subscribers: JKatzWMF, KLans_WMF.EditedSep 21 2015, 4:32 PM

@JKatzWMF can you weigh in/clarify the numerical outcome for this and testing/rollout plan?

Jdlrobson updated the task description. (Show Details)Sep 24 2015, 6:02 PM
JKatzWMF updated the task description. (Show Details)Oct 8 2015, 5:57 PM
JKatzWMF added a subscriber: Jdlrobson.

@Jdlrobson I edited the EPIC description to fit Reading's new EPIC standards. I included metrics and measurement goals. One question: I know we had talked about over-writable read-more. Is that still the case? If not, let me know and I will edit accordingly.

phuedx added a comment.Oct 8 2015, 6:15 PM

@JKatzWMF: Yes. My first pass at this uses the related articles available via RelatedArticles. I'll soon have it falling back to the "morelike" API if there aren't any set.

phuedx added a comment.Oct 8 2015, 6:16 PM

Sorry to jump on the question @Jdlrobson!

JKatzWMF updated the task description. (Show Details)Oct 13 2015, 4:24 PM
Jdlrobson updated the task description. (Show Details)Oct 13 2015, 8:04 PM
Tbayer added a subscriber: Tbayer.Oct 14 2015, 3:08 AM
dr0ptp4kt moved this task from Backlog to Current Quarter on the Reading-Admin board.
phuedx changed the status of subtask T115019: Remove the CustomData dependency in 30 days from Stalled to Open.Nov 11 2015, 4:55 PM
Jdlrobson closed this task as Resolved.Dec 15 2015, 1:19 AM
Jdlrobson claimed this task.

I'm declaring victory on this goal. Let's look at the data over the next month and work out the next steps - whether that be pushing feature to stable / killing it / iterating on it.

I'm declaring victory on this goal. Let's look at the data over the next month and work out the next steps - whether that be pushing feature to stable / killing it / iterating on it.

I assume "victory" refers to deployment of the feature (on mobile web beta only, right?), not yet the associated quarterly goal which also stipulated a "5% increase in links clicked". Which by the way does not match the success metric specified above in this task: "%CTR (click through rate, clicks/views) is >10%." Which one is it?

Regarding the third user story, what is the schema for this instrumentation?

@Tbayer, I think you are right regarding vicotry.

We can't measure the associated quarterly goal so we should use the 10% click through number*.

*Actually, on mobile we could look at referrals in the web logs between beta and stable and compare the ratio, before and after the related articles roadmap...let me know if you're up for that.

The schema for measuring the third user story is: https://meta.wikimedia.org/wiki/Schema:RelatedArticles

(I happened upon these questions, but generally only see things I am tagged in)

Atsme added a subscriber: Atsme.Dec 21 2015, 9:10 PM

Just figured out why I suddenly started seeing Related Articles below the category line. I love it - great feature - but would like to have more control over what articles pop-up. Also, can it be alternating with each load? Thanks for the new toy!! ~~~~

Atsme: so interesting! If we keep it generated by a machine, than rotating the articles is more achievable (and an interesting idea we hadn't considered). However, we have also made it so that editors can select the articles they would like to show up by adding this tag to the page:

{{#related:new page title1}}
{{#related:new page title2}}
{{#related:new page title3}}

Jdlrobson renamed this task from [EPIC] Read more related articles after end of current article to [EPIC] Push Read more related articles to stable.Dec 22 2015, 10:31 PM
Jdlrobson reopened this task as Open.

Hi, I created a new ticket T122260 to track the move from related articles in beta to stable
Are you guys okay with using that as a separate task? I think we should recognize that this goal was accomplished in Q2

JKatzWMF renamed this task from [EPIC] Push Read more related articles to stable to [EPIC] Read more related articles after end of current article.Dec 23 2015, 4:16 AM
JKatzWMF closed this task as Resolved.

Hi, I created a new ticket T122260 to track the move from related articles in beta to stable
Are you guys okay with using that as a separate task? I think we should recognize that this goal was accomplished in Q2

Sure. This task tracked the Q2 goal of releasing this feature as an MFβ/beta feature, which we've done.

Thanks @phuedx and @JKatzWMF that makes more sense :)

This has to be deactivated for the German Wikipedia ASAP (and never should have been activated) since the German WP does have editoral rules with which this tool does not comply, namely w:de:Wikipedia:Themenring. Why you never ask before you invent such a tool (which in measures of sense does not deliver any useful content as well)

Why do you not accept, that Wikipedia is an encyclopaedic project and not a playground for developpers?????

This has to be deactivated for the German Wikipedia ASAP (and never should have been activated) since the German WP does have editoral rules with which this tool does not comply, namely w:de:Wikipedia:Themenring.

That's nonsense. Speaking as one of the authors of the page that you quote, it does not apply to this situation at all.
A more suitable comparison would be "Siehe auch" (see also) links, which have been an established practice on the German Wikipedia since at least 2003 - see https://de.wikipedia.org/wiki/Wikipedia:Assoziative_Verweise .

That's nonsense. Speaking as one of the authors of the page that you quote, it does not apply to this situation at all.

Off course this is the case. This is a classical Themenring. It does no matter wether such an unwanted navigation element is created by a temlate or by software.

A more suitable comparison would be "Siehe auch" (see also) links, which have been an established practice on the German Wikipedia since at least 2003 - see https://de.wikipedia.org/wiki/Wikipedia:Assoziative_Verweise .

Also under this rule the extension would be unwanted, since Assoziative Verweise are only allowed if they link to a continuative artile, e.g. in the article on the river Rhine to the List of rivers in Germany. A link to let's say Mannheim or Kölner Dom (which both are located directly on its banks) would be unwanted. And the latter is the principle this extension works.

Also Wikipedia:Assoziative Verweise and/or common sense are discouraging the inclusion of links to articles already linked earlier in the text.