Create endpoint for "Anniversaries"
Closed, ResolvedPublic
Actions

Description

The content for this endpoint will come from parsing existing wiki pages that are currently maintained by the editing community.

The API should return data it derives from 2 different pages each day:

Selected Anniversaries pages

The current day can be seen here:
https://en.wikipedia.org/wiki/Wikipedia:On_this_day/Today
Which injected with data from the selected anniversaries of the current day (for example, August 19):
https://en.wikipedia.org/wiki/Wikipedia:Selected_anniversaries/August_19

Day of the Year pages

Every day of the year has a page like this:
https://en.wikipedia.org/wiki/May_20

@Mhurd has written some native parsing logic here:
https://github.com/wikimedia/wikipedia-ios/pull/701/files

Monte's structures seem pretty good. The top level dictionary would be something like this

{
    selectedAnniversaries: anniversaries,
    events: events,
    births: births,
    deaths: deaths,
    holidays: holidays
};

Where each item in the dictionary is an array of objects like this:

{
    year: 1912,
    yearPage: yearWikiPage,
    text: textExcludingYear,
    page: mainWikiPage,
    otherPages: otherWikiPages
};

Each page should be like other pages in MCS with thumbnails, wikidata description, etc…

Details

	Subject	Repo	Branch	Lines +/-
	"On This Day" endpoints	mediawiki/services/mobileapps	master	+2 K -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• bearND	T143408 Create endpoint for "Anniversaries"
Resolved	Bawolff	T153088 Security Review of On This Day Endpoint
Resolved	• Pchelolo	T143420 Make public "Anniversaries" endpoint
Resolved	• bearND	T157647 Anniversaries crash for unsupported wikis

Event Timeline

• Fjalapeno created this task.Aug 19 2016, 12:55 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2016, 12:55 PM

• Fjalapeno mentioned this in T143420: Make public "Anniversaries" endpoint.Aug 19 2016, 2:50 PM

• JMinor subscribed.Aug 19 2016, 4:55 PM

• JMinor moved this task from Needs Triage to Product Backlog on the Wikipedia-iOS-App-Backlog board.Aug 19 2016, 5:03 PM

• bearND moved this task from Incoming to Backlog on the Mobile-Content-Service board.Aug 19 2016, 7:58 PM

• Mholloway subscribed.Aug 25 2016, 4:09 PM

• bearND updated the task description. (Show Details)Aug 25 2016, 5:00 PM

• JMinor moved this task from Product Backlog to Tracking on the Wikipedia-iOS-App-Backlog board.Sep 2 2016, 12:26 AM

• JMinor mentioned this in T135771: Create fetcher for en day pages.Sep 2 2016, 12:52 AM

• Fjalapeno added a subtask: T153088: Security Review of On This Day Endpoint.Dec 13 2016, 4:41 PM

TODOs from a meeting with @Mhurd and @Fjalapeno:

Instead of title use the $merge property. $merge: [ mUtil.getRbPageSummaryUrl(app, domain, dbTitle) ]. Example: rGMOAa359e3d3922bd3c992a8ae82267ccf0152ec0512
The revision for the etag could be retrieved using the revision from Parsoid. See getRevisionFromEtag() in the code base.
Consider creating an all type of endpoint with includes selectedAnniversaries, events, births, deaths, holidays. I think RESTBase could just use that one to load the backing page via the Parsoid endpoint once, parse all the subsections it needs and store all On this day portions in the same bucket. (@Pchelolo to confirm).
Consider moving the meat of the implementation into a library file for easy unit testing. The different routes can stay in the route file you already have. (No need to create different route files.)
Add unit tests.

Any of theses could be done in a subsequent patch as well. Up to you, @Mhurd.

Mhurd claimed this task.Dec 21 2016, 11:48 PM

Mhurd added a project: iOS-app-v5.4.0-Hiking-Boot.Dec 26 2016, 9:05 AM

Mhurd moved this task from Tasks from Product Backlog to Doing on the iOS-app-v5.4.0-Hiking-Boot board.

@bearND @Fjalapeno

Hey guys I didn't get the tests finished last week but I was able to implement some pretty good coverage over the weekend.

Did a bit of refactoring too - cleaning up the promise chaining etc.

Going to play with the '$merge' hydration bits today...

I got title '$merge' hydration working. I'll push to Gerrit soon.

I want to add a couple hydration tests first and take a hack at adding etags, per @bearND's comment.
I'll try to update the documentation as well.

For now my WIP branch is here:
https://github.com/montehurd/mediawiki-services-mobileapps/commits/on-this-day-endpoint

How will this scale to other wikis? Have you considered using MediaWiki-extensions-FeaturedFeeds?

@Nemo_bis we are looking into this on some other tasks now along with In the News and the Featured article: https://phabricator.wikimedia.org/T150806

Do you know how many wikis that extension is active on?

Mhurd moved this task from Doing to Needs Code Review on the iOS-app-v5.4.0-Hiking-Boot board.Jan 6 2017, 12:47 AM

Here's my patch:
https://gerrit.wikimedia.org/r/#/c/330820/

And a commit which makes it work with the older version of node which production is using:
https://gerrit.wikimedia.org/r/#/c/330828/
( After node gets updated https://phabricator.wikimedia.org/T149331#2867141 I'd like to revert that last commit. )
( Edit: I forgot the Gerrit workflow prefers amendment. I amended the original and abandoned this separate commit. )

In T143408#2919628, @Fjalapeno wrote:

@Nemo_bis we are looking into this on some other tasks now along with In the News and the Featured article: https://phabricator.wikimedia.org/T150806

Do you know how many wikis that extension is active on?

All wikis. The content of such feeds must be defined on-wiki, so that the feature naturally applies to all wikis.

• Pchelolo mentioned this in T154797: Support On This Day endpoint in RESTBase.Jan 6 2017, 7:27 PM

• Fjalapeno claimed this task.Jan 19 2017, 2:17 AM

• Fjalapeno moved this task from Backlog to Kanban on the Mobile-Content-Service board.

• Fjalapeno edited projects, added Mobile-Content-Service (Kanban); removed Mobile-Content-Service.

• Fjalapeno moved this task from To Do to Doing on the Mobile-Content-Service (Kanban) board.

• Fjalapeno moved this task from Doing to Code Review on the Mobile-Content-Service (Kanban) board.

Notes from handoff:

What needs to be finished?

Languages abstracted
Potentially tests for languages with fixtures
Address Micahels comments
Hydraton needs to be added in RESTBase

Any open questions?

How to name the selected endpoint?
endpoint naming
All endpint?
is topic "flag"
Negative sorting?

Location of language patch?

Found, in code

Node

updated for 6 (promises)

Other Notes:

use lang links on desktop to get other lang version for month names for lang
used to use css selector, but they can't handle how german and other langs which intersperse h3's
prob need tests for listElementsByHeadingID (and yearListElementRegEx?) and BC for other langs
prob need fixture files for other langs (now just DE and EN)
need stronger modeling for lang info? a controller which vends lang objects?
gabriel said ok to use built-in promises instead of BB for things which aren't perf critical
aside: neat Node 6 upgrade heap memory image: https://phabricator.wikimedia.org/T149331#2948472


https://de.wikipedia.org/wiki/20._Mai

• Fjalapeno renamed this task from Create endpoint for "On This Day" to Create endpoint for "Anniversaries".Jan 23 2017, 7:59 PM

@bearND renamed the endpoint on the ticket per the discussion in gerrit on "anniversaries"

@bearND what do you think about the negative numbers for BC that @Mhurd was talking about at the hand off? Think we should do that?

In T143408#2963702, @Fjalapeno wrote:

@bearND renamed the endpoint on the ticket per the discussion in gerrit on "anniversaries"

Great. Sounds good to me.

@bearND what do you think about the negative numbers for BC that @Mhurd was talking about at the hand off? Think we should do that?

Not sure if we should expose it like that anymore. On one hand it's nice that it's an integer value if you want to use it for sorting, on the other hand it may look odd to some people if a client prints a negative number. Additionally, using a simple integer could be controversial and up to interpretation when one wants to be exact and not be off by 1 year. See https://en.wikipedia.org/wiki/0_(year).

I'm leaning towards:
Externally we should provide the string representation we find on the page.
Internally for sorting we should keep it a (potentially negative) integer.

@bearND keeping negative numbers internal sounds good to me!

• Fjalapeno reassigned this task from • Fjalapeno to • bearND.Jan 24 2017, 4:27 PM

• Fjalapeno removed a project: iOS-app-v5.4.0-Hiking-Boot.Jan 24 2017, 7:35 PM

Just in case it's not clear, this should still be blocked on using FeaturedFeeds (T148680: Investigate FeaturedFeeds).

@Nemo_bis not sure if you checked out the email from the other week, but we really need to get a version we can evaluate before we invest in using feature feeds or specialized mark up or a combination of both.

The featured feed and markup tickets are on going - likely they will solve for featured articles and in the news first (as they have been deployed and evaluated already), and then we will look to using similar techniques for the anniversaries endpoint. The good news is that since we will have developed a technique for scaling both featured articles and in the news across projects, it should be easier to adapt this strategy to the anniversaries endpoint.

• Fjalapeno added a subtask: T143420: Make public "Anniversaries" endpoint.Jan 27 2017, 6:33 PM

• Fjalapeno added a project: Reading Epics (New Feed Content).

• bearND moved this task from Code Review to Doing on the Mobile-Content-Service (Kanban) board.Jan 31 2017, 7:33 AM

Change 330820 had a related patch set uploaded (by BearND):
"On This Day" endpoints

https://gerrit.wikimedia.org/r/330820

gerritbot added a project: Patch-For-Review.Feb 1 2017, 5:04 PM

• Fjalapeno mentioned this in T153088: Security Review of On This Day Endpoint.Feb 1 2017, 6:26 PM

• Fjalapeno mentioned this in T157059: Rename Mobile Content Service?.Feb 2 2017, 5:28 PM

• Pchelolo moved this task from Backlog to watching on the Services board.Feb 3 2017, 9:58 PM

• Pchelolo edited projects, added Services (watching); removed Services.

Change 330820 merged by jenkins-bot:
"On This Day" endpoints

https://gerrit.wikimedia.org/r/330820