Page MenuHomePhabricator

Create endpoint for "Anniversaries"
Closed, ResolvedPublic

Description

The content for this endpoint will come from parsing existing wiki pages that are currently maintained by the editing community.

The API should return data it derives from 2 different pages each day:

  1. Selected Anniversaries pages

The current day can be seen here:
https://en.wikipedia.org/wiki/Wikipedia:On_this_day/Today
Which injected with data from the selected anniversaries of the current day (for example, August 19):
https://en.wikipedia.org/wiki/Wikipedia:Selected_anniversaries/August_19

  1. Day of the Year pages

Every day of the year has a page like this:
https://en.wikipedia.org/wiki/May_20

@Mhurd has written some native parsing logic here:
https://github.com/wikimedia/wikipedia-ios/pull/701/files

Monte's structures seem pretty good. The top level dictionary would be something like this

{
    selectedAnniversaries: anniversaries,
    events: events,
    births: births,
    deaths: deaths,
    holidays: holidays
};

Where each item in the dictionary is an array of objects like this:

{
    year: 1912,
    yearPage: yearWikiPage,
    text: textExcludingYear,
    page: mainWikiPage,
    otherPages: otherWikiPages
};

Each page should be like other pages in MCS with thumbnails, wikidata description, etc…

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2016, 12:55 PM
JMinor added a subscriber: JMinor.Aug 19 2016, 4:55 PM
bearND updated the task description. (Show Details)Aug 25 2016, 5:00 PM
bearND added a subscriber: Pchelolo.

TODOs from a meeting with @Mhurd and @Fjalapeno:

  • Instead of title use the $merge property. $merge: [ mUtil.getRbPageSummaryUrl(app, domain, dbTitle) ]. Example: rGMOAa359e3d3922bd3c992a8ae82267ccf0152ec0512
  • The revision for the etag could be retrieved using the revision from Parsoid. See getRevisionFromEtag() in the code base.
  • Consider creating an all type of endpoint with includes selectedAnniversaries, events, births, deaths, holidays. I think RESTBase could just use that one to load the backing page via the Parsoid endpoint once, parse all the subsections it needs and store all On this day portions in the same bucket. (@Pchelolo to confirm).
  • Consider moving the meat of the implementation into a library file for easy unit testing. The different routes can stay in the route file you already have. (No need to create different route files.)
  • Add unit tests.

Any of theses could be done in a subsequent patch as well. Up to you, @Mhurd.

Mhurd claimed this task.Dec 21 2016, 11:48 PM

@bearND @Fjalapeno

Hey guys I didn't get the tests finished last week but I was able to implement some pretty good coverage over the weekend.

Did a bit of refactoring too - cleaning up the promise chaining etc.

Going to play with the '$merge' hydration bits today...

Mhurd added a comment.Dec 27 2016, 4:00 AM

I got title '$merge' hydration working. I'll push to Gerrit soon.

I want to add a couple hydration tests first and take a hack at adding etags, per @bearND's comment.
I'll try to update the documentation as well.

For now my WIP branch is here:
https://github.com/montehurd/mediawiki-services-mobileapps/commits/on-this-day-endpoint

How will this scale to other wikis? Have you considered using MediaWiki-extensions-FeaturedFeeds?

@Nemo_bis we are looking into this on some other tasks now along with In the News and the Featured article: https://phabricator.wikimedia.org/T150806

Do you know how many wikis that extension is active on?

Mhurd added a comment.EditedJan 6 2017, 12:50 AM

Here's my patch:
https://gerrit.wikimedia.org/r/#/c/330820/

And a commit which makes it work with the older version of node which production is using:
https://gerrit.wikimedia.org/r/#/c/330828/
( After node gets updated https://phabricator.wikimedia.org/T149331#2867141 I'd like to revert that last commit. )
( Edit: I forgot the Gerrit workflow prefers amendment. I amended the original and abandoned this separate commit. )

@Nemo_bis we are looking into this on some other tasks now along with In the News and the Featured article: https://phabricator.wikimedia.org/T150806

Do you know how many wikis that extension is active on?

All wikis. The content of such feeds must be defined on-wiki, so that the feature naturally applies to all wikis.

Fjalapeno moved this task from To Do to Doing on the Mobile-Content-Service (Kanban) board.
Fjalapeno moved this task from Doing to Code Review on the Mobile-Content-Service (Kanban) board.
Fjalapeno claimed this task.
Fjalapeno added a comment.EditedJan 19 2017, 6:56 PM

Notes from handoff:

What needs to be finished?

  • Languages abstracted
  • Potentially tests for languages with fixtures
  • Address Micahels comments
  • Hydraton needs to be added in RESTBase

Any open questions?

  • How to name the selected endpoint?
  • endpoint naming
  • All endpint?
  • is topic "flag"
  • Negative sorting?

Location of language patch?

  • Found, in code

Node

  • updated for 6 (promises)

Other Notes:

  • use lang links on desktop to get other lang version for month names for lang
  • used to use css selector, but they can't handle how german and other langs which intersperse h3's
  • prob need tests for listElementsByHeadingID (and yearListElementRegEx?) and BC for other langs
  • prob need fixture files for other langs (now just DE and EN)
  • need stronger modeling for lang info? a controller which vends lang objects?
  • gabriel said ok to use built-in promises instead of BB for things which aren't perf critical
  • aside: neat Node 6 upgrade heap memory image: https://phabricator.wikimedia.org/T149331#2948472

https://de.wikipedia.org/wiki/20._Mai
Fjalapeno renamed this task from Create endpoint for "On This Day" to Create endpoint for "Anniversaries".Jan 23 2017, 7:59 PM

@bearND renamed the endpoint on the ticket per the discussion in gerrit on "anniversaries"

@bearND what do you think about the negative numbers for BC that @Mhurd was talking about at the hand off? Think we should do that?

@bearND renamed the endpoint on the ticket per the discussion in gerrit on "anniversaries"

Great. Sounds good to me.

@bearND what do you think about the negative numbers for BC that @Mhurd was talking about at the hand off? Think we should do that?

Not sure if we should expose it like that anymore. On one hand it's nice that it's an integer value if you want to use it for sorting, on the other hand it may look odd to some people if a client prints a negative number. Additionally, using a simple integer could be controversial and up to interpretation when one wants to be exact and not be off by 1 year. See https://en.wikipedia.org/wiki/0_(year).

I'm leaning towards:
Externally we should provide the string representation we find on the page.
Internally for sorting we should keep it a (potentially negative) integer.

@bearND keeping negative numbers internal sounds good to me!

Fjalapeno reassigned this task from Fjalapeno to bearND.Jan 24 2017, 4:27 PM
Nemo_bis added a comment.EditedJan 26 2017, 9:01 AM

Just in case it's not clear, this should still be blocked on using FeaturedFeeds (T148680: Investigate FeaturedFeeds).

@Nemo_bis not sure if you checked out the email from the other week, but we really need to get a version we can evaluate before we invest in using feature feeds or specialized mark up or a combination of both.

The featured feed and markup tickets are on going - likely they will solve for featured articles and in the news first (as they have been deployed and evaluated already), and then we will look to using similar techniques for the anniversaries endpoint. The good news is that since we will have developed a technique for scaling both featured articles and in the news across projects, it should be easier to adapt this strategy to the anniversaries endpoint.

Change 330820 had a related patch set uploaded (by BearND):
"On This Day" endpoints

https://gerrit.wikimedia.org/r/330820

Pchelolo moved this task from Backlog to watching on the Services board.Feb 3 2017, 9:58 PM
Pchelolo edited projects, added Services (watching); removed Services.

Change 330820 merged by jenkins-bot:
"On This Day" endpoints

https://gerrit.wikimedia.org/r/330820

bearND closed this task as Resolved.Feb 27 2017, 10:28 PM