Page MenuHomePhabricator

Create first public endpoints for feeds
Closed, ResolvedPublic

Description

We've deployed all currently planned feed endpoints:

  • feed/featured (=aggregated endpoint)
  • page/news
  • page/featured
  • page/most-read
  • page/random/summary
  • media/image/featured

Would you add the corresponding mapping to expose these publicly? We would like to be able to hit the endpoints from the apps.
We want to release a beta version of the Android app which accesses the feed by July 1st, 2016.

We could have very basic caching for now, and refine later. Change Propagation and storage or pre-generation of results can be set up later.

Strawman proposal for basic caching:

  • feed/featured: 1 minute
  • page/news: 1 minute
  • page/featured: 1 hour
  • page/most-read: 1 hour
  • page/random/summary: no caching
  • media/image/featured: 1 hour

Event Timeline

mobrovac subscribed.

For the sake of simplicity, would it be acceptable for you to target the labs instance while you conduct the experiments and tweak the output?

@mobrovac With yesterday's deploy I think the endpoint is stable enough in terms of output format that it would be great to have it available on the production RB cluster.
While possible, switching the app to use labs is a bit cumbersome.

I've updated the description to reflect the latest deployment. All mentioned endpoints are deployed now and the Android team is very eager to get the public endpoints for them in the next couple of days. Would this be possible?

@GWicke via email:

We have some concerns with how the end points currently return content directly, rather than redirecting. This complicates caching (basically can't purge random responses) and documentation. Our preference would be to respond with a redirect to a random title instead.

Initially, we would be happy with just the following endpoints made public:

  • feed/featured: 1 minute caching
  • page/random/summary: no caching

The aggregated endpoint (feed/featured) provides unique content, so a redirect doesn't apply here. The random endpoint doesn't get storage nor caching at this time, so a redirect doesn't seem critical. Still, I'm going to look into adding an implementation for this (T139424) to better wrap my head around this.

The next priority is probably page/most-read. Again, this emits unique content, so redirects don't apply.
Same for media/image/featured and page/news.

page/featured is one that could redirect but that one doesn't get updated very often, so the benefits are not great. Let's wait until we have the random endpoint using the redirect before proceeding with this one.

So, I've started a patch which would do the redirect solution for the random endpoint.
There are a couple of issues with using the summary output:

  • It doesn't include a Wikidata description. I'll need to check with designers if we really need this.
  • The thumbnail is too small. This might be alleviated once we get a smarter image URL service. Until then this seems to be blocked. (We're only comfortable lowering the resolution but not increasing it on the client side since we won't know if a particular image has a higher res thumbnail available.)

Why does page/random/summary need to differ from page/summary ? It seems more consistent for both to have the exact same format for consistency. that would also allow redirects to be transparent. If there are things that need improvement / changes, I think we should be working on them for page/summary instead of having two different code paths which return different results for essentially the same thing.

Yes, I would like them to be the same, too. It's just for the feed the apps want a higher resolution thumbnail but for link previews the apps want a smaller resolution thumbnail. That was the thinking up to now, at least.

I just learned that for the random endpoint the Android app only needs the title. So, I'm going to add a title endpoint and deprecate the summary endpoint.

So, we should be good on the random endpoint then. If we get that one and the aggregated feed endpoint, the Android app would be ready to go.

@Mholloway has a PR open to add descriptions to the summary endpoint BTW. Once that lands and we get the smart thumbnail service the summary should make it back in there. An easier and more immediate endpoint would be a page/random/mobile-sections-lead. That one doesn't have the issues we experience with summary endpoints.

I just learned that for the random endpoint the Android app only needs the title. So, I'm going to add a title endpoint and deprecate the summary endpoint.

So page/random/title instead of page/random/summary ? It does make sense since then that title can be used by consumers for any other endpoint (be it summary or mobile-sections or html). We can't make a redirect in that case, though.

So, we should be good on the random endpoint then. If we get that one and the aggregated feed endpoint, the Android app would be ready to go.

@Mholloway has a PR open to add descriptions to the summary endpoint BTW. Once that lands and we get the smart thumbnail service the summary should make it back in there. An easier and more immediate endpoint would be a page/random/mobile-sections-lead. That one doesn't have the issues we experience with summary endpoints.

Actually, page/random/{format} where {format} is any of the URIs under /page/ makes sense to me. That way, we could have only one routine calculating the random title name and then have a proper redirect. Thoughts?

We can't make a redirect in that case, though.

That's ok.

Actually, page/random/{format} where {format} is any of the URIs under /page/ makes sense to me. That way, we could have only one routine calculating the random title name and then have a proper redirect. Thoughts?

We can add more once we actually need them. I'm thinking we could have format = {title, summary, mobile-sections-lead}. For now, let's just focus on the ones we need right now, which is title for random. My patch for page/random/title is ready for review. I hope we can merge it before the deploy window today.

Currently we just reuse the same code and run through the logic again for the aggregated endpoint. I'm not sure how we would handle redirects for the aggregated endpoint. We can't just do a single redirect. It combines several endpoints.

I'm thinking we could do the same also for the featured article of the day. page/featured/{format}/{yyyy}/{mm}/{dd} instead of just page/featured/{yyyy}/{mm}/{dd}.

/feed/featuredand /page/random/{format} have been deployed and are now live in prod !

bearND claimed this task.