Page MenuHomePhabricator

Remove summary api_urls field
Closed, ResolvedPublic

Description

Some of the endpoints referenced in the summary endpoint under api_urls are going away. It seems that these fields are not used nor useful.

Example of fields be removed

"api_urls": {
  "summary": "https://en.wikipedia.org/api/rest_v1/page/summary/Cat",
  "metadata": "https://en.wikipedia.org/api/rest_v1/page/metadata/Cat",
  "references": "https://en.wikipedia.org/api/rest_v1/page/references/Cat",
  "media": "https://en.wikipedia.org/api/rest_v1/page/media/Cat",
  "edit_html": "https://en.wikipedia.org/api/rest_v1/page/html/Cat",
  "talk_page_html": "https://en.wikipedia.org/api/rest_v1/page/html/Talk:Cat"
}

We'll should bump the version as well.

Event Timeline

LGoto renamed this task from Update summary appi_urls to Update summary api_urls.Mar 18 2020, 3:52 PM
LGoto triaged this task as Medium priority.
bearND removed bearND as the assignee of this task.
bearND claimed this task.
bearND updated the task description. (Show Details)
bearND updated the task description. (Show Details)

@phuedx or @Jdlrobson Do you know if the api_urls in the summary endpoint are used by the web team? (Are these fields useful?) I don't see them mentioned in https://www.mediawiki.org/wiki/Page_Previews/API_Specification. So, probably not but I wanted to make sure.

@Pchelolo If we only remove the api_urls to already undeployed endpoints (metadata, media, references) is a minor version bump enough?
(https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_summary__title_)

@bearND up to you, I think it would be fine.

@phuedx or @Jdlrobson Do you know if the api_urls in the summary endpoint are used by the web team? (Are these fields useful?) I don't see them mentioned in https://www.mediawiki.org/wiki/Page_Previews/API_Specification. So, probably not but I wanted to make sure.

We're not using them no. Smaller summary endpoint responses would also be better for users (not sure how significantly they contribute to the payload).

Change 585039 had a related patch set uploaded (by BearND; owner: BearND):
[mediawiki/services/mobileapps@master] Update summary api_urls, part 2

https://gerrit.wikimedia.org/r/585039

Change 585039 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Update summary api_urls, part 2

https://gerrit.wikimedia.org/r/585039

dr0ptp4kt subscribed.

I've requested that we hold off just temporarily while we work out how to communicate the change. One or more of the fields are already pointing at things that don't actually work, so it might not really be a problem in the real world (or if it is no one's noticing), but we'll work out the process, follow that process, and then commence to rolling this out.

Change 585287 had a related patch set uploaded (by BearND; owner: BearND):
[mediawiki/services/mobileapps@master] Revert changes to summary api_urls

https://gerrit.wikimedia.org/r/585287

Change 585287 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Revert changes to summary api_urls

https://gerrit.wikimedia.org/r/585287

Verbiage to be discussed in a meeting tomorrow.

Re-blocking on the sending of an announcement email and the passage of the announced notice period.

bearND renamed this task from Update summary api_urls to Remove summary api_urls field.May 29 2020, 11:41 PM
bearND updated the task description. (Show Details)

Seems like this deprecation already went all the way through? We found out through breakage (e.g., https://en.wikipedia.org/api/rest_v1/page/references/The_Register-Guard/957644254 )

Was there verbiage we were supposed to have seen? Somewhere we should subscribe for updates?

@mahmoud, the decommission of some of the endpoints that are referenced from within api_urls has taken place already, yes. Those endpoints were deemed experimental and thus subject to change without notice. However, I'm sorry to learn that breakage was the way you spotted this.

However, there are some additional endpoints presently referred to from within api_urls that are deemed stable and will not be removed. The plan is to remove the api_urls field altogether, but keep the stable endpoints running.

I think an observation here is that if there are URIs embedded as server calculated URLs (as opposed to being UGC, where hyperlinks have no guarantee of stability by definition of it being UGC) with in API response, in the future we'll want to be a bit more cautious about removal of serving of the corresponding endpoints. Perhaps one thing to avoid is adding server calculated URLs to responses that refer to non-stable endpoints when the serving endpoint is itself stable, or at least adding verbose details to implementers in documentation about such fields.

@mahmoud, to clarify, have you been depending on the api_urls field to get URLs such as the references one you provided as an example? Would you please point us to corresponding code pegged to the latest commit using api_urls? It would be helpful to understand this particular use case. I forget if you and I have met, and so please accept my apology in case I know your code but forgot! I was thinking maybe this was Hatnote based on your GitHub profile, but wasn't sure.

We're planning to email updates about this forthcoming removal of the api_urls field soon, but are wrapping up final data analysis to ensure we have some level of visibility into the prospective additional change (the email will request people to update their code and provide an opportunity to let us know if there are massive clients dependent on api_urls for which a code update is problematic, so there's lead time for downstream API consumers).

We've been doing a postmortem around the unannounced decommission. If you haven't already done so, would you please ensure you're subscribed to the wikitech-l and mediawiki-api-announce mailing lists? We intend to share updates more rigorously there in the future.

Hey @dr0ptp4kt! Not sure if we've met, either, but nice to e-meet you here. This is for _a_ Hatnote project, but not one of the well-known ones. Here's a permalink to some Python in our Pacetrack project. I guess the main thing @Slaporte and I could hope for now is another straightforward endpoint where we could get similar citations?

@mahmoud I believe such an API was being explored in T247858: References API. @eprodromou , heads up in case this is on the nearish term roadmap.

Meantime, @JoeWalsh @ssastry is there a stable versioned endpoint response type that Pacetrack's code could iterate over for this sort of data in a reliable fashion? I was wondering if utilizing the old experimental references API code to iterate over the Parsoid data structure might be a decent interim approach.

This field has not been removed on master.[2] We still need to wait for the announced time to pass: on or after 14 July 2020.[1]

[1] https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2020-June/000154.html
[2] http://localhost:8888/en.wikipedia.org/v1/page/summary/Cat

@bearND would you please point us to the old code that processed references for the experimental API, the one that was decom'd? We looked around a bit and it seems like porting that into clients would probably be the most expedient way for clients to process stuff until there's a new stable References API (iOS is looking at this first).

@dr0ptp4kt The code was removed from mobileapps last week in this patch, the revert of which should show what we had for the references endpoint.

Thanks @bearND! If I'm reading the patch correctly, @mahmoud you may be able to access Parsoid HTML via the /page/html/{title} or ​/page​/html​/{title}​/{revision} route described at https://en.wikipedia.org/api/rest_v1/ and apply this sort of approach to replicate what used to be available in the experimental API (there may be some utility methods requiring some code spelunking, beware). Hope this helps some.

LGoto subscribed.

As per Adam's email: "We plan to remove the api_urls field described here on or after 14 July 2020."

As per Adam's email: "We plan to remove the api_urls field described here on or after 14 July 2020."

And I don't see any replies to that email on either list (wikitech-l or mediawiki-api-announce).

Change 615575 had a related patch set uploaded (by BearND; owner: BearND):
[mediawiki/services/mobileapps@master] summary: remove api_urls

https://gerrit.wikimedia.org/r/615575

I had to look back to this task just now to remember whether we agreed to update api_urls (as @MSantos's already-merged patch did) or remove it (as @bearND's patch does). From T247991#6304842, I see that removing it altogether is what was announced. I'll merge Bernd's patch now.

Change 615575 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] summary: remove api_urls

https://gerrit.wikimedia.org/r/615575