Page MenuHomePhabricator

MCS/PCS endpoints not available for ~300 wikis
Open, LowPublic

Description

Based on the documentation here: https://www.mediawiki.org/wiki/Wikimedia_Apps/Team/RESTBase_services_for_apps#Routes

I would expect to find MCS running for WikiSpecies here: https://species.wikipedia.org/api/rest_v1/page/mobile-sections/Main_Page

Just like it does for EN: https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Main_Page

Does MCS run for species? If so, what is the url to use?

Issue for MWOffliner: https://github.com/openzim/mwoffliner/issues/767

Event Timeline

Isnit001 created this task.Jun 7 2019, 9:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Isnit001 added a subscriber: Kelson.Jun 7 2019, 9:15 AM
Isnit001 updated the task description. (Show Details)Jun 7 2019, 9:25 AM
Kelson added a comment.Jun 7 2019, 9:42 AM

This problem stops us to provide a new offline version of Wikispecies.

bearND added a subscriber: bearND.

Services team, could we add MCS for WikiSpecies?

mobrovac added a subscriber: mobrovac.

Yes, we could, but likely with an ETA of 2 months. When you say MCS, does that include all of MCS-backed end points?

bearND added a comment.EditedJun 7 2019, 3:29 PM

I think we'd like this is for all MCS/PCS endpoints (mobile-sections, mobile-html, and related PCS JSON and CSS/JS endpoints).

@mobrovac We're only actually using mobile-sections in MWOffliner. So the mobile-sections endpoint is our "priority" :)

@mobrovac We're only actually using mobile-sections in MWOffliner. So the mobile-sections endpoint is our "priority" :)

I am aware of that, but we have to strive for consistency across our projects. And actually, technically speaking, allowing only mobile-sections for species.wm.org would take more time than enabling all MCS-backed end points :)

The reason why I mentioned mobile-html and the other PCS endpoints is that those will replace mobile-sections in the future.

BTW, it.wikibooks.org seems to be also impacted. Do we have a list somewhere of all the wikis where the mobile API endpoints are not "properly" available?

BTW, it.wikibooks.org seems to be also impacted. Do we have a list somewhere of all the wikis where the mobile API endpoints are not "properly" available?

The list of domains and their config type can be found here. All domains that have the default_project type associated to them don't expose the mobile APIs. That invcludes all wikibooks, wikinews, wikiversities, wikiquotes and wikisources.

@mobrovac Thx, so I see that this concerns 307 web sites and for us, as around 20 are meta web sites that we don't want to scrape anyway, this is around 280 wikis we hardly can scrape.

$ curl "https://github.com/wikimedia/mediawiki-services-restbase-deploy/blob/4b34370ea3f80ac0995333b756897e158d4f5a3d/scap/vars.yaml" 2> /dev/null | grep default_project | wc
    307    2763   51494
Kelson renamed this task from MCS Url on WikiSpecies to MCS/PCS endpoints not available for ~300 wikis.Jun 15 2019, 7:13 AM

@mobrovac Thx, so I see that this concerns 307 web sites and for us, as around 20 are meta web sites that we don't want to scrape anyway, this is around 280 wikis we hardly can scrape.

The mobile end points are purpose-built and targeted towards specific projects. Unfortunately, due to the very different formatting conventions used by various projects, we can't guarantee meaningful results for all projects. @bearND and @Mholloway can comment on the specifics.

The mobile-sections endpoint and most of the other MCS/PCS endpoints were built to support the Wikipedia apps. So, most endpoints have only been tested with *.wikipedia.org domains. (The exception is the definitions endpoint but you probably don't care about that one.)
May I ask what features of mobile-sections you are most interested in and are the reason for not using Parsoid directly?