Remove noimages transform (was inconsistent parameter names for action=mobileview when you compare with action=parse)
Open, Stalled, NormalPublic

Description

Our api supports a noimages parameter which was built for WMF but no longer being used inside WMF. It's hard to justify keeping this around in the state the code is so we should look to remove it.

Precursors

We'll want to check usage per T99009#4747024 before removing this code and consider deprecation notices to existing API users if the number is significant.

Original bug report

The core ApiParse supports disablepp, disableeditsection and disabletoc, but MobileFrontend adds noimages.
We should deprecate the noimages parameter – since we're extending a core API – in favour of the new and improved disableimages parameter.

Developer notes

Per https://phabricator.wikimedia.org/T99009#4746807 no WMF clients are using the noimages parameter.
There might be users outside WMF but unmaintained code should be avoided.
Let's remove it altogether.

Ricordisamoa updated the task description. (Show Details)
Ricordisamoa raised the priority of this task from to Needs Triage.
Ricordisamoa added a project: MediaWiki-API.
Ricordisamoa added a subscriber: Ricordisamoa.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 13 2015, 9:03 PM
Anomie added a subscriber: Anomie.May 14 2015, 3:12 AM

ApiParse does not have a "noimages" parameter.

MobileFrontend does add such a parameter, though.

Anomie set Security to None.
Restricted Application added a project: Readers-Web-Backlog. · View Herald TranscriptMay 14 2015, 3:12 AM

@Anomie sorry for having overlooked that!

Ricordisamoa updated the task description. (Show Details)May 14 2015, 9:53 AM
phuedx added a subscriber: phuedx.May 15 2015, 4:08 PM

Deprecate the noimages parameter – since we're extending a core API – in favour of the new and improved disableimages parameter.

phuedx added subscribers: BGerstle-WMF, bearND.

Mobile-Apps folks (@BGerstle-WMF, @bearND) : do you use the noimages parameter in any of your requests?

@phuedx Yes, the Android app uses that parameter if the user has selected to not show images.

OK. Then part of this task should be ensuring that Mobile-Apps is notified that the noimages parameter has been deprecated.

Jdlrobson updated the task description. (Show Details)Aug 4 2015, 6:37 PM
Jdlrobson moved this task from Backlog to Bugs on the MobileFrontend board.
Jhernandez triaged this task as Normal priority.Aug 17 2015, 11:19 AM

I found this ticket while doing a bit of grooming on the neglected Mobile-Apps board a week or so ago, and I have to admit I'm a bit confused. The noimagesparameter isn't marked as deprecated in MobileFrontend, and there is no disableimages parameter (though there is apparently a cookie of that name). We've been happily using noimages for the year+ since this ticket was filed.

Is it still valid? Should we stop using noimages?

OK, I think I finally understand what's going on. This was originally a task about naming consistency, but then took a 90-degree turn and became about not using the noimages param. disableImages is a cookie and always has been, and we should use it instead of noimages (presumably to avoid splitting the cache).

For anyone with expertise and opinions on such matters, a friendly debate inspired by this task over how to handle image disabling in the Android app is happening over at https://gerrit.wikimedia.org/r/#/c/293457/.

@Mholloway: This task isn't that noimages is deprecated, it's that someone thinks that it should be deprecated and disableimages created to be used instead for consistency with the core ApiParse parameters.

If a "disableImages" cookie changes how the page is parsed, it needs to split the cache (and then I'd say a cookie isn't the right way to do it). Otherwise visitor A comes along with the cookie set, the lacking-images response is cached, then visitor B comes along without the cookie but gets the cached output that lacks images anyway. Or vice versa.

@Anomie Oh, I was under the impression from T120151 that we (now) knew how to vary the cache based on disableImages.

(I'm sure my understanding of cache variation is incomplete at best, but I think this means multiple copies of a given page that vary on specific criteria are cached to be served from the same URL.)

Yes, it appears from T120151 that the cache is varied on that cookie; I was mainly responding to your "presumably to avoid splitting the cache" comment there, which is the exact opposite of what T120151 was about.

OK, I think I understand.

In any case, there doesn't appear to be anything else for us to do here other than note that deprecating noimages is under discussion; noted. (For the record, the iOS app doesn't use it.)

Jdlrobson renamed this task from Inconsistent parameter names for action=parse to Inconsistent parameter names for action=mobileview when you compare with action=parse.Aug 1 2017, 4:40 PM

Change 402130 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Remove noimages flag from MobileView API

https://gerrit.wikimedia.org/r/402130

Change 402130 abandoned by Jdlrobson:
Remove noimages flag from MobileView API

https://gerrit.wikimedia.org/r/402130

Change 402130 restored by Jdlrobson:
Remove noimages flag from MobileView API

https://gerrit.wikimedia.org/r/402130

@JoeWalsh what would it take to get rid of the noimages parameter (and eventually the entire mobileview api?)
https://gerrit.wikimedia.org/r/402130 does the removal, but I'm guessing we need to wean certain clients off this API, first?

As far as I know, the noimages parameter is not currently in use by any WMF clients. The Android app stopped using it about a year ago (https://gerrit.wikimedia.org/r/#/c/apps/android/wikipedia/+/389898/), the iOS app never used it (AFAIK), and we're not using it in our lingering mobileview request in MCS.

Jdlrobson renamed this task from Inconsistent parameter names for action=mobileview when you compare with action=parse to Remove noimages transform (was inconsistent parameter names for action=mobileview when you compare with action=parse).Wed, Nov 14, 4:11 PM
Jdlrobson updated the task description. (Show Details)

Thanks @Mholloway I guess this falls on my team then to throw this away. Knowing when we can deprecate the mobileview API (TT186627) from your perspective would also be great to know!

AFAICT there's one client that still uses passes a noimages parameter to the parse API: https://github.com/wikimedia/labs-tools-wikidipendenza

@JoeWalsh what would it take to get rid of the noimages parameter (and eventually the entire mobileview api?)
https://gerrit.wikimedia.org/r/402130 does the removal, but I'm guessing we need to wean certain clients off this API, first?

Before you can wean clients off the API (if that's even necessary), you need to know who/what they are. Since API requests are web requests, usages of the parse API or the mobileview API with the noimages parameter set will be in the wmf.webrequest table. My recommendation would be to look there first.

Jdlrobson updated the task description. (Show Details)Wed, Nov 14, 4:45 PM

Traffic to the api with noimages flag for single day:

select count(*) from webrequest where year = 2018 and month = 11 and day = 1 and uri_query LIKE '%noimages%' AND uri_path LIKE '%api.php%' AND uri_query LIKE "%action=mobileview%";

208, 410

How many of those were unique ip?

select count(distinct client_ip) from webrequest where year = 2018 and month = 11 and day = 1 and uri_query LIKE '%noimages%' AND uri_path LIKE '%api.php%' AND uri_query LIKE "%action=mobileview%";

It turns out a much smaller amount:
37,881

I took a look at user agents:

select user_agent, count(*) from webrequest where year = 2018 and month = 11 and day = 1 and uri_query LIKE '%noimages%' AND uri_path LIKE '%api.php%' AND uri_query LIKE "%action=mobileview%" group by user_agent;

The majority were from Wikipedia app user agent e.g. WikipediaApp v2.*.
Other user agents: Dalvik, http://gottcode.org/connectagram/ and okhttp (the biggest consumer - I'm guessing bots?)

When I cut out the app traffic it becomes clear Apps are not the biggest consumer. In fact the biggest consumer is okhttp (An HTTP & HTTP/2 client for Android and Java applications). Is this also coming indirectly from our apps?

Minus WikipediaApp traffic:

select count(*) from webrequest where year = 2018 and month = 11 and day = 1 and uri_query LIKE '%noimages%' AND uri_path LIKE '%api.php%' AND uri_query LIKE "%action=mobileview%" AND user_agent NOT LIKE 'WikipediaApp/%';

204, 199

Removing the noimages param will send a warning to these clients that we've dropped it and they will still receive content - just they will now be given images where previously they were not (and it will be up to them to remove them). Given the low usage, it feels like we should just cut this off.

Agreed, @phuedx @Mholloway @JoeWalsh ?

If not, the alternative is to mark this a deprecated and revisit this in a few months.

When I cut out the app traffic it becomes clear Apps are not the biggest consumer. In fact the biggest consumer is okhttp (An HTTP & HTTP/2 client for Android and Java applications). Is this also coming indirectly from our apps?

Our Android app does use okhttp as its HTTP client library, but it's very (deservedly) popular, so it's not clear to me that the requests with 'okhttp' user agents are coming from us. I'm not sure of any case in which that would be sent and not our expected WikipediaApp UA. And those numbers sound quite high at least for our app, considering that in the past @Dbrant looked and found that only a tiny fraction of users had actually chosen to disable images.

Removing the noimages param will send a warning to these clients that we've dropped it and they will still receive content - just they will now be given images where previously they were not (and it will be up to them to remove them). Given the low usage, it feels like we should just cut this off.

Agreed, @phuedx @Mholloway @JoeWalsh ?

I'd better defer to @Dbrant for the official word as regards the Android app. It's a bit touchy because there are certain users who are not able to update their app version because their devices don't run a sufficiently current Android version, and they'll suddenly start receiving images despite not wanting to (presumably to save on data). At the same time, we can't run legacy code forever to support a tiny number of very outdated Android devices, and we have to draw the line somewhere.

Thanks, this is definitely safe to remove from our end. We removed our reliance on the noimages parameter over a year ago, and even then its usage was minuscule. Any remaining users of app versions older than 1 year should update to the latest, or update their OS to accept newer versions.

Okay I've setup T210808 and have created https://www.mediawiki.org/wiki/Extension:MobileFrontend/MobileViewAPI to start plotting a road map here for making this happen. This is stalled until T210808 has happened.

Jdlrobson changed the task status from Open to Stalled.Thu, Nov 29, 11:11 PM

This is stalled until T210808 has happened.

Is this correct? We've confirmed that the mobile apps don't rely on the noimages parameter, not that they don't rely on the mobileview API.

bearND added a comment.EditedWed, Dec 5, 5:13 AM

action=mobileview is still used by the apps, and by MCS (for the main pages).

phuedx added a subscriber: Jhernandez.EditedWed, Dec 5, 2:47 PM

Here are my notes from Monday's Engineering Managers meeting:

phuedx added a comment.Wed, Dec 5, 2:51 PM

☝️ It looks like deprecating and removing the noimages parameter is possible now and deprecating the mobileview API query module is possible but removing it is blocked on a lot of work (most of which is currently being worked on).