Page MenuHomePhabricator

Help English Wikivoyage restrict certain file type uploads to address illicit file uploads
Closed, ResolvedPublic

Description

Wikivoyagers on the English Wikivoyage have reported issues with file uploads being abused for pirated music and videos.

https://en.wikivoyage.org/wiki/Wikivoyage:Travellers%27_pub#File_Spam

I've asked the community to build consensus (as a mater of best practices and protocol) on which file uploads to restrict.

Event Timeline

Looking at the upload log on English Wikivoyage it looks like the following file types would be the most logical to restrict.

  • .ogg
  • .flac
  • . wav
  • .webm

You know, I was going to tag it as such @Framawiki but wasn't 100% if that was appropriate. I'm not sure how to discern if it's folk abusing Zero or just general abuse. Any suggestions?

Dispenser added a parent task: Restricted Task.Sep 25 2017, 9:43 PM

These are all known WP0 abusers--The persistent type. They'll wait a week, make edits, whatever to get passed the filters.

You can prevent those uploads with AbuseFilter. Commons has a few filters for that, linked in the description of T129845 (unfortunately it seems that the filters are private, you'll need to find a helpful Commons admin or look at them with a WMF user account with special 'staff' powers).

Again, the community is smarter than the individual. Thanks all. I was just notified that the English Wikivoyage's image policy outlines acceptable media formats and explicitly states they do not use audio/video files.

https://en.wikivoyage.org/wiki/Wikivoyage:Image_policy#Image_formats

https://en.wikivoyage.org/wiki/Wikivoyage:Image_policy#Other_media

So that part seems a little more clear to me. Looks like I need to find an admin. :)

...Commons has a few filters for that, linked in the description of T129845 (unfortunately it seems that the filters are private...

When writing new filters,AbuseFilter 166 is a nice reference with every file type. And Meta has global filters.

Taking into account the policy, shouldn't these file types be disabled in MediaWiki's configuration for Wikivoyage (en and others)? Just asking, I don't know MediaWiki's internals and don't know if it is even possible. Regarding Abuse Filter, this rule rejects the file types mentioned above:

(action == "upload" | action == "stashupload")
& (file_mime == "video/webm" | file_mime == "application/ogg" | file_mime = "audio/wav" | file_mime == "audio/x-flac")

Of course "Prevent the user from performing the action in question" should be turned on for the rule.

That community has a very clear policy on file types at https://en.wikivoyage.org/wiki/Wikivoyage:Image_policy#Image_formats

The only acceptable formats for the wiki are JPEGs, PNGs, and SVGs. The policy has been in place and supported by the community for years. The last serious discussion about image formats was in 2014, and the result was to continue supporting the policy and to explicitly ban animated GIFs.

It is likely (although someone would have to check) that identical rules are in place at most of the Wikivoyages, and given the belief that these abusers may be persistent, then it might be desirable to check all of the Wikivoyage's image policies and implement similar server-side restrictions for every Wikivoyage with an explicit local policy.

Taking into account the policy, shouldn't these file types be disabled in MediaWiki's configuration for Wikivoyage (en and others)? Just asking, I don't know MediaWiki's internals and don't know if it is even possible.

Oh, actually, yes. Our config is messy and makes this a little annoying, but you could probably add an override for wmgFileExtensions to disable all the extra formats we normally allow on WIkimedia wikis.

Of the 18 Wikivoyage, 8 have an explicit image policy. We could reach out to the rest and ask if it's OK to set this for all WV projects.

https://www.wikidata.org/wiki/Q6618850

Unanimous agreement for the English Wikivoyage is documented here: https://en.wikivoyage.org/wiki/Wikivoyage_talk:Image_policy#Proposal_to_change_Special:Upload_page

There are additional supportive comments at the local equivalent of the village pump. I've seen no opposition to this proposed config change, and as a practical matter, it would make things easier on the local admins if they didn't have to patrol file types that can't be used there.

I'll figure out how to do it properly and do this one, so that it's easier for future deployers to change this setting for other wikis.

Documented below is how the list of allowed file types is generated. Turns out that my "messy" and "a little annoying" were understatements and the configuration is in fact completely insane. This list may not be exhaustive, some of the entries listed below may be duplicated somewhere else (duplicates in the list of allowed extensions are ignored).

For reference, the currently allowed file types on most wikis (including English Wikivoyage) are: tiff, tif, png, gif, jpg, jpeg, webp, xcf, pdf, mid, ogg, ogv, svg, djvu, oga, flac, opus, wav, webm.


First, allowed extensions are gathered from the following sources, in this order:

  • Extensions using new-style extension registration:
    • TiffHandler: "FileExtensions": [ "tiff", "tif" ],
  • MediaWiki core defaults: $wgFileExtensions = [ 'png', 'gif', 'jpg', 'jpeg', 'webp' ];
  • WMF configuration:
    • 'wmgFileExtensions' => [ 'default' => [ 'xcf', 'pdf', 'mid', 'ogg', 'ogv', 'svg', 'djvu', ], ... ],
      • For foundationwiki only: [ 'otf', 'ai' ]
      • For incubatorwiki only: [ 'xml' ]
    • (additionally, on wikis with wmgPrivateWikiUploads set to true, 27 more file extensions, apparently anything that anyone in the WMF office ever wanted to share)
  • Extensions using old-style PHP files:
    • TimedMediaHandler: $wgTmhFileExtensions = [ 'ogg', 'ogv', 'oga', 'flac', 'opus', 'wav', 'webm', 'mp4', 'mp3' ];

Second, some extensions are removed from the list based on the following sources:

  • MediaWiki core defaults: $wgFileBlacklist = [ ... ]; (30 file types that no one could conceivably want to upload, or that we disallow for security reasons)
  • WMF configuration: $wgFileBlacklist[] = 'txt'; $wgFileBlacklist[] = 'mht';
  • TimedMediaHandler:
    • Removes mp3 if $wgTmhEnableMp3Uploads is false
    • Removes mp4 if $wgTmhEnableMp4Uploads is false

Finally, any duplicates in the list are removed before it is shown to the user (e.g. on Special:Upload).


Therefore:

To add a file extension to the list:

  • Add it to wmgFileExtensions in WMF configuration for the desired wiki and ensure nothing removes it later

To remove a file extension from the list:

  • If it is listed in wmgFileExtensions in WMF configuration for the desired wiki, remove it from there
  • If it is added by an extension or otherwise magically, also add it to wmgFileBlacklist in WMF configuration for the desired wiki

Unanimous agreement for the English Wikivoyage is documented here: https://en.wikivoyage.org/wiki/Wikivoyage_talk:Image_policy#Proposal_to_change_Special:Upload_page

Per that proposal, only JPG/JPEG, GIF and PNG would be allowed. I think we also need to allow TIF/TIFF and SVG, given that files of both types currently exist on the wiki (https://en.wikivoyage.org/wiki/Special:MediaStatistics) and disallowing them would probably prevent those from being reuploaded. (If these are deleted, we can safely disallow them, but neither TIF/TIFF nor SVG files are common abuse vectors.)

Change 382749 had a related patch set uploaded (by Bartosz Dziewoński; owner: Bartosz Dziewoński):
[operations/mediawiki-config@master] Disallow most file types from upload to enwikivoyage

https://gerrit.wikimedia.org/r/382749

Change 382749 merged by jenkins-bot:
[operations/mediawiki-config@master] Disallow most file types from upload to enwikivoyage

https://gerrit.wikimedia.org/r/382749

Mentioned in SAL (#wikimedia-operations) [2017-10-09T13:26:31Z] <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:382749|Disallow most file types from upload to enwikivoyage (T176647)]] (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-10-09T13:27:28Z] <zfilipin@tin> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:382749|Disallow most file types from upload to enwikivoyage (T176647)]] (duration: 00m 47s)