Page MenuHomePhabricator

Enable MP3 uploads on Wikimedia Commons and TMH playback
Closed, ResolvedPublic

Assigned To
Authored By
Jdforrester-WMF
Dec 3 2015, 9:34 PM
Referenced Files
None
Tokens
"Party Time" token, awarded by RandomDSdevel."Love" token, awarded by MichaelSchoenitzer."Love" token, awarded by MichaelSchoenitzer_WMDE."Like" token, awarded by Daniel_Mietchen."Love" token, awarded by Trizek-WMF."Barnstar" token, awarded by Keegan."Evil Spooky Haunted Tree" token, awarded by Quiddity."Pterodactyl" token, awarded by Deskana.

Description

NOTE: To be updated with more detailed plans.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Once the switch is flipped on the server right? I've collected some interesting patents (these are probably more significant than MP3 though):

Agree with @Base, this should not be enabled on Commons unless a reviewing interface is built (similar to CopyPatrol) or the right to upload mp3s is restricted to a subset of users (at least to begin with). It is extremely difficult currently for Commons volunteers to review audio for copyright violation. If the world discovers that Commons is now a free mp3 hosting service and we have no way to patrol what is being uploaded, we will quickly have a crisis on our hands that is worse than selfie-pocalypse.

^ I've filed T167815: Conduct MP3 patrol discussion just in case. This is a task to be coordinated through the Community Liaisons team in the Foundation. @Elitre will need some time to identify scope of work and assignees.

It's possible that patroller consensus arrives at a different conclusion, but I agree there's a heightened risk and we can and should further discuss potential threats and countermeasures.

TheDJ changed the task status from Open to Stalled.Jun 14 2017, 8:25 AM

OK, I think we should interpret the above as a consensus to not enable MP3 support on Wikimedia for the immediate future in that case.

I'm an eventualist as much as the next person, but if we wait for a copyright violation tool to be developed we may be waiting a long time. That makes me sad that we'd potentially miss out on a sizable amount of useful content.

What if we approached this in phases?

Phase 0 - Uploading MP3s is a user right (that the Commons folks could help us figure out the requirements for) initially. Uploads are only allowed by folks in this group. Maybe figure out a way to tag/track these uploads for accountability? Idea stolen from kaldari's comments above. T120288#3343239. I could help with the outreach on this to Commons.
Phase 1 - A tool is developed that perhaps an expanded group could use to patrol. This sounds something that might be accomplished via a Project Grant. Dispenser mentions a few services that make make this possible: T132650#3273150
Phase 2 - An inline tool in the upload process that everyone uses. It checks every upload and never makes a mistake. :)

There's probably more details in here than what I described, but what do others think?

@CKoerner_WMF: I like your plan and think it would make sense to present something like it to the Commons community for discussion. My suggestion for the implementation of phase 0 would be:

  • Create a new user right (upload-mp3)
  • Either create a new user group for that right on Commons or assign it to the existing user groups "Administrators" and "Image reviewers"
  • Add a check for the user right to all the uploading interfaces (probably can just modify UploadBase::verifyUpload() or something similar)
  • Add a check for the user right to all the uploading interfaces (probably can just modify UploadBase::verifyUpload() or something similar)

We probably don't want to hack mw core only to add this. AF might be better suited for this.

We probably don't want to hack mw core only to add this. AF might be better suited for this.

Using AbuseFilter for this would be great from a developer point of view, but would suck for end users. Imagine going through all the steps in Upload Wizard only to be told that your upload was rejected (since AbuseFilter is only triggered at the last stage of uploading). I'm pretty sure there's a hook for adding valid mime-types and there's also an UploadVerification hook in verifyUpload(), so this could be handled by an extension (rather than in core). Maybe the UploadWizard extension (although it's a bit of an awkward fit since this will affect non-UploadWizard uploads as well).

@CKoerner_WMF I think the most important thing that needs to be understood here is that any decision to enable mp3 uploading on Commons that is not implemented as the result of a explicit decision by the Commons community is likely to end badly. Now that the legal issues have been cleared up, and it's established that the technical implementation is not a problem, most of this discussion really needs to move onwiki, so that the community as a whole will have an 'investment' in the process of deciding how it is implemented. If a plan such as yours (which is not bad) is presented to the community as a 'fait accompli', even if merely for discussion, there will be grumbling.

If the decision is presented to the community as a request for them to 'rubber stamp' a process that has been predecided, not only will it get complaints at the time, but if there are issues then the WMF is likely to be blamed. It's unfortunate, but realistic.

@Revent: Good point. We shouldn't overthink this until we have feedback from the Commons community.

@Revent agreed. I'm taking the liberty of subscribing you to T167815: Conduct MP3 patrol discussion, you'll find the discussion there to be relevant.

@Revent We are in agreement.

Interested folks should wander over to the task Keegan mentioned ( T167815) and discuss starting a conversation on Commons.

It been 7 months since MP3 decoding support dropped in RedHat Fedora Linux, Chromium (the FLOSS fork that doesn't support H.264) will now have MP3 support. Will Wikipedia be the last Open Source project to get MP3 support?

Since all browsers will have playback support (no need to transcode). I suggest before Wikimania (Aug 9) we allow .mp3 uploads T162395.

Simply adding playback to a FLOSS desktop app is vastly different that adding support for uploads and management (patrol etc) of mp3s. One has real on-going human time impact (patroling etc), the other doesn't. Please don't be so dismissive.

@greg We have been deleting an average of 260 uploads per week since January 1, 2017 from Wikipedia Zero pirates T129845. Many videos uploads are from YouTube. And frankly YouTube is a more convenient music listening experience.

@greg We have been deleting an average of 260 uploads per week since January 1, 2017 from Wikipedia Zero pirates T129845. Many videos uploads are from YouTube. And frankly YouTube is a more convenient music listening experience.

@Dispenser I'm not sure what the WP0 piracy issue and YouTube have to do with the timelines for getting this deployed on Commons. Could you clarify how the points relate to each other?

@Keegan I've research and built tools for audio related uploads due to the Wikipedia Zero piracy problem. I have an IRC bot which fingerprints the audio track of audio and video files and lists acoustid.org matches. I also have an agreement with ACRCloud (strings attached) to scan music. I think that Wikipedia's use as a piracy host (for non-zero users) is lessened with easier free listening on YouTube (or Pandora or Spotify). This directly address @greg patrolling comment where he alludes to T167815.

@Dispenser Got it, thank you for the clarification.

@Dispenser: Can your IRC bot already handle mp3s? Where does it list the matches?

@kaldari I tested python-acoustid to work with an MP3. The bot posts results to #wikimedia-commons-admin and ##wikimedia-commons-newbie-av on Freenode. Its hastily built (just to see if anyone wanted it), runs in anti-nuisance hourly updates, and I added Acoustid to see how well it works. It'll be converted it into a web tool so admins can delete and patrol without opening windows and at their convenience.

@CKoerner_WMF I think the most important thing that needs to be understood here is that any decision to enable mp3 uploading on Commons that is not implemented as the result of a explicit decision by the Commons community is likely to end badly. Now that the legal issues have been cleared up, and it's established that the technical implementation is not a problem, most of this discussion really needs to move onwiki, so that the community as a whole will have an 'investment' in the process of deciding how it is implemented. If a plan such as yours (which is not bad) is presented to the community as a 'fait accompli', even if merely for discussion, there will be grumbling.

If the decision is presented to the community as a request for them to 'rubber stamp' a process that has been predecided, not only will it get complaints at the time, but if there are issues then the WMF is likely to be blamed. It's unfortunate, but realistic.

This has been a problem for as long as Wikipedia existed though. Do the average user even know what Ogg Vorbis is even now?

It been 7 months since MP3 decoding support dropped in RedHat Fedora Linux, Chromium (the FLOSS fork that doesn't support H.264) will now have MP3 support. Will Wikipedia be the last Open Source project to get MP3 support?

Since all browsers will have playback support (no need to transcode). I suggest before Wikimania (Aug 9) we allow .mp3 uploads T162395.

Thinking about, it would be a fun discussion with Jimbo Wales at Wikimania.

@Dispenser: Any update on building a web interface to the Acoustid script?

Per the previous discussions on Commons, I started an RFC to settle the question of which user groups should initially be allowed to upload mp3s: https://commons.wikimedia.org/wiki/Commons:Village_pump#RFC:_Which_user_groups_should_be_allowed_to_upload_mp3s.3F

So there wasn't a clear consensus on what groups should be allowed to upload to commons, but it seems like the idea with the most support is:

  • Create a new "MP3 uploaders" group
  • Give it the following permissions to start with: Not be affected by IP-based rate limits (autoconfirmed), Upload files (upload), Overwrite existing files (reupload)
  • Allow admins to assign users to the new user group
  • Create an edit filter that limits mp3 uploads to users that are either admins or MP3 uploaders.

I created a new task (T180002) to deal with the new user group.

To ask the question explicitly ...

This requested change will be to allow MP3 files for Commons alone. Other wikis that wish to have MP3 files will need to explicitly request MP3 files be allowed through a separate RFC. [In contrast to MP3 files are going to be allowed for all wikis, and Commons needs a means to better manage this, hence this process and discussion.]

Can someone tell me why we still don't allow mp3 uploads for 3rd party users of MediaWiki ? It seems unfair to punish them for the restraint of Wikimedians...

@TheDJ I'm not sure I understand. $wgFileExtensions can be configured to allow/disallow file types including mp3s.

@TheDJ I'm not sure I understand. $wgFileExtensions can be configured to allow/disallow file types including mp3s.

@CKoerner_WMF no, you need to explicitly set the undocumented option wgTmhEnableMp3Uploads to true, otherwise the extension will remove the mp3 file extension from the configured list of allowed extensions.

@CKoerner_WMF no, you need to explicitly set the undocumented option wgTmhEnableMp3Uploads to true, otherwise the extension will remove the mp3 file extension for the configured list of allowed extensions.

That sounds like a bug to me.

AIUI as soon as this gets released to Commons then wgTmhEnableMp3Uploads will be removed. I wouldn't worry about it.

kaldari renamed this task from Once it's not patent-encumbered, enable MP3 support for Commons uploads and TMH/etc. playback to Enable MP3 uploads on Wikimedia Commons and TMH playback.Nov 21 2017, 5:21 PM
kaldari updated the task description. (Show Details)
kaldari added subscribers: Ricordisamoa, Jeff_G, Kizule.

Change 392765 had a related patch set uploaded (by Kaldari; owner: kaldari):
[operations/mediawiki-config@master] Enable MP3 uploading on Commons on Beta Cluster

https://gerrit.wikimedia.org/r/392765

Can folks take a look at https://gerrit.wikimedia.org/r/392765 and tell me if it looks sane? Am I missing anything?

Change 392765 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MP3 uploading on Commons on Beta Cluster

https://gerrit.wikimedia.org/r/392765

MP3 uploading and playback is now working on the Beta Cluster: https://commons.wikimedia.beta.wmflabs.org/wiki/File:Chopin-waltz-op-69-no-2-in-b-minor.mp3. Transclusion and playback from other Beta Cluster wikis is also working: https://simple.wikipedia.beta.wmflabs.org/wiki/Piano.

If there are no issues, we can turn this on for Commons during the next SWAT window.

Change 393661 had a related patch set uploaded (by Kaldari; owner: kaldari):
[operations/mediawiki-config@master] Enable MP3 uploads on Commons

https://gerrit.wikimedia.org/r/393661

Change 393661 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MP3 uploads on Commons

https://gerrit.wikimedia.org/r/393661

Mentioned in SAL (#wikimedia-operations) [2017-11-30T19:11:59Z] <thcipriani@tin> Synchronized wmf-config: SWAT: [[gerrit:393661|Enable MP3 uploads on Commons]] T120288 (duration: 00m 51s)

kaldari claimed this task.

Done.