Page MenuHomePhabricator

VisualEditor file upload is broken for new user causing loose of >0.5 million files
Open, HighPublic

Description

(this is summary of a discussion in hewiki: Something strange happened yesterday in the visual editor while teaching students in Be'er Sheva)
How to reproduce:

  1. (this should be done as a new user)
  2. Open an arbitrary article, and press "Edit" to enter visualeditor
  3. Upload file: Insert => Media => Upload ...
  4. An error message that isn't clear enough will appear "We could not determine whether this file is suitable for Wikimedia Commons..."

Technical description:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

If you want to read about what led to the current solution, see T120867 (it's a very long discussion though). T138273 is the solution everyone eventually mostly agreed on.

This effectively de feature upload from VisualEditor which is intended mainly for new users. If the upload dialog interface isn't good enough / doesn't instruct user properly this should be fixed in the upload dialog and not by abusing the abuse filter

We tested a few different designs trying to instruct the users, but we found that users just do not care. https://www.mediawiki.org/wiki/Reading/Multimedia/December_2015_cross-wiki_upload_A/B_test

There were also vague plans to embed UploadWizard inside that dialog, instead of the current simplified process. But it's not clear whether that would help (if you're going to ignore all the licensing advice, it's almost as easy to click through UW as through the current dialog), and it would require some significant effort (I'd estimate a few weeks for a 2-3 person team, to allow UW to upload files cross-domain, to make it not require its current special page to work, and to finish the work on updating its interface to match VisualEditor).

  • Formalize this decision with proper user right checks (see mw.Upload.BookletLayout.prototype.initialize)

This would make the conditions a bit more difficult to change (although that might not be a problem – while the filter has been heavily edited after it was set up, it hasn't been changed for a while now: https://commons.wikimedia.org/wiki/Special:AbuseFilter/history/153). We'd also need to somehow make this configurable, so that these restrictions are not enforced for non-Wikimedia users of MediaWiki.

  • Change the message to instruct the user to upload with UploadWizard

Technically it's easily done, although I wonder if the message doesn't intentionally omit that link, since Commons folks do not want it to be too easy to upload unwanted content.


I'll suggest another possible improvement:

  • Allow the current abuse filter to prevent the upload immediately after uploading the file (on the first step of the dialog), not only after entering the description and other details (second step). It currently works this way because the filter relies on the "Cross-wiki upload from …" edit summary, which is only submitted when publishing the upload. We'd need to implement some way in AbuseFilter to detect cross-wiki uploads at the first step, though.

If you want to read about what led to the current solution, see T120867 (it's a very long discussion though). T138273 is the solution everyone eventually mostly agreed on.

This effectively de feature upload from VisualEditor which is intended mainly for new users. If the upload dialog interface isn't good enough / doesn't instruct user properly this should be fixed in the upload dialog and not by abusing the abuse filter

We tested a few different designs trying to instruct the users, but we found that users just do not care. https://www.mediawiki.org/wiki/Reading/Multimedia/December_2015_cross-wiki_upload_A/B_test

There were also vague plans to embed UploadWizard inside that dialog, instead of the current simplified process. But it's not clear whether that would help (if you're going to ignore all the licensing advice, it's almost as easy to click through UW as through the current dialog), and it would require some significant effort (I'd estimate a few weeks for a 2-3 person team, to allow UW to upload files cross-domain, to make it not require its current special page to work, and to finish the work on updating its interface to match VisualEditor).

@matmarex thanks for the referring to T120867 - I just read most of the discussion there.

  • As for UI approach, integrating UW inside dialog could be done quick &dirty in ~1h with <iframe src="https://commons.wikimedia.org/wiki/Special:UploadWizard?printable=yes"> + playing with mPreventClickjacking for uploadwizard. Although ugly it should work, and possibly simple enough to be tested.
  • Ideally pre knowledge about copyright/images with help of commons community/statistics should guide the UI (actions based on EXIF exists, high/low resolution, etc) and not only instructions. it seems like users don't read instructions.
  • Formalize this decision with proper user right checks (see mw.Upload.BookletLayout.prototype.initialize)

This would make the conditions a bit more difficult to change (although that might not be a problem – while the filter has been heavily edited after it was set up, it hasn't been changed for a while now: https://commons.wikimedia.org/wiki/Special:AbuseFilter/history/153). We'd also need to somehow make this configurable, so that these restrictions are not enforced for non-Wikimedia users of MediaWiki.

The conditions are whether or not the user have access right/belong to group. How do the user actually gets access to user group is up to regular configurations such as https://www.mediawiki.org/wiki/Manual:$wgAutopromote While it may be less accessible to commons admins to control it, it is still easily changeable with regular configuration.

  • Change the message to instruct the user to upload with UploadWizard

Technically it's easily done, although I wonder if the message doesn't intentionally omit that link, since Commons folks do not want it to be too easy to upload unwanted content.


I'll suggest another possible improvement:

  • Allow the current abuse filter to prevent the upload immediately after uploading the file (on the first step of the dialog), not only after entering the description and other details (second step). It currently works this way because the filter relies on the "Cross-wiki upload from …" edit summary, which is only submitted when publishing the upload. We'd need to implement some way in AbuseFilter to detect cross-wiki uploads at the first step, though.

Both ideas are good improvements, but they aren't solutions, as new users are still blocked with no clear explanation (from their perspective)

The file upload is not technically broken, it's working as expected (i.e. in a very user-unfriendly way because it was not designed to help users respect the copyright expectations of Wikimedia wikis). I suggest to close this as invalid, or to repurpose it as a bug to disable cross-wiki upload altogether on all wikis with any target.

Formalize this decision with proper user right checks (see mw.Upload.BookletLayout.prototype.initialize)

Not a good idea. Fine-tuning of content filtering shouldn't be hardcoded by developers. The AbuseFilter (which was meant as a short-term solution until the rethinking of cross-wiki upload) does not merely check for user rights or conditions, but tries to identify which files are most likely to be copyright violations based on their content and metadata.

Change the message to instruct the user to upload with UploadWizard

The message already does so: it links the tutorial, which directs users to the UploadWizard. Of course it would be easier if the VisualEditor buttons just sent people to the UploadWizard directly.

Change 356811 had a related patch set uploaded (by Eranroz; owner: Eranroz):
[mediawiki/core@master] WIP: Introducing crosswiki-upload right

https://gerrit.wikimedia.org/r/356811

The file upload is not technically broken, it's working as expected (i.e. in a very user-unfriendly way because it was not designed to help users respect the copyright expectations of Wikimedia wikis). I suggest to close this as invalid, or to repurpose it as a bug to disable cross-wiki upload altogether on all wikis with any target.

New users can not technically upload files - so it is broken. The bug here aims to solve it:

  • If we want to not let them upload - we should block this option in the UI properly. It isn't possible to do that with AbuseFilter, and this is usually done either with configuration or with user rights.
  • If we do want - we should refine the dialog to let them upload files without breaking copyright.

Formalize this decision with proper user right checks (see mw.Upload.BookletLayout.prototype.initialize)

Not a good idea. Fine-tuning of content filtering shouldn't be hardcoded by developers.

commons admins can still define AbuseFilters if needed to handle vandalism/immediate action. The specific filter of new users should be replaced with user group - this is up to commons community to define how/who have "cross-upload" right- and based on user groups in the repository wiki (commons) not based on local wiki. To be more specific:

  • if commons have good abusefilter that can filter copyvios - all users can have "cross-upload" right, and you can let the abuse filter handle the minority
  • if commons aren't satisfied with the current UI for crosswiki upload - commons should define the user right only for trusted groups. For example based on auto-promote according to criterias/manual promotion etc. Whenever the criterias need to be changed - ask for configuration change using regular configuration process (Commons:VP consensus => phabricator task => SWAT deployment)

New users can not technically upload files

Not true. They are allowed to upload JPEG files above 5 MPx and 50 kB, as well as non-PNG and non-JPEG images.
https://commons.wikimedia.org/wiki/Special:AbuseFilter/153

commons admins can still define AbuseFilters if needed

An abusefilter wouldn't be able to override a userright. I don't see any need to further restrict rights of new users.

eranroz renamed this task from VisualEditor file upload is broken for new user to VisualEditor file upload is broken for new user causing loose of >0.5 million files.Oct 20 2017, 2:05 PM
eranroz updated the task description. (Show Details)

Change 356811 abandoned by Eranroz:
WIP: Introducing crosswiki-upload right

Reason:
abandon as there is no agreement in the task for this solution

https://gerrit.wikimedia.org/r/356811

Just got this reported by a new user on IRC... very annoying!

saper triaged this task as High priority.Feb 10 2020, 1:17 AM

So, I have just spent an HOUR trying to troubleshoot this with a very nice and patient new user. It was not obvious at all what is the problem (I was suspecting some wrong license settings first but we double checked).

Then I found this and we tried to work around this by using UploadWizard interface, which didn't work again, because the user hit filter https://commons.wikimedia.org/wiki/Special:AbuseFilter/156 for small images.

Thank goodness I am abuse filter operator on plwiki so I know what this all means and I told her how to cheat the filter.

This is 1 a.m. and nobody was really around to on IRC to help the user...

JTannerWMF subscribed.

Unfortunately, this experience depends on policy that our team has no ability to change.