Scoring revisions of the file page instead of revisions of the image is a somewhat dirty hack but doesn't really have disadvantages other than maybe performance (there are more page revisions than file revisions) since a new file revision is created every time a file is uploaded or reuploaded. Some care would have to be taken in how changes are detected (sometimes the revisions are null revisions; sometimes they aren't sent to recentchanges) but other than that, it's viable, and probably saves work that isn't necessary long term as eventually the image table is supposed to be merged into the revision table, with images becoming MCR slots.

In T214201#5341287, @Tgr wrote:

eventually the image table is supposed to be merged into the revision table, with images becoming MCR slots.

Interesting. Is there a Phab task or wiki page describing this?

Recent comments on T28741: Migrate file tables to a modern layout (image/oldimage; file/file_revision; add primary keys) look relevant.

T28741 is about having sane keys and revision-like structure (revisions are currently in the revision table, keyed by an autoincremented ID, and moved to archive on page deletion, and flagged on revision deletion; for images the current version is in the image table, moved to oldimage when a new version is uploaded, and to filearchive when the image or a revision of it gets deleted, which results in a lot of cross-table copying; and they are keyed by upload timestamp, resulting various hacks to deal with two images being uploaded at the same time - you'll find a sleep(1) somewhere in the codebase as a last-ditch defense). The MCR task is T96384: Integrate file revisions with description page history. It would be a huge refactoring (and even MCR is not finished yet) so not something to expect soon, though.

open_nsfw is basically a (publicly available) snapshot of a neural network, not something we'd train ourselves, right? Which means that while it's definitely a good stopgap measure, a clever attacker could probably easily defeat it using an adversarial network.

In T214201#5349087, @Tgr wrote:

...a clever attacker could probably easily defeat it using an adversarial network.

I think that's probably true, however, we are more likely trying to stop the drive-by vandal than a serious coordinated attack. If we could reduce by 10% the amount of inappropriate images admins had to deal with, we'd be saving hundreds of hours of people's time.

The technical merits but have flaws but the real-world application could have great benefits.

In T214201#5055393, @Halfak wrote:

With all that said, I'm interested in trying to cram this into volunteer time and will give you updates when I manage to make some progress.

@Halfak would now be a good time to bring this back up?

In T225664 @Mholloway developed an API implementation of open_nsfw based on open_nsfw--. Utilizing this implementation hosted at nsfw.wmflabs.org, I wrote a python script to benchmark the processing speed of the model.

from urllib.request import urlopen
from bs4 import BeautifulSoup

import requests
import re
import time

html = urlopen('https://commons.wikimedia.org/w/index.php?title=Special:NewFiles&offset=&limit=500&user=&mediatype%5B0%5D=BITMAP&mediatype%5B1%5D=ARCHIVE&mediatype%5B2%5D=DRAWING&mediatype%5B3%5D=MULTIMEDIA&mediatype%5B4%5D=UNKNOWN&start=&end=&wpFormIdentifier=specialnewimages')

bs = BeautifulSoup(html, 'html.parser')
timeSum = float()
count = int()

images = bs.find_all('img', {'src':re.compile('.jpg')})
for image in images:
    url = image['src']
    start = time.time()
    req = requests.post("https://nsfw.wmflabs.org", data={"url":url})
    # print((req.text))
    timeSum += time.time() - start
    count += 1

avgTime = timeSum/count
print(timeSum)
print(avgTime)
print(count)

Output:

801.2340695913221
1.804581237815277
444

This script scrapes the commons site for the 500 latest images uploaded and passes their URL to the model hosted on labs and times the whole thing.

The output shows 444 images processed which may be because of the fact that the model is not able to process very large images. This is a potential complexity which will need to be addressed in the task.

On average it takes about 1.8s per image which is good performance generally speaking but may cause a roadblock if we want upload time classification.

I also ran the API locally to test compute and memory usage and the model is very efficient in both these regards.

TL;DR

The model is already hosted at nsfw.wmflabs.org and serves API requests that post image URLs.
On testing the model with 500 latest images uploaded to commons, model successfully processed 444 images.

Issues:

1. Why aren't all images processed?
2. How to ensure all images are processed?
3. If we cannot make the model faster, how do we achieve upload time classification?
4. Should we just score images when they are being added to a wiki and ask the user to wait for the scoring to complete before confirming the edit?

@MusikAnimal Some follow up questions after the preliminary work I have done on the task:

1. What is the maximum threshold of time we can wait for the model to process the image to allow upload time classification?

2. Are we open to custom implementations of a Deep Neural Net that can be trained in wmflabs and then deployed into production?

3. Can you shed some light on how I could begin integrating nsfw.wmflabs.org instance to track RC and alert AbuseFilter?

Would really get the task moving. Thank you so much!

Love to all for all this hardworking activities ❤❤❤❤❤❤❤💓💓💓💓💓💓💓💕💕💕💕💖💖💖💖💖🌍🌎🌐🗺🌞🌞🌞🌞✨✨✨✨💡💡💡💡🕯🕯🕯🕯✨✨✨

After speaking with @Daimona I have been informed that we require a MediaWiki extension if we wish to communicate with AbuseFilter. Is it possible to use the MachineVision extension for this or should I write a custom extension for this task?

Please correct me if I have made any wrong assumptions

Thank you

In T214201#6026319, @Chtnnh wrote:

1. What is the maximum threshold of time we can wait for the model to process the image to allow upload time classification?

If you are scoring at upload time, I think you could do it in a deferred update so that it doesn't block giving a response to the user. So the scoring just sort of happens in the background and won't slow anything down for the user. This does mean it's possible the score won't immediately be available as soon as the user adds the image to an article, but in theory we're only talking about a 1-2 second window, which I think is acceptable. Hopefully that answers the question.

2. Are we open to custom implementations of a Deep Neural Net that can be trained in wmflabs and then deployed into production?

Sure! I don't know how much work that involves, but I've no opposition. All I can say is open_nsfw works like a charm out of the box.

3. Can you shed some light on how I could begin integrating nsfw.wmflabs.org instance to track RC and alert AbuseFilter?

There's no "alerting" AbuseFilter. Rather, when an edit is made that adds an image, AbuseFilter would look up the score for it, and be able to disallow the edit if it matches some threshold. Basically for this project, I don't think we need to be concerned with AbuseFilter, rather just making it possible for AbuseFilter to use the data. The solution there (as I understand it) is to simply store the image scores at upload time in a database table. Later, AbuseFilter can be updated to fetch these scores.

I think you could do it in a deferred update so that it doesn't block giving a response to the user.

Could you please elaborate on how I would go about doing that?

All I can say is open_nsfw works like a charm out of the box.

If that is the case, I surely don't mind going forward with it, as that would significantly reduce the scope and time of this task.

The solution there (as I understand it) is to simply store the image scores at upload time in a database table. Later, AbuseFilter can be updated to fetch these scores.

So at this point, the scope of this task would only be to implement the scoring mechanism and the DB to store the given scores, if I understand correctly?

Thank you so much for the clarifications!

I also had some additional questions after doing some more work on the task:

1. Since production cannot rely on services that are hosted on wmflabs.org, do you need me to create a phabricator task to bring the API implementation to production first?
2. After my discussion with @Daimona , I understand that allowing AbuseFilter to retrieve the scores in the DB is a nontrivial task. Do you want me to file a phabricator task for that as well?

Thank you!

Since production cannot rely on services that are hosted on wmflabs.org, do you need me to create a phabricator task to bring the API implementation to production first?

That is correct. You have to be prepared though, the standards for services deployed within production are significantly higher than on tool forge thus will probably require a bunch of work in polishing your service. Anyway, you'd need to file a ticket following the process described here - don't bother with the rest of that page, it's for node.js services, so it probably wouldn't apply to yours.

As for integration with MediaWiki - I strongly suggest talking with @Mholloway and trying to integrate your new classifier into the MachineVision extension - what you're going to do seems like a very good fit there.

Thank you so much for your review @Pchelolo,

I will currently wait for some approval from @MusikAnimal and @Mholloway before filing the ticket. Anything else you might want to add or should I proceed with filing the ticket?

The new service request has been filed (T250110) and if you have any input on it please feel free to comment on the task.

FYI that there is some talk about WMF deploying a NSFW classifier for commons on the year 2020-2021

@Nuria, feeling good and happy to know that.

Ahem, i think it got removed from the planning cause i no longer see it

😣

Hello everyone.
I wanted to propose an image moderation project for Outreachy round 21 T258524. I had a chat with @Chtnnh and he pointed me to this project. Since the theme for this round is 'data science and engineering', I believe this would be an ideal project for the internship. We would like to build upon this implementation. We have penned down few tasks which can be covered in the scope of the internship.

Tasks:

• Model performance benchmarking
• Dockerizing the project
• Video processing module for directly processing videos, performing frame segmentation and the passing these frames to the model.
• Use a different API framework

In addition to these, we also plan on incorporating the issues mentioned above.

We would love to know what everyone here feel about this. 😃

Hi all! @Chtnnh and @Abbasidaniyal are planning to propose this project in Outreachy Round 21 and are interested in being a mentor. If you could share any concerns you may have in moving forward with this idea, please share.

@calbon Any reason we removing the Machine Learning Platform (Research) tag from this task?

Might be in scope for Machine-Learning-Team instead (but that's a team project, so it's up to that team)?

@Aklapper the ML team (cc @calbon) will be working on the platform to deploy ML in the near term not specific projects. My 2 cents here is that for an outreachy project is probably of interest to work on the model on the cloud environment but I would not spend energy coming up with a way to deploy it to production/dockerization.

@Aklapper @Chtnnh @Nuria For the next 9 months the team will be focused on building a model training, deployment, and management infrastructure. We can deploy models other folks create but we don't have the bandwidth to tackle this ourselves.

Hi!
As @Nuria suggested, @Chtnnh and I decided to go forward with working on the model. We have submitted a proposal on the Outreachy website as well. The proposal submitted is as follows:

Project title
Not Safe for Work (NSFW) media Classifier for Wikimedia Commons

Project Description
Wikimedia Commons is an online repository of free-use images, sounds, other media, and JSON files. Anyone can upload media to the Commons portal. The uploads are moderated by members and volunteers of the foundation manually. This project aims to build a classifier that can flag NSFW images/media for review.

Upon successful completion of this internship, the intern would have designed, implemented and tested a machine learning model that would be able to classify image and video media as SFW or NSFW with a high accuracy. They would also be given the chance to deploy the model to WIkimedia test and production servers. Further, they would build a data processing pipeline and an API for the model.

How to make contributions / Selection criteria
Since this is a scratch project, applicants are required to do some research initially. A basic comparison of the existing NSFW classifiers along with their computational requirements is required. All applicants are expected to read various research papers, and draw comparisons between them.
They are expected to come up with a report detailing their research, the various options that can be used to implement the model and what they are proposing to do if they are selected. This report should also detail implementational methods and procedures.

Project Goals

• Model Development
• Model performance benchmarking
• Video Processing Module
• API Setup

Skills
Tensorflow / PyTorch for model creation
Flask / Django / FastAPI for the API

Do let us know if you have any suggestions or improvement to this proposal. Thanks! 😄

It is a work in progress here: T264045

Two issues to watch out for in this project. The first is that if you are going to flag images as NSFW you need a module that works equally across all skin colours. I know this should be assumed, but there are some big organisations whose flesh colour definition was built from a non global sample and they wound up unintentionally with a racist product.

The second is your definition of NSFW - this is culturally variable and will be very different around the world. Either you need to be able to set this to different settings of NSFW, which could be complex, or you need to decide on which particular culture's NSFW setting you are working to, which of course does not fit with community values and WMF strategy.

What community standards are going to be used in some sort of scoring system for this? What is considered "safe for work" easily varies widely throughout the world.

Regarding concerns about the definition of "NSFW" being culturally dependent: from my interpretation of the task description, it seems "NSFW" here means "image with a tendency to be added to unrelated articles by vandals for shock value", which seems both culture-independent and easier to calculate.

It is however not what is commonly understood as "NSFW". If you take a word that means A and use it to name concept B you invariably end up with people assuming that it is about A.

True. I propose a rename to "Machine learning filter for shock image vandalism" or something like that. There are probably better terms than "shock image", too, but at least I would like to get rid of the term "NSFW". I'm not gonna unilaterally do it, since there's an Outreachy task for this, so it'll involve renaming lots of stuff or at least lots of confusion.

Agree with others above that using different words to describe a NSFW classifier, is still a NSFW classifier.

The consensus on Wikimedia Commons is that not only is this not needed, but it would also be unhelpfully divisive and lead to feeding anti-nudity and anti-sexuality warriors that are frequently a source of disruptive vandalism. Contributors here may enjoy addressing the points raised at the Commons Village Pump discussion for this task, by responding on Wikimedia Commons.

If anyone would care more formally to assess the Commons consensus before investing scarce resources into this controversial task, then COM:VPC is the right place to do it.

The Foundation really shouldn't be assigning intern/outreach tasks that impact the community, when there is a known-or-likely risk of the project receiving hostility or rejection from the community.
Note that the Commons discussion received UNANIMOUSLY negative community reception, with the only positive comments coming from a WMF staff member trying to defend the project.

This project, narrowly construed as non-filtering and offering only advisory information, might in theory be tolerable to the community. However the community never asked for this, and there are concerns that this will ultimately be harmful to our work. If anyone wants to open further discussion with the community, that's swell. But work on this project clearly should not proceed when the community response has been negative. Especially not when the response has been unanimously negative.

https://github.com/infinitered/nsfwjs might be interesting for implementation too (it can be run on Node)