Train an image classifier to identify classes of images that are top candidates for deletion.
According to T340546: [XL] Analysis of deletion requests on Commons (see last Viable reasons frequency section), 13.4% of all deletions are:

1. logos
2. books
3. screenshots
4. album covers

The initial direction is to:

• take a pre-trained EfficientNet V2 model
• gather a dataset from Wikipedias (fair-use images) and/or Commons (free ones) with roughly 10k samples per class
• fine-tune the model on our 4 classes with a train/validation dataset split
• evaluate against a separate dataset of available images (class to be extracted from Commons categories)
• evaluate against a separate dataset of deleted images (class to be extracted from their reason for deletion)

Results

Evaluation metrics

• accuracy
• area under the curve (AUC), computed separately for each class and then averaged across classes, see also here
  • AUC precision/recall
  • AUC ROC
• model's loss function, i.e., categorical cross-entropy

Legend

• all scores are percentages
• best performances in bold
• numbers in round brackets are the training epochs that obtained the best scores. Stars denote the same epoch for all metrics.

Dataset: available images

Source: Commons images with class categories

Dataset: deleted images

Source: T350020: Access request to deleted image files in the production Swift cluster

Observations

• the logo classifier is clearly the best one
• all performances decrease with the deleted images dataset. Based on manual checks, the dataset looks more noisy compared to the available images one. This is possibly caused by:
  • the image extraction method, i.e., reason for deletion text VS Commons categories
  • the randomness of non-class samples, which seems to have penalized the classifiers

Code

• training
• evaluation
• demo classification