Confirming that the SQL query produces the same result (except for today's data) on the commonswiki master:

mysql:wikiadmin@db1040 [commonswiki]> select left(us_timestamp,8), count(*) from uploadstash group by left(us_timestamp,8);
+----------------------+----------+
| left(us_timestamp,8) | count(*) |
+----------------------+----------+
| 20120914             |        2 |
| 20130131             |        1 |
| 20130211             |        1 |
| 20130215             |        3 |
| 20130220             |        1 |
| 20130428             |        2 |
| 20131113             |        1 |
| 20131211             |        1 |
| 20140111             |        1 |
| 20140418             |        1 |
| 20140420             |        2 |
| 20140501             |        1 |
| 20140526             |        1 |
| 20140528             |        1 |
| 20140729             |        1 |
| 20140903             |        1 |
| 20140912             |        1 |
| 20140914             |        1 |
| 20140925             |        1 |
| 20141012             |        1 |
| 20141013             |        1 |
| 20141108             |        1 |
| 20150122             |        1 |
| 20150504             |        1 |
| 20150916             |        1 |
| 20160318             |     3136 |
| 20160319             |     3859 |
| 20160320             |      612 |
+----------------------+----------+
28 rows in set (0.13 sec)

I guess the best debugging step, would be for someone to run cleanupUploadStash.php by hand, and see if its outputting any errors.

After some code browsing, I don't think this is actually a security issue (it's just really lame), this could probably be made public.

In T130478#2138065, @Bawolff wrote:

I guess the best debugging step, would be for someone to run cleanupUploadStash.php by hand, and see if its outputting any errors.

It is apparently outputting so many that the cron job redirects the output to /dev/null (f65a3264c9936a1cfd790d4bc60bebd0530adabc).

https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/manifests/maintenance/cleanup_upload_stash.pp

In T130478#2138065, @Bawolff wrote:

I guess the best debugging step, would be for someone to run cleanupUploadStash.php by hand, and see if its outputting any errors.

It produces 30 errors, all like this:

Failed removing stashed upload with key: ${KEY} (UploadStashBadPathException)

Then it outputs a bunch of progress counts, which by itself would be enough to result in f65a3264c I believe.

Trying to delete those 30 keys with mwrepl reveals the same message in all those UploadStashBadPathExceptions: path doesn't exist. My guess is that the actual files were deleted long ago, and now the maintenance script can't take care of the database entries because it errors out trying to load them.

Trying to delete those 30 keys with mwrepl reveals the same message in all those UploadStashBadPathExceptions: path doesn't exist. My guess is that the actual files were deleted long ago, and now the maintenance script can't take care of the database entries because it errors out trying to load them.

If someone can confirm those 30 manually, can we just delete the db entries (manually), in that case?

The script should be able to delete any "orphaned" files, if they still remain, so yeah, we should be able to just remove the database entries.

I ran a similar query against all databases (select count(*) from uploadstash where us_timestamp<'20160320'), and stale uploadstash entries are present on commonswiki (30), enwikivoyage (1), hewiki (4) and testwiki (1).

We removed the Security tag and bug settings since this seems to have no security implications.

Having been recently reminded of this task over my foul language in the opening comment, I re-checked it and the old rows dating back to 2012 still exist in the database.

There is also a bunch of new ones from 2016-2018, so I guess this might require fixing the 'cleanupUploadStash.php' maintenance script rather than a one-off DELETE query.

Thanks for closing the other ticket as duplicate. I was merely reporter of an issue though. Since I have nothing to contribute to actually fix it, let me remove myself. Cheers