Page MenuHomePhabricator

MediaWiki file operations are fragile, causing occasional data loss
Open, Needs TriagePublic

Event Timeline

Tgr created this task.Dec 18 2016, 12:47 AM
Restricted Application added projects: Multimedia, Commons. · View Herald TranscriptDec 18 2016, 12:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr added a subscriber: daniel.Dec 18 2016, 1:08 AM

The typical way (or maybe not, but the one we understand) for files to get lost is that a storage operation fails but the corresponding DB operation succeeds (or the other way around) and the filename in the DB ends up wrong; the file still sits on the disk but we don't know where. There are a few things that could be done about this:

  • T149847: RFC: Use content hash based image / thumb URLs will avoid this in some cases but not all (notably file deletion/undeletion)
  • we could create some abstract concept of transactions in MediaWiki which could cover operations with DB and non-DB elements. IIRC @daniel has been working on this.
  • or we could just store original files in the database, in which case it would become the single source of truth. We are talking about a hundred terabytes though.
Tgr renamed this task from File update operations are fragile, causing occasional data loss to MediaWiki file operations are fragile, causing occasional data loss.Dec 18 2016, 1:08 AM

The filerepos do have support for using a lockmanager

Poyekhali moved this task from Incoming to Backlog on the Commons board.Dec 18 2016, 1:31 AM
Poyekhali triaged this task as High priority.
Poyekhali added a subscriber: Poyekhali.

We don't want our files to be permanently lost, if truly the file operations are fragile, then more files may be affected by this bug.

Poyekhali moved this task from Untriaged to Triaged on the Multimedia board.Dec 18 2016, 1:31 AM
Aklapper raised the priority of this task from High to Needs Triage.Dec 18 2016, 8:57 AM
Nemo_bis added a subscriber: Nemo_bis.
Tgr added a comment.EditedDec 21 2016, 8:30 PM

The filerepos do have support for using a lockmanager

Locking just avoids race conditions. For rollback/atomicity you need to remember the old state.

try {
    copy file to new name
    update file name in database
    delete file under old name
} catch {
    delete file under new name
    restore old file name in database
}

or something like that. It can be tricky, especially when considering that the rollback operations can themselves fail, especially if the original failure is caused by some external problem. (And I suspect copying huge files instead of moving them can get tricky on its own right.)

zhuyifei1999 updated the task description. (Show Details)Mar 16 2017, 5:49 PM
MarkTraceur moved this task from Triaged to Tracking on the Multimedia board.Jun 5 2017, 3:09 PM
Tgr updated the task description. (Show Details)Jun 28 2018, 3:01 PM
jcrespo updated the task description. (Show Details)Jun 29 2018, 5:38 AM
Yann added a subscriber: Yann.Jun 29 2018, 10:13 AM