Once the fsck of store/now is complete, inspect the resulting filesystem to recover files requested by endusers then remove the remaining volumes that live on the raid6 shelves.
Subtasks of this task for individual recovery requests.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | MaxSem | T103272 Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available | |||
| Resolved | dschwen | T104417 WikiMiniAtlas (wma.wmflabs.org) is still down | |||
| Resolved | coren | T103358 Labs: increase size of the volume for the maps project and restore | |||
| Resolved | coren | T103265 Labs: Salvage, then remove volumes on labstores' raid6 | |||
| Resolved | coren | T103268 Recover files for project liangent-php | |||
| Resolved | coren | T103350 Recover cssk tool's log files | |||
| Resolved | coren | T103205 Recover homedir of "ci" tool | |||
| Resolved | coren | T103689 Restore some files for tools-magog | |||
| Resolved | coren | T103696 Recover /data/project/home/botbot on bots project | |||
| Resolved | coren | T103708 Recovering my Python files in the root | |||
| Resolved | coren | T103725 Recover files for taxonbot | |||
| Resolved | coren | T104134 Recover some sql queries | |||
| Declined | coren | T105014 Log file needs restored. |
Event Timeline
Comment Actions
All but one user request has been satisfied, but we need to keep the broken filesystem around until we are done with an ongoing copy to labstore2001.
Comment Actions
This is now only pending on the copy to labstore2001 (in progress, from labstore1002)
Comment Actions
https://phabricator.wikimedia.org/P871 is list of files that are irretrievably corrupted.
Comment Actions
As an important caveat, those are the files for which no data can be recovered, but is not an exhaustive list of possibly damaged files (in particular, files that were being actively written to at and near the time of the crash are often cross-linked with each other in such a way that blocks from both can be read from them). Any log file is automatically suspect, for instance.
Comment Actions
Since there is now a copy to recover files from, removing the dependency now and asploding the broken filesystem.