At the very least on the three good shelves in codfw to avoid having a single server hold all the data.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T103356 Labs: Make a new backup of the Labs storage to codfw | |||
| Resolved | Papaul | T102626 Labstore2001 controller or shelf failure |
Event Timeline
Comment Actions
Papaul is having some difficulty getting the shelves recognized in codfw. Getting two shelves attached to labstore1001 would allow us to make a copy to eqiad in the meantime.
Comment Actions
labstore2001 has the new controller, but it's proving to be amusing to configure to our needs.
In the meantime, and to make sure we do have a valid backup as quickly as possible, it would be possible to:
- Reuse the space once used by scratch and dumps on labstore1001, freeing 10 internal disks (~9T with raid10), enough to hold a backup
- scrap the incomplete restore of the maps project on labstore1002, freeing 2T for a snapshot
- make a snapshot of tools, back up to labstore1001
- make a snapshot of others, back up to labstore1001
@mark, what do you think?
Comment Actions
This is now in progress:
- The broken fs (/mnt/broken/project) is currently being copied to labstore2001:/srv/backup
- A snapshot of the good fs (/mnt/backup/project) is currently being copied to labstore1001:/srv/backup
Comment Actions
The broken mount is done, and there's a new copy of a new snapshot going to /srv/backup--* on labstore2001 atm