To be done when labstore1002 is the active server, as part of the switchover test.
Things that may be worthwhile to inspect in detail:
A good load test is also likely to be necessary.
The original problem has not yet reoccurred since the last cold start, but we should do everything we can to be confident in the hardware before labstore1001 is switch back as primary.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | yuvipanda | T105720 Labs team reliability goal for Q1 2015/16 | |||
| Resolved | coren | T106479 Ensure that labstore machine is 'known good' hardware | |||
| Resolved | coren | T95293 Inspect and diagnose labstore1001's H800 controler | |||
| Declined | coren | T94607 Test labstore switchover | |||
| Resolved | None | T85604 Storage capacity & redundancy expansion (tracking) | |||
| Resolved | None | T85606 Replicate data between codfw and eqiad | |||
| Resolved | coren | T85605 Set storage service up in codfw | |||
| Resolved | coren | T93740 Upgrade labstore2001 to Jessie | |||
| Declined | yuvipanda | T85608 Process for user backups | |||
| Resolved | coren | T93792 Sync up the new labs NFS project filesystem with the live one | |||
| Resolved | coren | T85607 Increase storage available to labs NFS server | |||
| Resolved | coren | T91640 Upgrade labstore1002 to Jessie | |||
| Resolved | • Cmjohnson | T91677 labstore1002 fails to enter PERC bios, hangs on detecting devices |
@yuvipanda: No, the switchover test never took place and other concerns overrode this, and now labstore1001 is disconnected from the shelves.
At this point, I don't think we have any reason to believe the labstore1001 hardware is flaky. Hardware with known issue are: the labstore1002 H800 controler is known to randomly no pass POST, and one of the labstore2001 shelves is not working.
Resolved by the switchover test to end all switchover tests: labstore1001 is now back to being the primary server.