Change Details

While setting up the new wcqs service on wcqs[12]00[123] we started an import, the same import process that we have run many times on the wcqs-beta instance. After two days half of the instances became unresponsive, and the remaining half appear to have stopped importing but have not 100% hung the instance (yet). Attached is the dmesg for all instances that were still accessible, in all cases there is a single moment where the kernel starts complaining about hung tasks related to disk IO. The kernel doesn't seem to complain about anything before or after that ~2 minute failure period. wcqs1001: P17667 wcqs1003: P17668 wcqs2002: P17669

While setting up the new wcqs service on wcqs[12]00[123] we started an import, the same import process that we have run many times on the wcqs-beta instance and once before on these instances. After two days half of the instances became unresponsive, and the remaining half appear to have stopped importing but have not 100% hung the instance (yet). Attached is the dmesg for all instances that were still accessible, in all cases there is a single moment where the kernel starts complaining about hung tasks related to disk IO. The kernel doesn't seem to complain about anything before or after that ~2 minute failure period. wcqs1001: P17667 wcqs1003: P17668 wcqs2002: P17669

While setting up the new wcqs service on wcqs[12]00[123] we started an import, the same import process that we have run many times on the wcqs-beta instance and once before on these instances. After two days half of the instances became unresponsive, and the remaining half appear to have stopped importing but have not 100% hung the instance (yet). Attached is the dmesg for all instances that were still accessible, in all cases there is a single moment where the kernel starts complaining about hung tasks related to disk IO. The kernel doesn't seem to complain about anything before or after that ~2 minute failure period. wcqs1001: P17667 wcqs1003: P17668 wcqs2002: P17669