Page MenuHomePhabricator

Delete CirrusSearch dumps for week 20170320 and 20170327
Closed, ResolvedPublic

Description

Recent CirrusSearch dumps generated have been missing a lot of data. Almost all the files in the 20170320 dump were missing data (which was subsequently deleted, but was archived here). The 20170327 dump has some files that are missing data as well.

Please do update once this gets fixed. I have temporarily disabled archiving of the CirrusSearch dumps until this is resolved.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
debt triaged this task as Medium priority.Apr 6 2017, 5:09 PM
debt subscribed.

We can delete the bad dumps and a recent fix should get the dumps working again. I don't think there is anything to do until the next dump run happens and it doesn't work.

Basically what happened is one kind of dump in our dumps infrastructure took up all the resources, and cirrussearch dumps were then either unable to allocate enough memory to start, or were killed by the linux OOM handler. I think we will just delete that weeks dumps as well, next week should go better.

Deskana renamed this task from Missing data in CirrusSearch dumps on 20170320 and 20170327 to Delete CirrusSearch dumps for week 20170320 and 20170327.Apr 20 2017, 5:15 PM
Deskana added subscribers: Gehel, Deskana.

Repurposing task to delete dumps as documented by @debt and @EBernhardson above. Is this something that @Gehel can help with?

Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.

We waited enough that those dumps have been automatically cleaned. We can probably close this task now.

debt claimed this task.

Cool, thanks for taking a look!