wdqs100[78] database corruption
Closed, ResolvedPublic

Description

wdqs1007 started to produce weird write errors (see whole log in https://github.com/blazegraph/database/issues/114) and looks like database is corrupted. Probably needs to be reloaded from another server.

Restricted Application added a project: Wikidata. · View Herald TranscriptTue, Jan 8, 12:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as Unbreak Now! priority.Tue, Jan 8, 12:22 AM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptTue, Jan 8, 12:22 AM
Smalyshev added a comment.EditedTue, Jan 8, 4:00 AM

@Gehel According to the discussion in https://github.com/blazegraph/database/issues/114, it might be possible to fix the journal. So let's not reload it until we have checked it.

Smalyshev added a comment.EditedTue, Jan 8, 4:26 AM

Same happening with wdq8, 3 hours later. Something spooky is going on... Will talk tomorrow morning with Bryan from Blazegraph, not sure if it's possible to do anything till then.

Mentioned in SAL (#wikimedia-operations) [2019-01-08T04:26:38Z] <onimisionipe> depooling wdqs1008 - T213134

Gehel added a comment.Tue, Jan 8, 9:47 AM

Same happening with wdq8, 3 hours later. Something spooky is going on... Will talk tomorrow morning with Bryan from Blazegraph, not sure if it's possible to do anything till then.

I'm not touching wdqs100[78] yet, so that you have whatever is needed for investigation.

Addshore renamed this task from wdqs1007 database corruption to wdqs100[78] database corruption.Tue, Jan 8, 11:51 AM
Addshore added a subscriber: Addshore.
Smalyshev added a comment.EditedTue, Jan 8, 7:14 PM

Long story short, the reason for the issue is that we've hit a hard limit on the number of allocators in Blazegraph. See https://wiki.blazegraph.com/wiki/index.php/FixedAllocators for details - there can't be any more than 256K allocators. We'll have to rearrange our data so that we use less allocators. Allocator usage can be seen under http://localhost:9999/bigdata/status?dumpJournal - looking at the servers show we have a lot of small allocator blocks used. We should re-arrange the data so that we don't use this much since it's a constrained resource and raising the limit is currently impossible in Blazegraph.

Action plan (tasks to follow shortly):

Immediate:

  • Copy database from wdq[345] to wdq7 and wdq8
  • Restore updates on wdq7 and wdq8
  • Collect allocator stats everywhere and see which servers are also in danger
  • Write an incident report

Sort-term:

  • Split category namespace into a separate instance of Blazegraph

Longer-term (will require data reload):

  • Disable "raw records" in Blazegraph
  • Consider inlining values & references
  • Consider setting INLINE_TEXT_LITERALS so short strings would be inlined, this doesn't use allocators
  • Check what other things could be inlined
Smalyshev closed this task as Resolved.Thu, Jan 10, 12:48 AM
Smalyshev claimed this task.

The servers are normal now, so I am closing this one and the rest will be done in T213210.