As described in T213134: wdqs100[78] database corruption, we've approached - and on some servers, exceeded - the allocator limit on Blazegraph. This task will aggregate all tasks we need to perform to solve the issue. Related incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20190110-WDQS
Action plan (tasks to follow shortly):
Immediate:
- Copy database from wdq[345] to wdq7 and wdq8
- Restore updates on wdq7 and wdq8
- Collect allocator stats everywhere and see which servers are also in danger
- Write an incident report
Sort-term:
- Split category namespace into a separate instance of Blazegraph (T213212)
Longer-term (will require data reload):
- Disable "raw records" in Blazegraph
- Consider inlining values & references
- Consider setting INLINE_TEXT_LITERALS so short strings would be inlined, this doesn't use allocators
- Check what other things could be inlined