In T226035 it was investigated why Druid public's broker lock up when a mw snapshot/datasource is dropped. The task was closed since we thought we had a solution, but the issue re-appeared.
We had another occurrence of AQS alarms related to the edit API failing while 2 druid datasources were cleaned up (drop issued by our systemd timer).
After a chat with Joseph and some digging into logs, we found the following:
- All the broker timeouts seem to be related to historicals locking up for the mediawiki_reduced datasource configured in AQS (so the last one), that it is of course not the one getting dropped.
- The coordinator leader tries to re-shuffle data around the cluster (5 segments at the time maximum), causing segments to be loaded/dropped by historicals. In this case, segments for mediawiki_reduced_2020_11 (the last datasource indexed) were moved to different hosts right after the drop of the old datasources.
- We have segments replicated 2 times (default_tier's default) and the avg load across the public cluster is 25% (segment-cache space used on every node). We could try to increase the replication to 3 times, but we might incur into the same issue described above. It would make the cluster more resilient afterwards.
We tried to do the following:
- Limit the datasource drop to one datasource only if possible. In theory it should be like this, but I found logs in the coordinator telling me that mediawiki_reduced_2020_06 and `mediawiki_reduced_2020_07 were dropped at the same time.
- Increase replication for segments to 3.
- Figure out a way to limit the data shuffling caused by the coordinator.
Then we started wondering if the cache settings were at fault, since the issue seems happening when the Brokers' cache is full. More info in the task.