In some cases, the database is not able to keep up with events. They accumulate, and eventually EventLogging issues a massive write (e.g. 70,000 records at once) which some part of the database write path can not handle.
This led to the outage from 2015-02-05 to 2015-02-10
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Nuria | T89269 Backfilling EL events from 20150206 to 20150210 | |||
| Resolved | • Nuria | T90029 Fix issue with database writes not keeping up with events |
While I do not think recent code patches changed anything core they definitely mitigated issues that affect db writes when there are bursts of traffic. Also, better error reporting will help to troubleshoot issues like this in the future.
After couple days of monitoring after deploying EL latest code we see no drops of events thus moving this item to done.
There are improvements that can be done to the current code regarding db writes.