2 hour outage to update mysql on El slaves
Eventlogging mysql consumer needs to be stopped, we might want to debate whether we want to stop the whole system though. cc @Ottomata
What do you think?
2 hour outage to update mysql on El slaves
Eventlogging mysql consumer needs to be stopped, we might want to debate whether we want to stop the whole system though. cc @Ottomata
What do you think?
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | • Nuria | T120187 Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) | |||
Resolved | mforns | T121120 2 hour outage to update mysql on EL slaves {oryx} [3 pts] | |||
Resolved | mforns | T120209 Fix EL mysql consumer's deque push/pop {oryx} [3 pts] | |||
Resolved | mforns | T121384 Prepare EL queue sizes for backfilling {oryx} [3 pts] |
It is very important to send a notice to all users with enough time in advance. As the affected services are not yet clear, please remember it should be done as soon as it is decided.
BTW, this will only affect the MASTER (the data saving) not the slaves, that will continue to be available for querying, but not updated.
To deal with the mysql upgrade window, we analytics are planning to stop the (4) EL mysql consumers during that time span.
Kafka will buffer the events that come in the meantime, which will be automatically backfilled when the mysql consumers are restarted.
@jcrespo
Hi Jaime :]
When is the best time for you to do the mysql upgrade on Tuesday?
@jcrespo
The bug mentioned above has been fixed in production, we can proceed with the upgrade.
You should mark dependencies on this ticket, and/or mark them as resolved.
I proposed (T120187#1869800) 2 hours starting on 2015-12-15 at 10:00 UTC.
@jcrespo
I added the task as 'blocked by', and will close it as resolved in short.
Also, I'm ok with 2 hours starting on 2015-12-15 at 10:00 UTC. I will be logged in IRC and in Gmail at that time to help you in whatever I can.
Thanks!
Maintenance seems to have been done correctly, I will keep this ticket open for a few hours to check some minor issues, mostly to check that things are going correctly. But for now, inserts seem to be going, although slowly due to the recent restart.