Page MenuHomePhabricator

dbstore1002 lagged on creating database
Closed, ResolvedPublic

Description

@Volans pinged me on dbstore1002 has stuck replication on x1 and s4 after new database creation. After I was able to check it x1 was already catched up but s4 just lagged.
After trying out several ways to fix it I thought the disk system performs bad (the host has spinning drives, and it's RAID is degraded T206965) I was thinking maybe stopping all other slaves will make the table creations able to finish.
I stopped all slaves, restarted s4, and the tables were created.
I restarted all the slaves, and all were catched up (s4 is still in progress as it gathered more than 10000 seconds of lag)

However I am still not sure why the empty table creations were not able to finish.
I'll take a look in this later.

Related Objects

Event Timeline

Banyek created this task.Jan 3 2019, 10:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2019, 10:44 PM
Banyek triaged this task as Medium priority.Jan 3 2019, 10:44 PM
Banyek moved this task from Triage to Backlog on the DBA board.
elukey awarded a token.Jan 4 2019, 6:50 AM
elukey added a subscriber: elukey.
Peachey88 updated the task description. (Show Details)Jan 4 2019, 9:08 AM

That's a known issue on dbstore1002 load+tokudb+indexes on the new tables.
It always happens with new wiki creations.
dbstore1002 and tokudb will go away soon with the new infra, so probably not worth spending time researching this :-)

Banyek closed this task as Resolved.Jan 4 2019, 10:27 AM
Banyek claimed this task.

hm, then I close the ticket, thanks @Marostegui