-- Logs begin at Mon 2017-10-09 11:31:57 UTC, end at Wed 2017-10-11 05:20:26 UTC. -- Oct 11 03:18:17 dbstore2001 mysqld[44277]: InnoDB: Warning: Index qcc_type points to table uzwiki/querycachetwo and ib_table uzwiki/querycachetwo statistics is initialized 1 but index table uzwiki/querycachetwo initialized 0 mysql table is querycachetwo. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html Oct 11 03:18:17 dbstore2001 mysqld[44277]: InnoDB: Warning: Index qcc_type points to table uzwiki/querycachetwo and ib_table uzwiki/querycachetwo statistics is initialized 1 but index table uzwiki/querycachetwo initialized 1 mysql table is querycachetwo. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html Oct 11 03:18:17 dbstore2001 mysqld[44277]: InnoDB: Warning: Index qcc_title points to table uzwiki/querycachetwo and ib_table uzwiki/querycachetwo statistics is initialized 1 but index table uzwiki/querycachetwo initialized 1 mysql table is querycachetwo. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html Oct 11 03:18:17 dbstore2001 mysqld[44277]: InnoDB: Warning: Index qcc_titletwo points to table uzwiki/querycachetwo and ib_table uzwiki/querycachetwo statistics is initialized 1 but index table uzwiki/querycachetwo initialized 1 mysql table is querycachetwo. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html Oct 11 03:18:17 dbstore2001 mysqld[44277]: InnoDB: Warning: Index qcc_titletwo points to table uzwiki/querycachetwo and ib_table uzwiki/querycachetwo statistics is initialized 1 but index table uzwiki/querycachetwo initialized 1 mysql table is querycachetwo. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: clust index for table `uzwiki`.`querycachetwo` for index `qcc_type` not found Oct 11 03:26:30 dbstore2001 mysqld[44277]: 2017-10-11 03:26:30 7ff4a5464b00 InnoDB: Assertion failure in thread 140688721595136 in file row0row.cc line 652 Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: We intentionally generate a memory trap. Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com. Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: If you get repeated assertion failures or crashes, even Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: immediately after the mysqld startup, there may be Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: corruption in the InnoDB tablespace. Please refer to Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html Oct 11 03:26:30 dbstore2001 mysqld[44277]: InnoDB: about forcing recovery. Oct 11 03:26:30 dbstore2001 mysqld[44277]: 171011 3:26:30 [ERROR] mysqld got signal 6 ; Oct 11 03:26:30 dbstore2001 mysqld[44277]: This could be because you hit a bug. It is also possible that this binary Oct 11 03:26:30 dbstore2001 mysqld[44277]: or one of the libraries it was linked against is corrupt, improperly built, Oct 11 03:26:30 dbstore2001 mysqld[44277]: or misconfigured. This error can also be caused by malfunctioning hardware. Oct 11 03:26:30 dbstore2001 mysqld[44277]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Oct 11 03:26:30 dbstore2001 mysqld[44277]: We will try our best to scrape up some info that will hopefully help Oct 11 03:26:30 dbstore2001 mysqld[44277]: diagnose the problem, but since we have already crashed, Oct 11 03:26:30 dbstore2001 mysqld[44277]: something is definitely wrong and this may fail. Oct 11 03:26:30 dbstore2001 mysqld[44277]: Server version: 10.1.26-MariaDB Oct 11 03:26:30 dbstore2001 mysqld[44277]: key_buffer_size=1048576 Oct 11 03:26:30 dbstore2001 mysqld[44277]: read_buffer_size=131072 Oct 11 03:26:30 dbstore2001 mysqld[44277]: max_used_connections=22 Oct 11 03:26:30 dbstore2001 mysqld[44277]: max_threads=252 Oct 11 03:26:30 dbstore2001 mysqld[44277]: thread_count=4 Oct 11 03:26:30 dbstore2001 mysqld[44277]: It is possible that mysqld could use up to Oct 11 03:26:30 dbstore2001 mysqld[44277]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 554550 K bytes of memory Oct 11 03:26:30 dbstore2001 mysqld[44277]: Hope that's ok; if not, decrease some variables in the equation. Oct 11 03:26:30 dbstore2001 mysqld[44277]: Thread pointer: 0x7ff06c82b008 Oct 11 03:26:30 dbstore2001 mysqld[44277]: Attempting backtrace. You can use the following information to find out Oct 11 03:26:30 dbstore2001 mysqld[44277]: where mysqld died. If you see no messages after this, something went Oct 11 03:26:30 dbstore2001 mysqld[44277]: terribly wrong... Oct 11 03:26:30 dbstore2001 mysqld[44277]: stack_bottom = 0x7ff4a5463838 thread_stack 0x48400 Oct 11 03:26:31 dbstore2001 mysqld[44277]: *** buffer overflow detected ***: /opt/wmf-mariadb101//bin/mysqld terminated Oct 11 03:26:31 dbstore2001 mysqld[44277]: ======= Backtrace: ========= Oct 11 03:26:31 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7ff4a67a8bcb] Oct 11 03:26:31 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ff4a6831227] Oct 11 03:26:31 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(+0xf7360)[0x7ff4a682f360] Oct 11 03:26:31 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(+0xf919a)[0x7ff4a683119a] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(my_addr_resolve+0x4c)[0x55782413336c] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(my_print_stacktrace+0x1bb)[0x55782411caab] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(handle_fatal_signal+0x3bd)[0x557823c61f4d] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7ff4a82fc0c0] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7ff4a676afcf] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7ff4a676c3fa] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(+0x8ea890)[0x557823fa9890] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(+0x8f114e)[0x557823fb014e] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(+0x82e031)[0x557823eed031] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x211)[0x557823c66c91] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x62)[0x557823c6b962] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN7handler21multi_range_read_nextEPPv+0x88)[0x557823bd7aa8] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN23Mrr_simple_index_reader8get_nextEPPv+0x20)[0x557823bd7b30] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN10DsMrr_impl10dsmrr_nextEPPv+0x42)[0x557823bd8ee2] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x81)[0x557823d3edb1] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(+0x69dd95)[0x557823d5cd95] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyP13select_result+0x112a)[0x557823d7c08a] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_Z21mysql_execute_commandP3THD+0x2e58)[0x557823acf708] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x311)[0x557823ad5801] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x1371)[0x557823d2bed1] Oct 11 03:26:32 dbstore2001 mysqld[44277]: /opt/wmf-mariadb101//bin/mysqld(+0x38e46b)[0x557823a4d46b] Oct 11 03:26:35 dbstore2001 systemd[1]: mariadb@s3.service: Main process exited, code=killed, status=6/ABRT Oct 11 03:26:35 dbstore2001 systemd[1]: mariadb@s3.service: Unit entered failed state. Oct 11 03:26:35 dbstore2001 systemd[1]: mariadb@s3.service: Failed with result 'signal'. Oct 11 03:26:41 dbstore2001 systemd[1]: mariadb@s3.service: Service hold-off time over, scheduling restart. Oct 11 03:26:41 dbstore2001 systemd[1]: Stopped MariaDB database server. Oct 11 03:26:41 dbstore2001 systemd[1]: Starting MariaDB database server... Oct 11 03:26:43 dbstore2001 mysqld[2693]: 2017-10-11 3:26:43 140145493571840 [Note] /opt/wmf-mariadb101//bin/mysqld (mysqld 10.1.26-MariaDB) starting as process 2693 ... Oct 11 03:26:44 dbstore2001 mysqld[2693]: 2017-10-11 3:26:44 140145493571840 [ERROR] Plugin 'unix_socket' already installed Oct 11 03:26:44 dbstore2001 mysqld[2693]: 2017-10-11 03:26:44 7f762a5c7900 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html. Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Using mutexes to ref count buffer pool pages Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: The InnoDB memory heap is disabled Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Compressed tables use zlib 1.2.3 Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Using Linux native AIO Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Using SSE crc32 instructions Oct 11 03:26:45 dbstore2001 mysqld[2693]: 2017-10-11 3:26:45 140145493571840 [Note] InnoDB: Initializing buffer pool, size = 15.0G Oct 11 03:26:46 dbstore2001 mysqld[2693]: 2017-10-11 3:26:46 140145493571840 [Note] InnoDB: Completed initialization of buffer pool Oct 11 03:26:46 dbstore2001 mysqld[2693]: 2017-10-11 3:26:46 140145493571840 [Note] InnoDB: Highest supported file format is Barracuda. Oct 11 03:26:46 dbstore2001 mysqld[2693]: 2017-10-11 3:26:46 140145493571840 [Note] InnoDB: Starting crash recovery from checkpoint LSN=17828766693176 Oct 11 03:27:02 dbstore2001 mysqld[2693]: 2017-10-11 3:27:02 140145493571840 [Note] InnoDB: Processed 286 .ibd/.isl files Oct 11 03:27:19 dbstore2001 mysqld[2693]: 2017-10-11 3:27:19 140145493571840 [Note] InnoDB: Processed 484 .ibd/.isl files Oct 11 03:27:35 dbstore2001 mysqld[2693]: 2017-10-11 3:27:35 140145493571840 [Note] InnoDB: Processed 594 .ibd/.isl files Oct 11 03:27:54 dbstore2001 mysqld[2693]: 2017-10-11 3:27:54 140145493571840 [Note] InnoDB: Processed 682 .ibd/.isl files Oct 11 03:28:11 dbstore2001 mysqld[2693]: 2017-10-11 3:28:11 140145493571840 [Note] InnoDB: Processed 759 .ibd/.isl files Oct 11 03:28:30 dbstore2001 mysqld[2693]: 2017-10-11 3:28:30 140145493571840 [Note] InnoDB: Processed 847 .ibd/.isl files Oct 11 03:28:47 dbstore2001 mysqld[2693]: 2017-10-11 3:28:47 140145493571840 [Note] InnoDB: Processed 935 .ibd/.isl files Oct 11 03:29:05 dbstore2001 mysqld[2693]: 2017-10-11 3:29:05 140145493571840 [Note] InnoDB: Processed 990 .ibd/.isl files Oct 11 03:29:21 dbstore2001 mysqld[2693]: 2017-10-11 3:29:21 140145493571840 [Note] InnoDB: Processed 1100 .ibd/.isl files Oct 11 03:29:37 dbstore2001 mysqld[2693]: 2017-10-11 3:29:37 140145493571840 [Note] InnoDB: Processed 1375 .ibd/.isl files Oct 11 03:29:53 dbstore2001 mysqld[2693]: 2017-10-11 3:29:53 140145493571840 [Note] InnoDB: Processed 1727 .ibd/.isl files Oct 11 03:30:09 dbstore2001 mysqld[2693]: 2017-10-11 3:30:09 140145493571840 [Note] InnoDB: Processed 2277 .ibd/.isl files Oct 11 03:30:25 dbstore2001 mysqld[2693]: 2017-10-11 3:30:25 140145493571840 [Note] InnoDB: Processed 2739 .ibd/.isl files Oct 11 03:30:41 dbstore2001 mysqld[2693]: 2017-10-11 3:30:41 140145493571840 [Note] InnoDB: Processed 3080 .ibd/.isl files Oct 11 03:30:57 dbstore2001 mysqld[2693]: 2017-10-11 3:30:57 140145493571840 [Note] InnoDB: Processed 3399 .ibd/.isl files Oct 11 03:31:13 dbstore2001 mysqld[2693]: 2017-10-11 3:31:13 140145493571840 [Note] InnoDB: Processed 3707 .ibd/.isl files Oct 11 03:31:29 dbstore2001 mysqld[2693]: 2017-10-11 3:31:29 140145493571840 [Note] InnoDB: Processed 4059 .ibd/.isl files Oct 11 03:31:45 dbstore2001 mysqld[2693]: 2017-10-11 3:31:45 140145493571840 [Note] InnoDB: Processed 4411 .ibd/.isl files Oct 11 03:32:01 dbstore2001 mysqld[2693]: 2017-10-11 3:32:01 140145493571840 [Note] InnoDB: Processed 4873 .ibd/.isl files Oct 11 03:32:17 dbstore2001 mysqld[2693]: 2017-10-11 3:32:17 140145493571840 [Note] InnoDB: Processed 5445 .ibd/.isl files Oct 11 03:32:33 dbstore2001 mysqld[2693]: 2017-10-11 3:32:33 140145493571840 [Note] InnoDB: Processed 5940 .ibd/.isl files Oct 11 03:32:49 dbstore2001 mysqld[2693]: 2017-10-11 3:32:49 140145493571840 [Note] InnoDB: Processed 6204 .ibd/.isl files Oct 11 03:33:05 dbstore2001 mysqld[2693]: 2017-10-11 3:33:05 140145493571840 [Note] InnoDB: Processed 6369 .ibd/.isl files Oct 11 03:33:23 dbstore2001 mysqld[2693]: 2017-10-11 3:33:23 140145493571840 [Note] InnoDB: Processed 6512 .ibd/.isl files Oct 11 03:33:39 dbstore2001 mysqld[2693]: 2017-10-11 3:33:39 140145493571840 [Note] InnoDB: Processed 6776 .ibd/.isl files Oct 11 03:33:55 dbstore2001 mysqld[2693]: 2017-10-11 3:33:55 140145493571840 [Note] InnoDB: Processed 7051 .ibd/.isl files Oct 11 03:34:11 dbstore2001 mysqld[2693]: 2017-10-11 3:34:11 140145493571840 [Note] InnoDB: Processed 7513 .ibd/.isl files Oct 11 03:34:27 dbstore2001 mysqld[2693]: 2017-10-11 3:34:27 140145493571840 [Note] InnoDB: Processed 7964 .ibd/.isl files Oct 11 03:34:43 dbstore2001 mysqld[2693]: 2017-10-11 3:34:43 140145493571840 [Note] InnoDB: Processed 8580 .ibd/.isl files Oct 11 03:34:59 dbstore2001 mysqld[2693]: 2017-10-11 3:34:59 140145493571840 [Note] InnoDB: Processed 9042 .ibd/.isl files Oct 11 03:35:15 dbstore2001 mysqld[2693]: 2017-10-11 3:35:15 140145493571840 [Note] InnoDB: Processed 9405 .ibd/.isl files Oct 11 03:35:31 dbstore2001 mysqld[2693]: 2017-10-11 3:35:31 140145493571840 [Note] InnoDB: Processed 9900 .ibd/.isl files Oct 11 03:35:47 dbstore2001 mysqld[2693]: 2017-10-11 3:35:47 140145493571840 [Note] InnoDB: Processed 10505 .ibd/.isl files Oct 11 03:36:03 dbstore2001 mysqld[2693]: 2017-10-11 3:36:03 140145493571840 [Note] InnoDB: Processed 11099 .ibd/.isl files Oct 11 03:36:19 dbstore2001 mysqld[2693]: 2017-10-11 3:36:19 140145493571840 [Note] InnoDB: Processed 11539 .ibd/.isl files Oct 11 03:36:35 dbstore2001 mysqld[2693]: 2017-10-11 3:36:35 140145493571840 [Note] InnoDB: Processed 11891 .ibd/.isl files Oct 11 03:36:51 dbstore2001 mysqld[2693]: 2017-10-11 3:36:51 140145493571840 [Note] InnoDB: Processed 12155 .ibd/.isl files Oct 11 03:37:07 dbstore2001 mysqld[2693]: 2017-10-11 3:37:07 140145493571840 [Note] InnoDB: Processed 12353 .ibd/.isl files Oct 11 03:37:24 dbstore2001 mysqld[2693]: 2017-10-11 3:37:24 140145493571840 [Note] InnoDB: Processed 12617 .ibd/.isl files Oct 11 03:37:40 dbstore2001 mysqld[2693]: 2017-10-11 3:37:40 140145493571840 [Note] InnoDB: Processed 12881 .ibd/.isl files Oct 11 03:37:56 dbstore2001 mysqld[2693]: 2017-10-11 3:37:56 140145493571840 [Note] InnoDB: Processed 13211 .ibd/.isl files Oct 11 03:38:12 dbstore2001 mysqld[2693]: 2017-10-11 3:38:12 140145493571840 [Note] InnoDB: Processed 13640 .ibd/.isl files Oct 11 03:38:28 dbstore2001 mysqld[2693]: 2017-10-11 3:38:28 140145493571840 [Note] InnoDB: Processed 14201 .ibd/.isl files Oct 11 03:38:44 dbstore2001 mysqld[2693]: 2017-10-11 3:38:44 140145493571840 [Note] InnoDB: Processed 14553 .ibd/.isl files Oct 11 03:39:00 dbstore2001 mysqld[2693]: 2017-10-11 3:39:00 140145493571840 [Note] InnoDB: Processed 14839 .ibd/.isl files Oct 11 03:39:16 dbstore2001 mysqld[2693]: 2017-10-11 3:39:16 140145493571840 [Note] InnoDB: Processed 15213 .ibd/.isl files Oct 11 03:39:32 dbstore2001 mysqld[2693]: 2017-10-11 3:39:32 140145493571840 [Note] InnoDB: Processed 15620 .ibd/.isl files Oct 11 03:39:48 dbstore2001 mysqld[2693]: 2017-10-11 3:39:48 140145493571840 [Note] InnoDB: Processed 16126 .ibd/.isl files Oct 11 03:40:04 dbstore2001 mysqld[2693]: 2017-10-11 3:40:04 140145493571840 [Note] InnoDB: Processed 16698 .ibd/.isl files Oct 11 03:40:20 dbstore2001 mysqld[2693]: 2017-10-11 3:40:20 140145493571840 [Note] InnoDB: Processed 17237 .ibd/.isl files Oct 11 03:40:36 dbstore2001 mysqld[2693]: 2017-10-11 3:40:36 140145493571840 [Note] InnoDB: Processed 17556 .ibd/.isl files Oct 11 03:40:52 dbstore2001 mysqld[2693]: 2017-10-11 3:40:52 140145493571840 [Note] InnoDB: Processed 17908 .ibd/.isl files Oct 11 03:41:08 dbstore2001 mysqld[2693]: 2017-10-11 3:41:08 140145493571840 [Note] InnoDB: Processed 18370 .ibd/.isl files Oct 11 03:41:24 dbstore2001 mysqld[2693]: 2017-10-11 3:41:24 140145493571840 [Note] InnoDB: Processed 18942 .ibd/.isl files Oct 11 03:41:40 dbstore2001 mysqld[2693]: 2017-10-11 3:41:40 140145493571840 [Note] InnoDB: Processed 19437 .ibd/.isl files Oct 11 03:41:56 dbstore2001 mysqld[2693]: 2017-10-11 3:41:56 140145493571840 [Note] InnoDB: Processed 19943 .ibd/.isl files Oct 11 03:42:12 dbstore2001 mysqld[2693]: 2017-10-11 3:42:12 140145493571840 [Note] InnoDB: Processed 20306 .ibd/.isl files Oct 11 03:42:28 dbstore2001 mysqld[2693]: 2017-10-11 3:42:28 140145493571840 [Note] InnoDB: Processed 20625 .ibd/.isl files Oct 11 03:42:44 dbstore2001 mysqld[2693]: 2017-10-11 3:42:44 140145493571840 [Note] InnoDB: Processed 20955 .ibd/.isl files Oct 11 03:43:00 dbstore2001 mysqld[2693]: 2017-10-11 3:43:00 140145493571840 [Note] InnoDB: Processed 21329 .ibd/.isl files Oct 11 03:43:16 dbstore2001 mysqld[2693]: 2017-10-11 3:43:16 140145493571840 [Note] InnoDB: Processed 21725 .ibd/.isl files Oct 11 03:43:32 dbstore2001 mysqld[2693]: 2017-10-11 3:43:32 140145493571840 [Note] InnoDB: Processed 22242 .ibd/.isl files Oct 11 03:43:48 dbstore2001 mysqld[2693]: 2017-10-11 3:43:48 140145493571840 [Note] InnoDB: Processed 22858 .ibd/.isl files Oct 11 03:44:04 dbstore2001 mysqld[2693]: 2017-10-11 3:44:04 140145493571840 [Note] InnoDB: Processed 23320 .ibd/.isl files Oct 11 03:44:20 dbstore2001 mysqld[2693]: 2017-10-11 3:44:20 140145493571840 [Note] InnoDB: Processed 23694 .ibd/.isl files Oct 11 03:44:36 dbstore2001 mysqld[2693]: 2017-10-11 3:44:36 140145493571840 [Note] InnoDB: Processed 24101 .ibd/.isl files Oct 11 03:44:52 dbstore2001 mysqld[2693]: 2017-10-11 3:44:52 140145493571840 [Note] InnoDB: Processed 24519 .ibd/.isl files Oct 11 03:45:08 dbstore2001 mysqld[2693]: 2017-10-11 3:45:08 140145493571840 [Note] InnoDB: Processed 24970 .ibd/.isl files Oct 11 03:45:24 dbstore2001 mysqld[2693]: 2017-10-11 3:45:24 140145493571840 [Note] InnoDB: Processed 25388 .ibd/.isl files Oct 11 03:45:40 dbstore2001 mysqld[2693]: 2017-10-11 3:45:40 140145493571840 [Note] InnoDB: Processed 25905 .ibd/.isl files Oct 11 03:45:56 dbstore2001 mysqld[2693]: 2017-10-11 3:45:56 140145493571840 [Note] InnoDB: Processed 26312 .ibd/.isl files Oct 11 03:46:12 dbstore2001 mysqld[2693]: 2017-10-11 3:46:12 140145493571840 [Note] InnoDB: Processed 26587 .ibd/.isl files Oct 11 03:46:28 dbstore2001 mysqld[2693]: 2017-10-11 3:46:28 140145493571840 [Note] InnoDB: Processed 26939 .ibd/.isl files Oct 11 03:46:44 dbstore2001 mysqld[2693]: 2017-10-11 3:46:44 140145493571840 [Note] InnoDB: Processed 27324 .ibd/.isl files Oct 11 03:47:00 dbstore2001 mysqld[2693]: 2017-10-11 3:47:00 140145493571840 [Note] InnoDB: Processed 27753 .ibd/.isl files Oct 11 03:47:16 dbstore2001 mysqld[2693]: 2017-10-11 3:47:16 140145493571840 [Note] InnoDB: Processed 28248 .ibd/.isl files Oct 11 03:47:32 dbstore2001 mysqld[2693]: 2017-10-11 3:47:32 140145493571840 [Note] InnoDB: Processed 28765 .ibd/.isl files Oct 11 03:47:48 dbstore2001 mysqld[2693]: 2017-10-11 3:47:48 140145493571840 [Note] InnoDB: Processed 29260 .ibd/.isl files Oct 11 03:48:04 dbstore2001 mysqld[2693]: 2017-10-11 3:48:04 140145493571840 [Note] InnoDB: Processed 29689 .ibd/.isl files Oct 11 03:48:20 dbstore2001 mysqld[2693]: 2017-10-11 3:48:20 140145493571840 [Note] InnoDB: Processed 29986 .ibd/.isl files Oct 11 03:48:36 dbstore2001 mysqld[2693]: 2017-10-11 3:48:36 140145493571840 [Note] InnoDB: Processed 30327 .ibd/.isl files Oct 11 03:48:52 dbstore2001 mysqld[2693]: 2017-10-11 3:48:52 140145493571840 [Note] InnoDB: Processed 30712 .ibd/.isl files Oct 11 03:49:08 dbstore2001 mysqld[2693]: 2017-10-11 3:49:08 140145493571840 [Note] InnoDB: Processed 31119 .ibd/.isl files Oct 11 03:49:24 dbstore2001 mysqld[2693]: 2017-10-11 3:49:24 140145493571840 [Note] InnoDB: Processed 31548 .ibd/.isl files Oct 11 03:49:40 dbstore2001 mysqld[2693]: 2017-10-11 3:49:40 140145493571840 [Note] InnoDB: Processed 32164 .ibd/.isl files Oct 11 03:49:56 dbstore2001 mysqld[2693]: 2017-10-11 3:49:56 140145493571840 [Note] InnoDB: Processed 32571 .ibd/.isl files Oct 11 03:50:12 dbstore2001 mysqld[2693]: 2017-10-11 3:50:12 140145493571840 [Note] InnoDB: Processed 32890 .ibd/.isl files Oct 11 03:50:28 dbstore2001 mysqld[2693]: 2017-10-11 3:50:28 140145493571840 [Note] InnoDB: Processed 33187 .ibd/.isl files Oct 11 03:50:44 dbstore2001 mysqld[2693]: 2017-10-11 3:50:44 140145493571840 [Note] InnoDB: Processed 33462 .ibd/.isl files Oct 11 03:51:00 dbstore2001 mysqld[2693]: 2017-10-11 3:51:00 140145493571840 [Note] InnoDB: Processed 33825 .ibd/.isl files Oct 11 03:51:16 dbstore2001 mysqld[2693]: 2017-10-11 3:51:16 140145493571840 [Note] InnoDB: Processed 34188 .ibd/.isl files Oct 11 03:51:32 dbstore2001 mysqld[2693]: 2017-10-11 3:51:32 140145493571840 [Note] InnoDB: Processed 34639 .ibd/.isl files Oct 11 03:51:48 dbstore2001 mysqld[2693]: 2017-10-11 3:51:48 140145493571840 [Note] InnoDB: Processed 35189 .ibd/.isl files Oct 11 03:52:04 dbstore2001 mysqld[2693]: 2017-10-11 3:52:04 140145493571840 [Note] InnoDB: Processed 35585 .ibd/.isl files Oct 11 03:52:20 dbstore2001 mysqld[2693]: 2017-10-11 3:52:20 140145493571840 [Note] InnoDB: Processed 35915 .ibd/.isl files Oct 11 03:52:36 dbstore2001 mysqld[2693]: 2017-10-11 3:52:36 140145493571840 [Note] InnoDB: Processed 36179 .ibd/.isl files Oct 11 03:52:52 dbstore2001 mysqld[2693]: 2017-10-11 3:52:52 140145493571840 [Note] InnoDB: Processed 36553 .ibd/.isl files Oct 11 03:53:08 dbstore2001 mysqld[2693]: 2017-10-11 3:53:08 140145493571840 [Note] InnoDB: Processed 36894 .ibd/.isl files Oct 11 03:53:24 dbstore2001 mysqld[2693]: 2017-10-11 3:53:24 140145493571840 [Note] InnoDB: Processed 37290 .ibd/.isl files Oct 11 03:53:40 dbstore2001 mysqld[2693]: 2017-10-11 3:53:40 140145493571840 [Note] InnoDB: Processed 37807 .ibd/.isl files Oct 11 03:53:56 dbstore2001 mysqld[2693]: 2017-10-11 3:53:56 140145493571840 [Note] InnoDB: Processed 38258 .ibd/.isl files Oct 11 03:54:12 dbstore2001 mysqld[2693]: 2017-10-11 3:54:12 140145493571840 [Note] InnoDB: Processed 38599 .ibd/.isl files Oct 11 03:54:28 dbstore2001 mysqld[2693]: 2017-10-11 3:54:28 140145493571840 [Note] InnoDB: Processed 38863 .ibd/.isl files Oct 11 03:54:44 dbstore2001 mysqld[2693]: 2017-10-11 3:54:44 140145493571840 [Note] InnoDB: Processed 39171 .ibd/.isl files Oct 11 03:55:00 dbstore2001 mysqld[2693]: 2017-10-11 3:55:00 140145493571840 [Note] InnoDB: Processed 39468 .ibd/.isl files Oct 11 03:55:16 dbstore2001 mysqld[2693]: 2017-10-11 3:55:16 140145493571840 [Note] InnoDB: Processed 39798 .ibd/.isl files Oct 11 03:55:32 dbstore2001 mysqld[2693]: 2017-10-11 3:55:32 140145493571840 [Note] InnoDB: Processed 40150 .ibd/.isl files Oct 11 03:55:48 dbstore2001 mysqld[2693]: 2017-10-11 3:55:48 140145493571840 [Note] InnoDB: Processed 40546 .ibd/.isl files Oct 11 03:56:04 dbstore2001 mysqld[2693]: 2017-10-11 3:56:04 140145493571840 [Note] InnoDB: Processed 41107 .ibd/.isl files Oct 11 03:56:20 dbstore2001 mysqld[2693]: 2017-10-11 3:56:20 140145493571840 [Note] InnoDB: Processed 41569 .ibd/.isl files Oct 11 03:56:36 dbstore2001 mysqld[2693]: 2017-10-11 3:56:36 140145493571840 [Note] InnoDB: Processed 41877 .ibd/.isl files Oct 11 03:56:52 dbstore2001 mysqld[2693]: 2017-10-11 3:56:52 140145493571840 [Note] InnoDB: Processed 42185 .ibd/.isl files Oct 11 03:57:08 dbstore2001 mysqld[2693]: 2017-10-11 3:57:08 140145493571840 [Note] InnoDB: Processed 42493 .ibd/.isl files Oct 11 03:57:24 dbstore2001 mysqld[2693]: 2017-10-11 3:57:24 140145493571840 [Note] InnoDB: Processed 42823 .ibd/.isl files Oct 11 03:57:40 dbstore2001 mysqld[2693]: 2017-10-11 3:57:40 140145493571840 [Note] InnoDB: Processed 43197 .ibd/.isl files Oct 11 03:57:56 dbstore2001 mysqld[2693]: 2017-10-11 3:57:56 140145493571840 [Note] InnoDB: Processed 43626 .ibd/.isl files Oct 11 03:58:12 dbstore2001 mysqld[2693]: 2017-10-11 3:58:12 140145493571840 [Note] InnoDB: Processed 44198 .ibd/.isl files Oct 11 03:58:28 dbstore2001 mysqld[2693]: 2017-10-11 3:58:28 140145493571840 [Note] InnoDB: Processed 44605 .ibd/.isl files Oct 11 03:58:44 dbstore2001 mysqld[2693]: 2017-10-11 3:58:44 140145493571840 [Note] InnoDB: Processed 44902 .ibd/.isl files Oct 11 03:59:00 dbstore2001 mysqld[2693]: 2017-10-11 3:59:00 140145493571840 [Note] InnoDB: Processed 45199 .ibd/.isl files Oct 11 03:59:16 dbstore2001 mysqld[2693]: 2017-10-11 3:59:16 140145493571840 [Note] InnoDB: Processed 45529 .ibd/.isl files Oct 11 03:59:32 dbstore2001 mysqld[2693]: 2017-10-11 3:59:32 140145493571840 [Note] InnoDB: Processed 45793 .ibd/.isl files Oct 11 03:59:48 dbstore2001 mysqld[2693]: 2017-10-11 3:59:48 140145493571840 [Note] InnoDB: Processed 46200 .ibd/.isl files Oct 11 04:00:04 dbstore2001 mysqld[2693]: 2017-10-11 4:00:04 140145493571840 [Note] InnoDB: Processed 46618 .ibd/.isl files Oct 11 04:00:20 dbstore2001 mysqld[2693]: 2017-10-11 4:00:20 140145493571840 [Note] InnoDB: Processed 47146 .ibd/.isl files Oct 11 04:00:36 dbstore2001 mysqld[2693]: 2017-10-11 4:00:36 140145493571840 [Note] InnoDB: Processed 47685 .ibd/.isl files Oct 11 04:00:52 dbstore2001 mysqld[2693]: 2017-10-11 4:00:52 140145493571840 [Note] InnoDB: Processed 47982 .ibd/.isl files Oct 11 04:01:08 dbstore2001 mysqld[2693]: 2017-10-11 4:01:08 140145493571840 [Note] InnoDB: Processed 48290 .ibd/.isl files Oct 11 04:01:24 dbstore2001 mysqld[2693]: 2017-10-11 4:01:24 140145493571840 [Note] InnoDB: Processed 48576 .ibd/.isl files Oct 11 04:01:40 dbstore2001 mysqld[2693]: 2017-10-11 4:01:40 140145493571840 [Note] InnoDB: Processed 48895 .ibd/.isl files Oct 11 04:01:56 dbstore2001 mysqld[2693]: 2017-10-11 4:01:56 140145493571840 [Note] InnoDB: Processed 49192 .ibd/.isl files Oct 11 04:02:12 dbstore2001 mysqld[2693]: 2017-10-11 4:02:12 140145493571840 [Note] InnoDB: Processed 49577 .ibd/.isl files Oct 11 04:02:28 dbstore2001 mysqld[2693]: 2017-10-11 4:02:28 140145493571840 [Note] InnoDB: Processed 49995 .ibd/.isl files Oct 11 04:02:44 dbstore2001 mysqld[2693]: 2017-10-11 4:02:44 140145493571840 [Note] InnoDB: Processed 50534 .ibd/.isl files Oct 11 04:03:00 dbstore2001 mysqld[2693]: 2017-10-11 4:03:00 140145493571840 [Note] InnoDB: Processed 50985 .ibd/.isl files Oct 11 04:03:16 dbstore2001 mysqld[2693]: 2017-10-11 4:03:16 140145493571840 [Note] InnoDB: Processed 51216 .ibd/.isl files Oct 11 04:03:32 dbstore2001 mysqld[2693]: 2017-10-11 4:03:32 140145493571840 [Note] InnoDB: Processed 51469 .ibd/.isl files Oct 11 04:03:48 dbstore2001 mysqld[2693]: 2017-10-11 4:03:48 140145493571840 [Note] InnoDB: Processed 51700 .ibd/.isl files Oct 11 04:04:04 dbstore2001 mysqld[2693]: 2017-10-11 4:04:04 140145493571840 [Note] InnoDB: Processed 51997 .ibd/.isl files Oct 11 04:04:20 dbstore2001 mysqld[2693]: 2017-10-11 4:04:20 140145493571840 [Note] InnoDB: Processed 52305 .ibd/.isl files Oct 11 04:04:36 dbstore2001 mysqld[2693]: 2017-10-11 4:04:36 140145493571840 [Note] InnoDB: Processed 52701 .ibd/.isl files Oct 11 04:04:52 dbstore2001 mysqld[2693]: 2017-10-11 4:04:52 140145493571840 [Note] InnoDB: Processed 53152 .ibd/.isl files Oct 11 04:05:08 dbstore2001 mysqld[2693]: 2017-10-11 4:05:08 140145493571840 [Note] InnoDB: Processed 53680 .ibd/.isl files Oct 11 04:05:24 dbstore2001 mysqld[2693]: 2017-10-11 4:05:24 140145493571840 [Note] InnoDB: Processed 54087 .ibd/.isl files Oct 11 04:05:40 dbstore2001 mysqld[2693]: 2017-10-11 4:05:40 140145493571840 [Note] InnoDB: Processed 54351 .ibd/.isl files Oct 11 04:05:56 dbstore2001 mysqld[2693]: 2017-10-11 4:05:56 140145493571840 [Note] InnoDB: Processed 54626 .ibd/.isl files Oct 11 04:06:12 dbstore2001 mysqld[2693]: 2017-10-11 4:06:12 140145493571840 [Note] InnoDB: Processed 54912 .ibd/.isl files Oct 11 04:06:28 dbstore2001 mysqld[2693]: 2017-10-11 4:06:28 140145493571840 [Note] InnoDB: Processed 55264 .ibd/.isl files Oct 11 04:06:44 dbstore2001 mysqld[2693]: 2017-10-11 4:06:44 140145493571840 [Note] InnoDB: Processed 55660 .ibd/.isl files Oct 11 04:07:00 dbstore2001 mysqld[2693]: 2017-10-11 4:07:00 140145493571840 [Note] InnoDB: Processed 56199 .ibd/.isl files Oct 11 04:07:16 dbstore2001 mysqld[2693]: 2017-10-11 4:07:16 140145493571840 [Note] InnoDB: Processed 56727 .ibd/.isl files Oct 11 04:07:32 dbstore2001 mysqld[2693]: 2017-10-11 4:07:32 140145493571840 [Note] InnoDB: Processed 57101 .ibd/.isl files Oct 11 04:07:48 dbstore2001 mysqld[2693]: 2017-10-11 4:07:48 140145493571840 [Note] InnoDB: Processed 57387 .ibd/.isl files Oct 11 04:08:04 dbstore2001 mysqld[2693]: 2017-10-11 4:08:04 140145493571840 [Note] InnoDB: Processed 57750 .ibd/.isl files Oct 11 04:08:20 dbstore2001 mysqld[2693]: 2017-10-11 4:08:20 140145493571840 [Note] InnoDB: Processed 58278 .ibd/.isl files Oct 11 04:08:36 dbstore2001 mysqld[2693]: 2017-10-11 4:08:36 140145493571840 [Note] InnoDB: Processed 58839 .ibd/.isl files Oct 11 04:08:52 dbstore2001 mysqld[2693]: 2017-10-11 4:08:52 140145493571840 [Note] InnoDB: Processed 59488 .ibd/.isl files Oct 11 04:09:08 dbstore2001 mysqld[2693]: 2017-10-11 4:09:08 140145493571840 [Note] InnoDB: Processed 60115 .ibd/.isl files Oct 11 04:09:24 dbstore2001 mysqld[2693]: 2017-10-11 4:09:24 140145493571840 [Note] InnoDB: Processed 60676 .ibd/.isl files Oct 11 04:09:40 dbstore2001 mysqld[2693]: 2017-10-11 4:09:40 140145493571840 [Note] InnoDB: Processed 61303 .ibd/.isl files Oct 11 04:09:56 dbstore2001 mysqld[2693]: 2017-10-11 4:09:56 140145493571840 [Note] InnoDB: Processed 61985 .ibd/.isl files Oct 11 04:10:12 dbstore2001 mysqld[2693]: 2017-10-11 4:10:12 140145493571840 [Note] InnoDB: Processed 62656 .ibd/.isl files Oct 11 04:10:28 dbstore2001 mysqld[2693]: 2017-10-11 4:10:28 140145493571840 [Note] InnoDB: Processed 63228 .ibd/.isl files Oct 11 04:10:44 dbstore2001 mysqld[2693]: 2017-10-11 4:10:44 140145493571840 [Note] InnoDB: Processed 63811 .ibd/.isl files Oct 11 04:11:00 dbstore2001 mysqld[2693]: 2017-10-11 4:11:00 140145493571840 [Note] InnoDB: Processed 64328 .ibd/.isl files Oct 11 04:11:16 dbstore2001 mysqld[2693]: 2017-10-11 4:11:16 140145493571840 [Note] InnoDB: Processed 64955 .ibd/.isl files Oct 11 04:11:32 dbstore2001 mysqld[2693]: 2017-10-11 4:11:32 140145493571840 [Note] InnoDB: Processed 65637 .ibd/.isl files Oct 11 04:11:48 dbstore2001 mysqld[2693]: 2017-10-11 4:11:48 140145493571840 [Note] InnoDB: Processed 66198 .ibd/.isl files Oct 11 04:12:04 dbstore2001 mysqld[2693]: 2017-10-11 4:12:04 140145493571840 [Note] InnoDB: Processed 66825 .ibd/.isl files Oct 11 04:12:20 dbstore2001 mysqld[2693]: 2017-10-11 4:12:20 140145493571840 [Note] InnoDB: Processed 67452 .ibd/.isl files Oct 11 04:12:36 dbstore2001 mysqld[2693]: 2017-10-11 4:12:36 140145493571840 [Note] InnoDB: Processed 68145 .ibd/.isl files Oct 11 04:12:52 dbstore2001 mysqld[2693]: 2017-10-11 4:12:52 140145493571840 [Note] InnoDB: Processed 68761 .ibd/.isl files Oct 11 04:13:08 dbstore2001 mysqld[2693]: 2017-10-11 4:13:08 140145493571840 [Note] InnoDB: Processed 69300 .ibd/.isl files Oct 11 04:13:24 dbstore2001 mysqld[2693]: 2017-10-11 4:13:24 140145493571840 [Note] InnoDB: Processed 69883 .ibd/.isl files Oct 11 04:13:40 dbstore2001 mysqld[2693]: 2017-10-11 4:13:40 140145493571840 [Note] InnoDB: Processed 70422 .ibd/.isl files Oct 11 04:13:56 dbstore2001 mysqld[2693]: 2017-10-11 4:13:56 140145493571840 [Note] InnoDB: Processed 70873 .ibd/.isl files Oct 11 04:14:12 dbstore2001 mysqld[2693]: 2017-10-11 4:14:12 140145493571840 [Note] InnoDB: Processed 71434 .ibd/.isl files Oct 11 04:14:28 dbstore2001 mysqld[2693]: 2017-10-11 4:14:28 140145493571840 [Note] InnoDB: Processed 71863 .ibd/.isl files Oct 11 04:14:44 dbstore2001 mysqld[2693]: 2017-10-11 4:14:44 140145493571840 [Note] InnoDB: Processed 72292 .ibd/.isl files Oct 11 04:15:00 dbstore2001 mysqld[2693]: 2017-10-11 4:15:00 140145493571840 [Note] InnoDB: Processed 72666 .ibd/.isl files Oct 11 04:15:16 dbstore2001 mysqld[2693]: 2017-10-11 4:15:16 140145493571840 [Note] InnoDB: Processed 73095 .ibd/.isl files Oct 11 04:15:32 dbstore2001 mysqld[2693]: 2017-10-11 4:15:32 140145493571840 [Note] InnoDB: Processed 73535 .ibd/.isl files Oct 11 04:15:48 dbstore2001 mysqld[2693]: 2017-10-11 4:15:48 140145493571840 [Note] InnoDB: Processed 74030 .ibd/.isl files Oct 11 04:16:04 dbstore2001 mysqld[2693]: 2017-10-11 4:16:04 140145493571840 [Note] InnoDB: Processed 74547 .ibd/.isl files Oct 11 04:16:21 dbstore2001 mysqld[2693]: 2017-10-11 4:16:21 140145493571840 [Note] InnoDB: Processed 74976 .ibd/.isl files Oct 11 04:16:37 dbstore2001 mysqld[2693]: 2017-10-11 4:16:37 140145493571840 [Note] InnoDB: Processed 75306 .ibd/.isl files Oct 11 04:16:53 dbstore2001 mysqld[2693]: 2017-10-11 4:16:53 140145493571840 [Note] InnoDB: Processed 75625 .ibd/.isl files Oct 11 04:17:09 dbstore2001 mysqld[2693]: 2017-10-11 4:17:09 140145493571840 [Note] InnoDB: Processed 76054 .ibd/.isl files Oct 11 04:17:25 dbstore2001 mysqld[2693]: 2017-10-11 4:17:25 140145493571840 [Note] InnoDB: Processed 76560 .ibd/.isl files Oct 11 04:17:41 dbstore2001 mysqld[2693]: 2017-10-11 4:17:41 140145493571840 [Note] InnoDB: Processed 77099 .ibd/.isl files Oct 11 04:17:57 dbstore2001 mysqld[2693]: 2017-10-11 4:17:57 140145493571840 [Note] InnoDB: Processed 77638 .ibd/.isl files Oct 11 04:18:13 dbstore2001 mysqld[2693]: 2017-10-11 4:18:13 140145493571840 [Note] InnoDB: Processed 78034 .ibd/.isl files Oct 11 04:18:29 dbstore2001 mysqld[2693]: 2017-10-11 4:18:29 140145493571840 [Note] InnoDB: Processed 78441 .ibd/.isl files Oct 11 04:18:45 dbstore2001 mysqld[2693]: 2017-10-11 4:18:45 140145493571840 [Note] InnoDB: Processed 78826 .ibd/.isl files Oct 11 04:19:01 dbstore2001 mysqld[2693]: 2017-10-11 4:19:01 140145493571840 [Note] InnoDB: Processed 79189 .ibd/.isl files Oct 11 04:19:17 dbstore2001 mysqld[2693]: 2017-10-11 4:19:17 140145493571840 [Note] InnoDB: Processed 79651 .ibd/.isl files Oct 11 04:19:33 dbstore2001 mysqld[2693]: 2017-10-11 4:19:33 140145493571840 [Note] InnoDB: Processed 80146 .ibd/.isl files Oct 11 04:19:49 dbstore2001 mysqld[2693]: 2017-10-11 4:19:49 140145493571840 [Note] InnoDB: Processed 80762 .ibd/.isl files Oct 11 04:20:05 dbstore2001 mysqld[2693]: 2017-10-11 4:20:05 140145493571840 [Note] InnoDB: Processed 81246 .ibd/.isl files Oct 11 04:20:21 dbstore2001 mysqld[2693]: 2017-10-11 4:20:21 140145493571840 [Note] InnoDB: Processed 81587 .ibd/.isl files Oct 11 04:20:37 dbstore2001 mysqld[2693]: 2017-10-11 4:20:37 140145493571840 [Note] InnoDB: Processed 81906 .ibd/.isl files Oct 11 04:20:53 dbstore2001 mysqld[2693]: 2017-10-11 4:20:53 140145493571840 [Note] InnoDB: Processed 82368 .ibd/.isl files Oct 11 04:21:09 dbstore2001 mysqld[2693]: 2017-10-11 4:21:09 140145493571840 [Note] InnoDB: Processed 82830 .ibd/.isl files Oct 11 04:21:25 dbstore2001 mysqld[2693]: 2017-10-11 4:21:25 140145493571840 [Note] InnoDB: Processed 83424 .ibd/.isl files Oct 11 04:21:41 dbstore2001 mysqld[2693]: 2017-10-11 4:21:41 140145493571840 [Note] InnoDB: Processed 83842 .ibd/.isl files Oct 11 04:21:57 dbstore2001 mysqld[2693]: 2017-10-11 4:21:57 140145493571840 [Note] InnoDB: Processed 84238 .ibd/.isl files Oct 11 04:22:13 dbstore2001 mysqld[2693]: 2017-10-11 4:22:13 140145493571840 [Note] InnoDB: Processed 84667 .ibd/.isl files Oct 11 04:22:29 dbstore2001 mysqld[2693]: 2017-10-11 4:22:29 140145493571840 [Note] InnoDB: Processed 85085 .ibd/.isl files Oct 11 04:22:45 dbstore2001 mysqld[2693]: 2017-10-11 4:22:45 140145493571840 [Note] InnoDB: Processed 85503 .ibd/.isl files Oct 11 04:23:01 dbstore2001 mysqld[2693]: 2017-10-11 4:23:01 140145493571840 [Note] InnoDB: Processed 86020 .ibd/.isl files Oct 11 04:23:15 dbstore2001 mysqld[2693]: 2017-10-11 4:23:15 140145493571840 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer... Oct 11 04:23:22 dbstore2001 mysqld[2693]: 2017-10-11 4:23:22 140145493571840 [Note] InnoDB: Read redo log up to LSN=17828766823936 Oct 11 04:23:31 dbstore2001 mysqld[2693]: 2017-10-11 4:23:31 140145493571840 [Note] InnoDB: Starting final batch to recover 8342 pages from redo log Oct 11 04:23:37 dbstore2001 mysqld[2693]: 2017-10-11 4:23:37 140127791720192 [Note] InnoDB: To recover: 8047 pages from log Oct 11 04:23:52 dbstore2001 mysqld[2693]: 2017-10-11 4:23:52 140127808505600 [Note] InnoDB: To recover: 5651 pages from log Oct 11 04:24:07 dbstore2001 mysqld[2693]: 2017-10-11 4:24:07 140127816898304 [Note] InnoDB: To recover: 1116 pages from log Oct 11 04:24:12 dbstore2001 mysqld[2693]: InnoDB: In a MySQL replication slave the last master binlog file Oct 11 04:24:12 dbstore2001 mysqld[2693]: InnoDB: position 0 1022925931, file name db1038-bin.000776 Oct 11 04:24:12 dbstore2001 mysqld[2693]: InnoDB: Last MySQL binlog file position 0 1020571881, file name ./db2074-bin.000029 Oct 11 05:14:44 dbstore2001 mysqld[2693]: 2017-10-11 5:14:44 140145493571840 [Note] InnoDB: 128 rollback segment(s) are active. Oct 11 05:14:44 dbstore2001 mysqld[2693]: 2017-10-11 5:14:44 140145493571840 [Note] InnoDB: Waiting for purge to start Oct 11 05:14:44 dbstore2001 mysqld[2693]: 2017-10-11 5:14:44 140145493571840 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 17828780825567 Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 5:14:46 140127158372096 [Note] InnoDB: Dumping buffer pool(s) not yet started Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 05:14:46 7f71e57f8700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 5:14:46 140145493571840 [Note] Plugin 'FEEDBACK' is disabled. Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 5:14:46 140145493571840 [Note] Recovering after a crash using tc.log Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 5:14:46 140145493571840 [Note] Starting crash recovery... Oct 11 05:14:46 dbstore2001 mysqld[2693]: 2017-10-11 5:14:46 140145493571840 [Note] Crash recovery finished. Oct 11 05:14:47 dbstore2001 mysqld[2693]: 2017-10-11 5:14:47 140145493571840 [Note] Server socket created on IP: '::'. Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493571840 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493571840 [Warning] Checking table: './mysql/event' Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493571840 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493011200 [Note] Event Scheduler: scheduler thread started with id 1 Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493571840 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=dbstore2001-relay-bin' to avoid this problem. Oct 11 05:14:48 dbstore2001 mysqld[2693]: 2017-10-11 5:14:48 140145493571840 [Note] /opt/wmf-mariadb101//bin/mysqld: ready for connections. Oct 11 05:14:48 dbstore2001 mysqld[2693]: Version: '10.1.26-MariaDB' socket: '/run/mysqld/mysqld.s3.sock' port: 3313 MariaDB Server Oct 11 05:14:48 dbstore2001 systemd[1]: Started MariaDB database server.
Description
Description
Event Timeline
Comment Actions
Note mysql and replication is running- we could just drop and reload the "asserted" table.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-10-11T10:24:55Z] <jynus> stopping dbstore2001:s3 for maintenance T177908
Comment Actions
Given the crash happened after the backup finished, it used GTID and there is anoteher copy of s3 on dbstore2002, I have just dropped and reloaded the table while replication was stopped. I do not think we had data loss, although it is possible that at the end of the backup, UNDO cleared and triggered some issue with indexes on that table. Let's close and monitor next week if it happens again.
In a way, this confirms multi-instance was the right decision- only s3 crashed, and gtid worked as it should. We may want to load from a logical backup the future dbstore1001 to avoid format issues?