Looks like dbstore2001 has crashed - I am not sure what's going on with this host as it didn't page. But it doesn't look healthy by looking at the stack trace:
2016-09-21 07:01:48 7f94694cb700 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 120:3, should be 161230:3! 2016-09-21 07:01:48 7f94694cb700 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 120:1, should be 161230:1! 2016-09-21 07:01:48 7f94694cb700 InnoDB: Assertion failure in thread 140275398522624 in file btr0btr.cc line 713 InnoDB: Failing assertion: mach_read_from_4(seg_header + FSEG_HDR_SPACE) == space InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 160921 7:01:48 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.22-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=15 max_threads=252 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 684383 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f1373f29008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f94694cae40 thread_stack 0x48000 (my_addr_resolve failure: fork) 160921 7:01:49 [Warning] Master 'x1': Slave SQL: Could not execute Update_rows_v1 event on table heartbeat.heartbeat; Can't find record in 'heartbeat', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db2033-bin.000190, end_log_pos 101199546, Gtid 0-171970580-439302824, Internal MariaDB error code: 1032 /opt/wmf-mariadb10/bin/mysqld(my_print_stacktrace+0x2e) [0xbdd6ee] /opt/wmf-mariadb10/bin/mysqld(handle_fatal_signal+0x390) [0x73dc40] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f9d4805a340] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f9d46e73cc9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f9d46e770d8] /opt/wmf-mariadb10/bin/mysqld() [0x9bf935] /opt/wmf-mariadb10/bin/mysqld() [0x9c30de] /opt/wmf-mariadb10/bin/mysqld() [0xa0e872] /opt/wmf-mariadb10/bin/mysqld() [0x97e69a] /opt/wmf-mariadb10/bin/mysqld() [0x97f274] /opt/wmf-mariadb10/bin/mysqld() [0x93d190] /opt/wmf-mariadb10/bin/mysqld() [0x93d792] /opt/wmf-mariadb10/bin/mysqld() [0x962f2c] /opt/wmf-mariadb10/bin/mysqld() [0x8c1961] /opt/wmf-mariadb10/bin/mysqld(ha_delete_table(THD*, handlerton*, char const*, char const*, char const*, bool)+0x22f) [0x741bff] /opt/wmf-mariadb10/bin/mysqld(mysql_rm_table_no_locks(THD*, TABLE_LIST*, bool, bool, bool, bool, bool)+0x899) [0x654f39] /opt/wmf-mariadb10/bin/mysqld(mysql_rm_table(THD*, TABLE_LIST*, char, char)+0x164) [0x655a44] /opt/wmf-mariadb10/bin/mysqld(mysql_execute_command(THD*)+0x2a0b) [0x5d4ddb] /opt/wmf-mariadb10/bin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1ca) [0x5d9a5a] /opt/wmf-mariadb10/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1864) [0x5db874] /opt/wmf-mariadb10/bin/mysqld(do_handle_one_connection(THD*)+0x21b) [0x6a585b] /opt/wmf-mariadb10/bin/mysqld(handle_one_connection+0x39) [0x6a5919] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f9d48052182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9d46f3747d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7e17c435f020): is an invalid pointer Connection ID (thread ID): 20049338 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=on,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 160921 07:02:07 mysqld_safe Number of processes running now: 0 160921 07:02:07 mysqld_safe mysqld restarted 160921 7:02:07 [Note] /opt/wmf-mariadb10/bin/mysqld (mysqld 10.0.22-MariaDB) starting as process 16225 ... 2016-09-21 07:02:07 7f930ee1e7c0 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html. 160921 7:02:07 [Note] InnoDB: Using mutexes to ref count buffer pool pages 160921 7:02:07 [Note] InnoDB: The InnoDB memory heap is disabled 160921 7:02:07 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 160921 7:02:07 [Note] InnoDB: Memory barrier is not used 160921 7:02:07 [Note] InnoDB: Compressed tables use zlib 1.2.8 160921 7:02:07 [Note] InnoDB: Using CPU crc32 instructions 160921 7:02:07 [Note] InnoDB: Initializing buffer pool, size = 24.0G 160921 7:02:08 [Note] InnoDB: Completed initialization of buffer pool 160921 7:02:09 [Note] InnoDB: Highest supported file format is Barracuda. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/email_capture'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."email_capture"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/povwatch_log'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."povwatch_log"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/povwatch_subscribers'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."povwatch_subscribers"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'test/eltx'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"test"."eltx"', ignored. 160921 7:02:10 [Warning] InnoDB: Resizing redo log from 2*131072 to 2*262144 pages, LSN=16181284385708 160921 7:02:10 [Warning] InnoDB: Starting to delete and rewrite log files. 160921 7:02:10 [Note] InnoDB: Setting log file ./ib_logfile101 size to 4096 MB InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/email_capture'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."email_capture"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/povwatch_log'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."povwatch_log"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'enwiki/povwatch_subscribers'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"enwiki"."povwatch_subscribers"', ignored. 2016-09-21 07:02:10 7f930ee1e7c0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 160921 7:02:10 [ERROR] InnoDB: Could not find a valid tablespace file for 'test/eltx'. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting-datadict.html for how to resolve the issue. 160921 7:02:10 [ERROR] InnoDB: Tablespace open failed for '"test"."eltx"', ignored. 160921 7:02:10 [Warning] InnoDB: Resizing redo log from 2*131072 to 2*262144 pages, LSN=16181284385708 160921 7:02:10 [Warning] InnoDB: Starting to delete and rewrite log files. 160921 7:02:10 [Note] InnoDB: Setting log file ./ib_logfile101 size to 4096 MB InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 160921 7:02:17 [Note] InnoDB: Setting log file ./ib_logfile1 size to 4096 MB InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 160921 7:02:24 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 160921 7:02:24 [Warning] InnoDB: New log files created, LSN=16181284385804 160921 7:02:24 [Note] InnoDB: 128 rollback segment(s) are active. 160921 7:02:24 [Note] InnoDB: Waiting for purge to start 160921 7:02:24 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 16181284385708 2016-09-21 07:02:24 7f8c68ff8700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool 160921 7:02:25 [Note] Plugin 'FEEDBACK' is disabled. Wed Sep 21 07:02:25 2016 TokuFT recovery starting in env /srv/sqldata/ Wed Sep 21 07:02:25 2016 TokuFT recovery scanning backward from 81524267354 Wed Sep 21 07:02:25 2016 TokuFT recovery bw_begin_checkpoint at 81524218206 timestamp 1474441285445442 (bw_newer) Wed Sep 21 07:02:25 2016 TokuFT recovery bw_end_checkpoint at 81524163046 timestamp 1474441255120929 xid 81524128379 (bw_newer) Wed Sep 21 07:02:25 2016 TokuFT recovery bw_begin_checkpoint at 81524128379 timestamp 1474441225445396 (bw_between) Wed Sep 21 07:02:25 2016 TokuFT recovery turning around at begin checkpoint 81524128379 time 29675533 Wed Sep 21 07:02:25 2016 TokuFT recovery starts scanning forward to 81524267354 from 81524128379 left 138975 (fw_between) Wed Sep 21 07:02:40 2016 TokuFT recovery scanning forward to 81524267354 at 81524135379 left 131975 (fw_between) Wed Sep 21 07:02:49 2016 TokuFT recovery has 1 live transaction recover_abort_live_txn 12341405105 recover_abort_live_txn 12341405105 Wed Sep 21 07:02:49 2016 TokuFT recovery closing 899 dictionaries Wed Sep 21 07:03:19 2016 TokuFT recovery making a checkpoint Wed Sep 21 07:03:19 2016 TokuFT recovery done 160921 7:03:19 [Note] Recovering after a crash using tc.log 160921 7:03:19 [Note] Starting crash recovery... 160921 7:03:19 [Note] Crash recovery finished. 160921 7:03:19 [Note] Server socket created on IP: '::'. 160921 7:03:19 [Note] Event Scheduler: scheduler thread started with id 1 160921 7:03:19 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=dbstore2001-relay-bin' to avoid this problem. 160921 7:03:19 [ERROR] Failed to open the relay log './db2034-relay-bin.000775' (relay_log_pos 907274003) 160921 7:03:19 [ERROR] Could not find target log during relay log initialization 160921 7:03:19 [ERROR] Failed to initialize the master info structure 160921 7:03:19 [ERROR] Failed to allocate memory for the Master Info structure 160921 7:03:19 [Note] /opt/wmf-mariadb10/bin/mysqld: ready for connections. Version: '10.0.22-MariaDB' socket: '/tmp/mysql.sock' port: 3306 MariaDB Server 160921 7:03:51 [ERROR] mysqld: Table './heartbeat/heartbeat' is marked as crashed and should be repaired 160921 7:03:51 [Warning] Checking table: './heartbeat/heartbeat'