Page MenuHomePhabricator

db2034 crashed
Closed, ResolvedPublic

Description

150805 11:32:49 [ERROR] Slave I/O: error reconnecting to master 'repl@db2016.codfw.wmnet:3306' - retry-time: 60  retries: 86400  message: Can't connect to MySQL server on 'db2016.codfw.wmnet' (110 "Connection timed out"), Internal MariaDB error code: 2003
150805 13:19:35 [ERROR] Event Scheduler: [root@208.80.154.151][ops.wmf_slave_wikiuser_sleep] Unknown thread id: 170052306
150805 13:19:35 [Note] Event Scheduler: [root@208.80.154.151].[ops.wmf_slave_wikiuser_sleep] event execution failed.
150805 13:19:42 [ERROR] Slave I/O: error reconnecting to master 'repl@db2016.codfw.wmnet:3306' - retry-time: 60  retries: 86400  message: Unknown MySQL server host 'db2016.codfw.wmnet' (-2), Internal MariaDB error code: 2005
2015-08-17 12:47:25 7f91e13e6700  InnoDB: Assertion failure in thread 140264525948672 in file os0file.cc line 5105
InnoDB: Failing assertion: slot != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
150817 12:47:25 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.0.16-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=536
max_threads=510
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1344554 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
mysys/stacktrace.c:247(my_print_stacktrace)[0xaeb5bb]
sql/signal_handler.cc:153(handle_fatal_signal)[0x70f478]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fb1bdf17340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fb1bc2bfcc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fb1bc2c30d8]
os/os0file.cc:5106(os_aio_linux_collect)[0x983af5]
fil/fil0fil.cc:5690(fil_aio_wait(unsigned long))[0xa823f7]
srv/srv0start.cc:512(io_handler_thread)[0x9e69f5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fb1bdf0f182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fb1bc38400d]

Event Timeline

jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added projects: DBA, acl*sre-team.
jcrespo subscribed.

mysql_upgrade hadn't been run on this server and it complains about mysql tables- may be or not related.

jcrespo claimed this task.