labsdb1009 crashed last night due to OOM (https://grafana.wikimedia.org/dashboard/db/server-board?refresh=1m&panelId=14&fullscreen&orgId=1&var-server=labsdb1009&var-network=eth0&from=1509213927891&to=1509225331129)
Oct 28 20:45:25 labsdb1009 mysqld[24005]: 2017-10-28 20:45:25 140063807833856 [ERROR] mysqld: Out of memory (Needed 28896192 bytes) Oct 28 20:45:25 labsdb1009 mysqld[24005]: 2017-10-28 20:45:25 140063807833856 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space Oct 28 20:45:25 labsdb1009 mysqld[24005]: 2017-10-28 20:45:25 140063807833856 [ERROR] mysqld: Out of memory (Needed 28896192 bytes) Oct 28 20:45:25 labsdb1009 mysqld[24005]: 2017-10-28 20:45:25 140063807833856 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space Oct 28 20:45:25 labsdb1009 mysqld[24005]: 171028 20:45:25 [ERROR] mysqld got signal 11 ; Oct 28 20:45:25 labsdb1009 mysqld[24005]: This could be because you hit a bug. It is also possible that this binary Oct 28 20:45:25 labsdb1009 mysqld[24005]: or one of the libraries it was linked against is corrupt, improperly built, Oct 28 20:45:25 labsdb1009 mysqld[24005]: or misconfigured. This error can also be caused by malfunctioning hardware. Oct 28 20:45:25 labsdb1009 mysqld[24005]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Oct 28 20:45:25 labsdb1009 mysqld[24005]: We will try our best to scrape up some info that will hopefully help Oct 28 20:45:25 labsdb1009 mysqld[24005]: diagnose the problem, but since we have already crashed, Oct 28 20:45:25 labsdb1009 mysqld[24005]: something is definitely wrong and this may fail. Oct 28 20:45:25 labsdb1009 mysqld[24005]: Server version: 10.1.28-MariaDB Oct 28 20:45:25 labsdb1009 mysqld[24005]: key_buffer_size=134217728 Oct 28 20:45:25 labsdb1009 mysqld[24005]: read_buffer_size=131072 Oct 28 20:45:25 labsdb1009 mysqld[24005]: max_used_connections=27 Oct 28 20:45:25 labsdb1009 mysqld[24005]: max_threads=1026 Oct 28 20:45:25 labsdb1009 mysqld[24005]: thread_count=9 Oct 28 20:45:25 labsdb1009 mysqld[24005]: It is possible that mysqld could use up to Oct 28 20:45:25 labsdb1009 mysqld[24005]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2384788 K bytes of memory Oct 28 20:45:25 labsdb1009 mysqld[24005]: Hope that's ok; if not, decrease some variables in the equation. Oct 28 20:45:25 labsdb1009 mysqld[24005]: Thread pointer: 0x7efcf1e9d008 Oct 28 20:45:25 labsdb1009 mysqld[24005]: Attempting backtrace. You can use the following information to find out Oct 28 20:45:25 labsdb1009 mysqld[24005]: where mysqld died. If you see no messages after this, something went Oct 28 20:45:25 labsdb1009 mysqld[24005]: terribly wrong... Oct 28 20:45:25 labsdb1009 mysqld[24005]: stack_bottom = 0x7f632582fc88 thread_stack 0x30000 Oct 28 20:45:25 labsdb1009 mysqld[24005]: (my_addr_resolve failure: fork) Oct 28 20:45:25 labsdb1009 mysqld[24005]: /opt/wmf-mariadb101//bin/mysqld(my_print_stacktrace+0x2e) [0x56441c724e3e] Oct 28 20:45:25 labsdb1009 mysqld[24005]: /opt/wmf-mariadb101//bin/mysqld(handle_fatal_signal+0x3bd) [0x56441c2699ad] Oct 28 20:45:25 labsdb1009 mysqld[24005]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f63285a00c0] Oct 28 20:45:25 labsdb1009 mysqld[24005]: /opt/wmf-mariadb101//bin/mysqld(st_join_table::save_explain_data(Explain_table_access*, unsigned long long, bool, st_join_table*)+0x6d) [0x56441c1278dd] Oct 28 20:45:25 labsdb1009 mysqld[24005]: /opt/wmf-mariadb101//bin/mysqld(st_join_table::update_explain_data(unsigned int)+0x37b) [0x56441c128c9b] Oct 28 20:45:25 labsdb1009 mysqld[24005]: /opt/wmf-mariadb101//bin/mysqld(+0x462ea9) [0x56441c128ea9] Oct 28 20:46:04 labsdb1009 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV Oct 28 20:46:04 labsdb1009 systemd[1]: mariadb.service: Unit entered failed state. Oct 28 20:46:04 labsdb1009 systemd[1]: mariadb.service: Failed with result 'signal'. Oct 28 20:46:09 labsdb1009 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart. Oct 28 20:46:09 labsdb1009 systemd[1]: Stopped MariaDB database server. Oct 28 20:46:09 labsdb1009 systemd[1]: Starting MariaDB database server... Oct 28 20:46:09 labsdb1009 mysqld[31730]: 2017-10-28 20:46:09 139666438564096 [Note] /opt/wmf-mariadb101//bin/mysqld (mysqld 10.1.28-MariaDB) starting as process 31730 ... Oct 28 20:46:09 labsdb1009 mysqld[31730]: 2017-10-28 20:46:09 139666438564096 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used; This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=labsdb1009' or '--log-bin=labsdb1009-bin' to avoid this problem. Oct 28 20:46:09 labsdb1009 mysqld[31730]: 2017-10-28 20:46:09 139666438564096 [ERROR] Plugin 'unix_socket' already installed
The table wikidatawiki.wb_changes got corrupted and broke replication.