Page MenuHomePhabricator
Paste P7558

Masterwork From Distant Lands
ActivePublic

Authored by ProdPasteBot on Sep 17 2018, 7:45 PM.
Aug 24 09:10:59 dbstore2002 mysqld[3550]: 2018-08-24 9:10:59 140612790758144 [ERROR] Slave I/O: error reconnecting to master 'repl@db2035.codfw.wmnet:3306' - retry-time: 60 maximum-retries: 86400 message: Can't connect to MySQL server on 'db2035.codfw.wmnet' (111 "Connection refused"), Internal MariaDB error code: 2003
Aug 24 09:12:59 dbstore2002 mysqld[3550]: 2018-08-24 9:12:59 140612790758144 [Note] Slave: connected to master 'repl@db2035.codfw.wmnet:3306',replication resumed in log 'db2035-bin.004846' at position 924796334
Sep 17 17:23:58 dbstore2002 mysqld[3550]: InnoDB: tried to purge sec index entry not marked for deletion in
Sep 17 17:23:58 dbstore2002 mysqld[3550]: InnoDB: index "cl_sortkey" of table "nowiki"."categorylinks"
Sep 17 17:23:58 dbstore2002 mysqld[3550]: InnoDB: tuple DATA TUPLE: 4 fields;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 0: len 36; hex 417274696b6c65725f6d65645f66696c6d6c656e6b65725f6672615f57696b6964617461; asc Artikler_med_filmlenker_fra_Wikidata;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 1: len 1; hex 01; asc ;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 2: len 25; hex 2b454b4f314333454b044d4f393f3f37314f3143011801dc17; asc +EKO1C3EK MO9??71O1C ;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 3: len 4; hex 00058d34; asc 4;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: InnoDB: record PHYSICAL RECORD: n_fields 4; compact format; info bits 0
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 0: len 30; hex 417274696b6c65725f6d65645f66696c6d6c656e6b65725f6672615f5769; asc Artikler_med_filmlenker_fra_Wi; (total 36 bytes);
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 1: len 1; hex 01; asc ;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 2: len 25; hex 2b454b4f314333454b044d4f393f3f37314f3143011801dc17; asc +EKO1C3EK MO9??71O1C ;;
Sep 17 17:23:58 dbstore2002 mysqld[3550]: 3: len 4; hex 00058d34; asc 4;;
Sep 17 17:25:05 dbstore2002 mysqld[3550]: 2018-09-17 17:25:05 7fe2f76d0b00 InnoDB: Assertion failure in thread 140612790455040 in file row0ins.cc line 285
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: Failing assertion: *cursor->index->name == TEMP_INDEX_PREFIX
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: We intentionally generate a memory trap.
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: If you get repeated assertion failures or crashes, even
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: immediately after the mysqld startup, there may be
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
Sep 17 17:25:05 dbstore2002 mysqld[3550]: InnoDB: about forcing recovery.
Sep 17 17:25:05 dbstore2002 mysqld[3550]: 180917 17:25:05 [ERROR] mysqld got signal 6 ;
Sep 17 17:25:05 dbstore2002 mysqld[3550]: This could be because you hit a bug. It is also possible that this binary
Sep 17 17:25:05 dbstore2002 mysqld[3550]: or one of the libraries it was linked against is corrupt, improperly built,
Sep 17 17:25:05 dbstore2002 mysqld[3550]: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep 17 17:25:05 dbstore2002 mysqld[3550]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Sep 17 17:25:05 dbstore2002 mysqld[3550]: We will try our best to scrape up some info that will hopefully help
Sep 17 17:25:05 dbstore2002 mysqld[3550]: diagnose the problem, but since we have already crashed,
Sep 17 17:25:05 dbstore2002 mysqld[3550]: something is definitely wrong and this may fail.
Sep 17 17:25:05 dbstore2002 mysqld[3550]: Server version: 10.1.35-MariaDB
Sep 17 17:25:05 dbstore2002 mysqld[3550]: key_buffer_size=1048576
Sep 17 17:25:05 dbstore2002 mysqld[3550]: read_buffer_size=131072
Sep 17 17:25:05 dbstore2002 mysqld[3550]: max_used_connections=20
Sep 17 17:25:05 dbstore2002 mysqld[3550]: max_threads=252
Sep 17 17:25:05 dbstore2002 mysqld[3550]: thread_count=15
Sep 17 17:25:05 dbstore2002 mysqld[3550]: It is possible that mysqld could use up to
Sep 17 17:25:05 dbstore2002 mysqld[3550]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 554601 K bytes of memory
Sep 17 17:25:05 dbstore2002 mysqld[3550]: Hope that's ok; if not, decrease some variables in the equation.
Sep 17 17:25:05 dbstore2002 mysqld[3550]: Thread pointer: 0x7fdd7c42b008
Sep 17 17:25:05 dbstore2002 mysqld[3550]: Attempting backtrace. You can use the following information to find out
Sep 17 17:25:05 dbstore2002 mysqld[3550]: where mysqld died. If you see no messages after this, something went
Sep 17 17:25:05 dbstore2002 mysqld[3550]: terribly wrong...
Sep 17 17:25:05 dbstore2002 mysqld[3550]: stack_bottom = 0x7fe2f76cf838 thread_stack 0x48400
Sep 17 17:25:06 dbstore2002 mysqld[3550]: *** buffer overflow detected ***: /opt/wmf-mariadb101/bin/mysqld terminated
Sep 17 17:25:06 dbstore2002 mysqld[3550]: ======= Backtrace: =========
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7fe3051a7bfb]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fe3052301f7]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(+0xf7330)[0x7fe30522e330]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(+0xf916a)[0x7fe30523016a]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(my_addr_resolve+0xd8)[0x562689cabcd8]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(my_print_stacktrace+0x1bb)[0x562689c9502b]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(handle_fatal_signal+0x3bd)[0x5626897d62fd]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fe306a880c0]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fe305169fff]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fe30516b42a]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x8dbd90)[0x562689b01d90]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x8dde8b)[0x562689b03e8b]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x8de2b4)[0x562689b042b4]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x8ea4f7)[0x562689b104f7]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x83f615)[0x562689a65615]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x4cf)[0x5626897e0eff]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72)[0x562689622af2]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x1216)[0x56268962ce66]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3a52)[0x562689641bc2]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x311)[0x5626896473b1]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x131d)[0x5626898a2d5d]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(+0x39903b)[0x5626895bf03b]
Sep 17 17:25:06 dbstore2002 mysqld[3550]: /opt/wmf-mariadb101/bin/mysqld(handle_slave_sql+0x2bf3)[0x5626895ca2c3]
Sep 17 17:25:07 dbstore2002 systemd[1]: mariadb@s2.service: Main process exited, code=killed, status=6/ABRT
Sep 17 17:25:07 dbstore2002 systemd[1]: mariadb@s2.service: Unit entered failed state.
Sep 17 17:25:07 dbstore2002 systemd[1]: mariadb@s2.service: Failed with result 'signal'.
Sep 17 17:25:13 dbstore2002 systemd[1]: mariadb@s2.service: Service hold-off time over, scheduling restart.
Sep 17 17:25:13 dbstore2002 systemd[1]: Stopped mariadb database server.
Sep 17 17:25:13 dbstore2002 systemd[1]: Starting mariadb database server...
Sep 17 17:25:13 dbstore2002 mysqld[23213]: 2018-09-17 17:25:13 140666393803008 [Note] /opt/wmf-mariadb101/bin/mysqld (mysqld 10.1.35-MariaDB) starting as process 23213 ...
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [ERROR] Plugin 'unix_socket' already installed
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 7fef726f3900 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: The InnoDB memory heap is disabled
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Compressed tables use zlib 1.2.3
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Using Linux native AIO
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Using SSE crc32 instructions
Sep 17 17:25:14 dbstore2002 mysqld[23213]: 2018-09-17 17:25:14 140666393803008 [Note] InnoDB: Initializing buffer pool, size = 20.0G
Sep 17 17:25:15 dbstore2002 mysqld[23213]: 2018-09-17 17:25:15 140666393803008 [Note] InnoDB: Completed initialization of buffer pool
Sep 17 17:25:15 dbstore2002 mysqld[23213]: 2018-09-17 17:25:15 140666393803008 [Note] InnoDB: Highest supported file format is Barracuda.
Sep 17 17:25:15 dbstore2002 mysqld[23213]: 2018-09-17 17:25:15 140666393803008 [Note] InnoDB: Starting crash recovery from checkpoint LSN=34186413407073
Sep 17 17:25:31 dbstore2002 mysqld[23213]: 2018-09-17 17:25:31 140666393803008 [Note] InnoDB: Processed 1650 .ibd/.isl files
Sep 17 17:25:32 dbstore2002 mysqld[23213]: 2018-09-17 17:25:32 140666393803008 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
Sep 17 17:25:32 dbstore2002 mysqld[23213]: 2018-09-17 17:25:32 140666393803008 [Note] InnoDB: Read redo log up to LSN=34186413537792
Sep 17 17:25:47 dbstore2002 mysqld[23213]: 2018-09-17 17:25:47 140666393803008 [Note] InnoDB: To recover: 194027 pages from log
Sep 17 17:25:48 dbstore2002 mysqld[23213]: InnoDB: 1 transaction(s) which must be rolled back or cleaned up
Sep 17 17:25:48 dbstore2002 mysqld[23213]: InnoDB: in total 1 row operations to undo
Sep 17 17:25:48 dbstore2002 mysqld[23213]: InnoDB: Trx id counter is 43927584256
Sep 17 17:25:48 dbstore2002 mysqld[23213]: 2018-09-17 17:25:48 140666393803008 [Note] InnoDB: Starting final batch to recover 193656 pages from redo log
Sep 17 17:26:02 dbstore2002 mysqld[23213]: 2018-09-17 17:26:02 140642936145664 [Note] InnoDB: To recover: 160469 pages from log
Sep 17 17:26:17 dbstore2002 mysqld[23213]: 2018-09-17 17:26:17 140642927752960 [Note] InnoDB: To recover: 116522 pages from log
Sep 17 17:26:32 dbstore2002 mysqld[23213]: 2018-09-17 17:26:32 140642919360256 [Note] InnoDB: To recover: 71081 pages from log
Sep 17 17:26:47 dbstore2002 mysqld[23213]: 2018-09-17 17:26:47 140642927752960 [Note] InnoDB: To recover: 33458 pages from log
Sep 17 17:26:59 dbstore2002 mysqld[23213]: InnoDB: In a MySQL replication slave the last master binlog file
Sep 17 17:26:59 dbstore2002 mysqld[23213]: InnoDB: position 0 713880055, file name db1024-bin.000534
Sep 17 17:26:59 dbstore2002 mysqld[23213]: InnoDB: Last MySQL binlog file position 0 140307185, file name ./db2056-bin.002629
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140666393803008 [Note] InnoDB: 128 rollback segment(s) are active.
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140642516723456 [Note] InnoDB: Starting in background the rollback of recovered transactions
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140666393803008 [Note] InnoDB: Waiting for purge to start
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140642516723456 [Note] InnoDB: To roll back: 1 transactions, 1 rows
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140666393803008 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.39-83.1 started; log sequence number 34188143428820
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140642516723456 [Note] InnoDB: Rollback of trx with id 43927583911 completed
Sep 17 17:27:24 dbstore2002 mysqld[23213]: 2018-09-17 17:27:24 140642516723456 [Note] InnoDB: Rollback of non-prepared transactions completed
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140642516723456 [Note] InnoDB: Dumping buffer pool(s) not yet started
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 7fe9e33fe700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Note] Plugin 'FEEDBACK' is disabled.
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Note] Recovering after a crash using tc.log
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Note] Starting crash recovery...
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Note] Crash recovery finished.
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Note] Server socket created on IP: '::'.
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [Warning] Checking table: './mysql/user'
Sep 17 17:28:52 dbstore2002 mysqld[23213]: 2018-09-17 17:28:52 140666393803008 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [Warning] Checking table: './mysql/db'
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [Warning] Checking table: './mysql/event'
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393242368 [Note] Event Scheduler: scheduler thread started with id 1
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--log-basename=#' or '--relay-log=dbstore2002-relay-bin' to avoid this problem.
Sep 17 17:28:53 dbstore2002 mysqld[23213]: 2018-09-17 17:28:53 140666393803008 [Note] /opt/wmf-mariadb101/bin/mysqld: ready for connections.
Sep 17 17:28:53 dbstore2002 mysqld[23213]: Version: '10.1.35-MariaDB' socket: '/run/mysqld/mysqld.s2.sock' port: 3312 MariaDB Server
Sep 17 17:28:53 dbstore2002 systemd[1]: Started mariadb database server.
Sep 17 17:31:32 dbstore2002 mysqld[23213]: 2018-09-17 17:31:32 7fe9e33fe700 InnoDB: Buffer pool(s) load completed at 180917 17:31:32

Event Timeline

ProdPasteBot edited the content of this paste. (Show Details)Sep 17 2018, 7:45 PM
ProdPasteBot changed the title of this paste from untitled to Masterwork From Distant Lands.