Page MenuHomePhabricator

db2201 memory failure (was: both mysql instances crashed in the last few days)
Closed, ResolvedPublic

Description

x1
Aug 08 17:11:30 db2201 mysqld[6648]: 260808 17:11:30 [ERROR] /opt/wmf-mariadb1011/bin/mysqld got signal 7 ;
Aug 08 17:11:30 db2201 mysqld[6648]: Sorry, we probably made a mistake, and this is a bug.
Aug 08 17:11:30 db2201 mysqld[6648]: Your assistance in bug reporting will enable us to fix this for the next release.
Aug 08 17:11:30 db2201 mysqld[6648]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Aug 08 17:11:30 db2201 mysqld[6648]: a bug on https://jira.mariadb.org/.
Aug 08 17:11:30 db2201 mysqld[6648]: Please include the information from the server start above, to the end of the
Aug 08 17:11:30 db2201 mysqld[6648]: information below.
Aug 08 17:11:30 db2201 mysqld[6648]: Server version: 10.11.18-MariaDB-log source revision: 197f92bee02d8e836f529f37625be69b83e7acbd
Aug 08 17:11:30 db2201 mysqld[6648]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Aug 08 17:11:30 db2201 mysqld[6648]: contains instructions to obtain a better version of the backtrace below.
Aug 08 17:11:30 db2201 mysqld[6648]: Following these instructions will help MariaDB developers provide a fix quicker.
Aug 08 17:11:30 db2201 mysqld[6648]: Attempting backtrace. Include this in the bug report.
Aug 08 17:11:30 db2201 mysqld[6648]: (note: Retrieving this information may fail)
Aug 08 17:11:30 db2201 mysqld[6648]: Thread pointer: 0x0
Aug 08 17:11:30 db2201 mysqld[6648]: stack_bottom = 0x0 thread_stack 0x49000
Aug 08 17:11:36 db2201 mysqld[6648]: mysys/stacktrace.c:216(my_print_stacktrace)[0x55d3c9a1d8de]
Aug 08 17:11:36 db2201 mysqld[6648]: sql/signal_handler.cc:230(handle_fatal_signal)[0x55d3c956acf9]
Aug 08 17:11:40 db2201 mysqld[6648]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f590465a050]
Aug 08 17:11:42 db2201 mysqld[6648]: 2026-08-08 17:11:42 0 [Note] /opt/wmf-mariadb1011/bin/mysqld (initiated by: unknown): Normal shutdown
Aug 08 17:11:42 db2201 mysqld[6648]: 2026-08-08 17:11:42 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 2
Aug 08 17:11:42 db2201 mysqld[6648]: 2026-08-08 17:11:42 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
Aug 08 17:11:42 db2201 mysqld[6648]: 2026-08-08 17:11:42 0 [Note] Event Scheduler: Stopped
Aug 08 17:11:44 db2201 mysqld[6648]: bits/string_fortified.h:59(buf_LRU_block_remove_hashed(buf_page_t*, page_id_t, buf_pool_t::hash_chain&, bool) [clo>
Aug 08 17:11:44 db2201 mysqld[6648]: buf/buf0lru.cc:817(buf_LRU_free_page(buf_page_t*, bool))[0x55d3c9953c88]
Aug 08 17:11:44 db2201 mysqld[6648]: buf/buf0flu.cc:1288(buf_do_LRU_batch(unsigned long, flush_counters_t*))[0x55d3c994fedf]
Aug 08 17:11:44 db2201 mysqld[6648]: buf/buf0flu.cc:1759(buf_flush_LRU(unsigned long))[0x55d3c995061e]
Aug 08 17:11:44 db2201 mysqld[6648]: psi/mysql_thread.h:785(buf_flush_page_cleaner())[0x55d3c9950aed]
Aug 08 17:11:49 db2201 mysqld[6648]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f59048d44a3]
Aug 08 17:11:53 db2201 mysqld[6648]: /lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f59046a71f5]
Aug 08 17:11:53 db2201 mysqld[6648]: /lib/x86_64-linux-gnu/libc.so.6(+0x1098ec)[0x7f59047278ec]
Aug 08 17:11:53 db2201 mysqld[6648]: Writing a core file...
Aug 08 17:11:53 db2201 mysqld[6648]: Working directory at /srv/sqldata.x1
Aug 08 17:11:53 db2201 mysqld[6648]: Resource Limits (excludes unlimited resources):
Aug 08 17:11:53 db2201 mysqld[6648]: Limit                     Soft Limit           Hard Limit           Units
Aug 08 17:11:53 db2201 mysqld[6648]: Max stack size            8388608              unlimited            bytes
Aug 08 17:11:53 db2201 mysqld[6648]: Max core file size        0                    0                    bytes
Aug 08 17:11:53 db2201 mysqld[6648]: Max processes             2061488              2061488              processes
Aug 08 17:11:53 db2201 mysqld[6648]: Max open files            200001               200001               files
Aug 08 17:11:53 db2201 mysqld[6648]: Max locked memory         8388608              8388608              bytes
Aug 08 17:11:53 db2201 mysqld[6648]: Max pending signals       2061488              2061488              signals
Aug 08 17:11:53 db2201 mysqld[6648]: Max msgqueue size         819200               819200               bytes
Aug 08 17:11:53 db2201 mysqld[6648]: Max nice priority         0                    0
Aug 08 17:11:53 db2201 mysqld[6648]: Max realtime priority     0                    0
Aug 08 17:11:53 db2201 mysqld[6648]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
Aug 08 17:11:53 db2201 mysqld[6648]: Kernel version: Linux version 6.1.0-50-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14+deb12u1) 1>
Aug 08 17:12:02 db2201 systemd[1]: mariadb@x1.service: Main process exited, code=killed, status=7/BUS
Aug 08 17:12:02 db2201 systemd[1]: mariadb@x1.service: Failed with result 'signal'.
Aug 08 17:12:02 db2201 systemd[1]: mariadb@x1.service: Unit process 774021 (addr2line) remains running after unit stopped.
Aug 08 17:12:02 db2201 systemd[1]: mariadb@x1.service: Consumed 1d 4h 50min 31.204s CPU time.
Aug 08 17:12:07 db2201 systemd[1]: mariadb@x1.service: Scheduled restart job, restart counter is at 1.
Aug 08 17:12:07 db2201 systemd[1]: Stopped mariadb@x1.service - mariadb database server.
Aug 08 17:12:07 db2201 systemd[1]: mariadb@x1.service: Consumed 1d 4h 50min 31.205s CPU time.
Aug 08 17:12:07 db2201 systemd[1]: Starting mariadb@x1.service - mariadb database server...
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Warning] Could not increase number of max_open_files to more than 200001 (request: 800329)
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] Starting MariaDB 10.11.18-MariaDB-log source revision 197f92bee02d8e836f529f37625be>
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-in>
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] mysqld: Aria engine: starting recovery
Aug 08 17:12:07 db2201 mysqld[774079]: recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 1 0
Aug 08 17:12:07 db2201 mysqld[774079]:  (0.0 seconds);
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] mysqld: Aria engine: recovery done
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] InnoDB: Compressed tables use zlib 1.3.2
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] InnoDB: Number of transaction pools: 1
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] InnoDB: Using AVX512 instructions
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] InnoDB: Using Linux native AIO
Aug 08 17:12:07 db2201 mysqld[774079]: 2026-08-08 17:12:07 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=131072m
Aug 08 17:12:13 db2201 mysqld[774079]: 2026-08-08 17:12:13 0 [Note] InnoDB: Completed initialization of buffer pool
Aug 08 17:12:13 db2201 mysqld[774079]: 2026-08-08 17:12:13 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Aug 08 17:12:14 db2201 mysqld[774079]: 2026-08-08 17:12:14 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=25862922759935
Aug 08 17:12:16 db2201 mysqld[774079]: 2026-08-08 17:12:16 0 [Note] InnoDB: End of log at LSN=25863320348323
Aug 08 17:12:16 db2201 mysqld[774079]: 2026-08-08 17:12:16 0 [Note] InnoDB: To recover: 105480 pages
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: Last binlog file './db2201-bin.021658', position 583876103
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: 128 rollback segments are active.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full>
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: log sequence number 25863320348323; transaction id 22906564035
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Plugin 'FEEDBACK' is disabled.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] InnoDB: Loading buffer pool(s) from /srv/sqldata.x1/ib_buffer_pool
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Recovering after a crash using db2201-bin
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Starting table crash recovery...
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Crash table recovery finished.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Server socket created on IP: '0.0.0.0', port: '3320'.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Server socket created on IP: '::', port: '3320'.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Server socket created on IP: '0.0.0.0', port: '3340'.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] Server socket created on IP: '::', port: '3340'.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Warning] 'proxies_priv' entry '@% root@db1029' ignored in --skip-name-resolve mode.
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 2 [Note] Event Scheduler: scheduler thread started with id 2
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break wh>
Aug 08 17:12:18 db2201 mysqld[774079]: 2026-08-08 17:12:18 0 [Note] /opt/wmf-mariadb1011/bin/mysqld: ready for connections.
Aug 08 17:12:18 db2201 mysqld[774079]: Version: '10.11.18-MariaDB-log'  socket: '/run/mysqld/mysqld.x1.sock'  port: 3320  MariaDB Server
Aug 08 17:12:18 db2201 systemd[1]: Started mariadb@x1.service - mariadb database server.
Aug 08 17:12:31 db2201 mysqld[774079]: 2026-08-08 17:12:31 0 [Note] InnoDB: Buffer pool(s) load completed at 260808 17:12:31
s5
Aug 11 00:02:03 db2201 mysqld[6569]: 260811  0:02:03 [ERROR] /opt/wmf-mariadb1011/bin/mysqld got signal 7 ;
Aug 11 00:02:03 db2201 mysqld[6569]: Sorry, we probably made a mistake, and this is a bug.
Aug 11 00:02:03 db2201 mysqld[6569]: Your assistance in bug reporting will enable us to fix this for the next release.
Aug 11 00:02:03 db2201 mysqld[6569]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Aug 11 00:02:03 db2201 mysqld[6569]: a bug on https://jira.mariadb.org/.
Aug 11 00:02:03 db2201 mysqld[6569]: Please include the information from the server start above, to the end of the
Aug 11 00:02:03 db2201 mysqld[6569]: information below.
Aug 11 00:02:03 db2201 mysqld[6569]: Server version: 10.11.18-MariaDB-log source revision: 197f92bee02d8e836f529f37625be69b83e7acbd
Aug 11 00:02:03 db2201 mysqld[6569]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Aug 11 00:02:03 db2201 mysqld[6569]: contains instructions to obtain a better version of the backtrace below.
Aug 11 00:02:03 db2201 mysqld[6569]: Following these instructions will help MariaDB developers provide a fix quicker.
Aug 11 00:02:03 db2201 mysqld[6569]: Attempting backtrace. Include this in the bug report.
Aug 11 00:02:03 db2201 mysqld[6569]: (note: Retrieving this information may fail)
Aug 11 00:02:03 db2201 mysqld[6569]: Thread pointer: 0x778ae6b1c0c8
Aug 11 00:02:03 db2201 mysqld[6569]: stack_bottom = 0x7f8ca8fb3000 thread_stack 0x49000
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 0 [Note] /opt/wmf-mariadb1011/bin/mysqld (initiated by: unknown): Normal shutdown
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 2
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 0 [Note] Event Scheduler: Stopped
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 14 [Note] Error reading relay log event: slave SQL thread was killed
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 14 [Note] Slave SQL thread exiting, replication stopped in log 'db2192-bin.001933' at position>
Aug 11 00:02:08 db2201 mysqld[6569]: 2026-08-11  0:02:08 13 [Note] Slave I/O thread exiting, read up to log 'db2192-bin.001933', position 424401482; GT>
Aug 11 00:02:12 db2201 mysqld[6569]: /opt/wmf-mariadb1011/bin/mysqld(my_print_stacktrace+0x2e)[0x55c180e6f8de]
Aug 11 00:02:12 db2201 mysqld[6569]: mysys/stacktrace.c:216(my_print_stacktrace)[0x55c1809bcc67]
Aug 11 00:02:20 db2201 mysqld[6569]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f8cc385a050]
Aug 11 00:02:28 db2201 mysqld[6569]: zlib/inflate.c:1062(inflate)[0x55c180ecfefc]
Aug 11 00:02:28 db2201 mysqld[6569]: page/page0zip.cc:2764(page_zip_decompress(page_zip_des_t*, unsigned char*, unsigned long))[0x55c180cff1c6]
Aug 11 00:02:28 db2201 mysqld[6569]: buf/buf0buf.cc:2542(buf_zip_decompress(buf_block_t*, bool))[0x55c180d98b2d]
Aug 11 00:02:28 db2201 mysqld[6569]: bits/atomic_base.h:628(std::__atomic_base<unsigned long>::fetch_sub(unsigned long, std::memory_order))[0x55c180615>
Aug 11 00:02:28 db2201 mysqld[6569]: buf/buf0buf.cc:3158(buf_page_get_low(page_id_t, unsigned long, rw_lock_type_t, buf_block_t*, unsigned long, mtr_t*>
Aug 11 00:02:28 db2201 mysqld[6569]: btr/btr0btr.cc:233(btr_block_get(dict_index_t const&, unsigned int, rw_lock_type_t, bool, mtr_t*, dberr_t*, bool*)>
Aug 11 00:02:28 db2201 mysqld[6569]: btr/btr0pcur.cc:528(btr_pcur_move_to_next_page(btr_pcur_t*, mtr_t*))[0x55c180d903f2]
Aug 11 00:02:28 db2201 mysqld[6569]: row/row0sel.cc:5825(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long>
Aug 11 00:02:28 db2201 mysqld[6569]: handler/ha_innodb.cc:9272(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int))[0x55c180c8f6e6]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/handler.cc:3720(handler::ha_index_next(unsigned char*))[0x55c1809c4c3d]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/handler.cc:6892(handler::read_range_next())[0x55c1809ca38d]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/multi_range_read.cc:492(handler::multi_range_read_next(void**))[0x55c1808cf5cc]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/multi_range_read.cc:555(Mrr_simple_index_reader::get_next(void**))[0x55c1808cf6c0]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/multi_range_read.cc:1659(DsMrr_impl::dsmrr_next(void**))[0x55c1808d0b74]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/opt_range.cc:12997(QUICK_RANGE_SELECT::get_next())[0x55c18068b5e6]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/records.cc:403(rr_quick(READ_RECORD*))[0x55c1806b56c9]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/records.h:81(READ_RECORD::read_record())[0x55c1807cae01]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_select.cc:22600(JOIN::exec_inner())[0x55c1807f6f60]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_select.cc:4808(JOIN::exec())[0x55c1807f7210]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_select.cc:5287(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Ite>
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_select.cc:601(handle_select(THD*, LEX*, select_result*, unsigned long long))[0x55c1807f5dbf]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_parse.cc:6461(execute_sqlcom_select(THD*, TABLE_LIST*))[0x55c180771c81]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_parse.cc:4040(mysql_execute_command(THD*, bool))[0x55c18077ff3e]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_parse.cc:8238(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55c180781806]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_parse.cc:1926(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x55c180783f05]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_parse.cc:1434(do_command(THD*, bool))[0x55c1807858af]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_connect.cc:1475(do_handle_one_connection(CONNECT*, bool))[0x55c18089bbfd]
Aug 11 00:02:28 db2201 mysqld[6569]: sql/sql_connect.cc:1387(handle_one_connection)[0x55c18089bf5d]
Aug 11 00:02:28 db2201 mysqld[6569]: perfschema/pfs.cc:2204(pfs_spawn_thread)[0x55c180bf6467]
Aug 11 00:02:31 db2201 mysqld[6569]: 2026-08-11  0:02:31 0 [Warning] /opt/wmf-mariadb1011/bin/mysqld: Thread 2411419 (user : 'dump') did not exit
Aug 11 00:02:36 db2201 mysqld[6569]: /lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f8cc38a71f5]
Aug 11 00:02:36 db2201 mysqld[6569]: /lib/x86_64-linux-gnu/libc.so.6(+0x1098ec)[0x7f8cc39278ec]
Aug 11 00:02:36 db2201 mysqld[6569]: Connection ID (thread ID): 2411419
Aug 11 00:02:36 db2201 mysqld[6569]: Status: KILL_SERVER
Aug 11 00:02:36 db2201 mysqld[6569]: Query (0x778ae68a9600): SELECT /*!40001 SQL_NO_CACHE */ * FROM `cebwiki`.`templatelinks` WHERE (`tl_from` >= 38177>
Aug 11 00:02:36 db2201 mysqld[6569]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_>
Aug 11 00:02:36 db2201 mysqld[6569]: Writing a core file...
Aug 11 00:02:36 db2201 mysqld[6569]: Working directory at /srv/sqldata.s5
Aug 11 00:02:36 db2201 mysqld[6569]: Resource Limits (excludes unlimited resources):
Aug 11 00:02:36 db2201 mysqld[6569]: Limit                     Soft Limit           Hard Limit           Units
Aug 11 00:02:36 db2201 mysqld[6569]: Max stack size            8388608              unlimited            bytes
Aug 11 00:02:36 db2201 mysqld[6569]: Max core file size        0                    0                    bytes
Aug 11 00:02:36 db2201 mysqld[6569]: Max processes             2061488              2061488              processes
Aug 11 00:02:36 db2201 mysqld[6569]: Max open files            200001               200001               files
Aug 11 00:02:36 db2201 mysqld[6569]: Max locked memory         8388608              8388608              bytes
Aug 11 00:02:36 db2201 mysqld[6569]: Max pending signals       2061488              2061488              signals
Aug 11 00:02:36 db2201 mysqld[6569]: Max msgqueue size         819200               819200               bytes
Aug 11 00:02:36 db2201 mysqld[6569]: Max nice priority         0                    0
Aug 11 00:02:36 db2201 mysqld[6569]: Max realtime priority     0                    0
Aug 11 00:02:36 db2201 mysqld[6569]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t
Aug 11 00:02:36 db2201 mysqld[6569]: Kernel version: Linux version 6.1.0-50-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14+deb12u1) 1>
Aug 11 00:03:01 db2201 systemd[1]: mariadb@s5.service: Main process exited, code=killed, status=7/BUS
Aug 11 00:03:01 db2201 systemd[1]: mariadb@s5.service: Failed with result 'signal'.
Aug 11 00:03:01 db2201 systemd[1]: mariadb@s5.service: Unit process 1805985 (addr2line) remains running after unit stopped.
Aug 11 00:03:01 db2201 systemd[1]: mariadb@s5.service: Consumed 1d 22h 28min 47.368s CPU time.
Aug 11 00:03:06 db2201 systemd[1]: mariadb@s5.service: Scheduled restart job, restart counter is at 1.
Aug 11 00:03:06 db2201 systemd[1]: Stopped mariadb@s5.service - mariadb database server.
Aug 11 00:03:06 db2201 systemd[1]: mariadb@s5.service: Consumed 1d 22h 28min 47.368s CPU time.
Aug 11 00:03:06 db2201 systemd[1]: Starting mariadb@s5.service - mariadb database server...
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Warning] Could not increase number of max_open_files to more than 200001 (request: 80032>
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] Starting MariaDB 10.11.18-MariaDB-log source revision 197f92bee02d8e836f529f37625b>
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-i>
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] mysqld: Aria engine: starting recovery
Aug 11 00:03:06 db2201 mysqld[1806162]: recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 1 0
Aug 11 00:03:06 db2201 mysqld[1806162]:  (0.0 seconds);
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] mysqld: Aria engine: recovery done
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] InnoDB: Compressed tables use zlib 1.3.2
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] InnoDB: Number of transaction pools: 1
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] InnoDB: Using AVX512 instructions
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] InnoDB: Using Linux native AIO
Aug 11 00:03:06 db2201 mysqld[1806162]: 2026-08-11  0:03:06 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=235520m
Aug 11 00:03:18 db2201 mysqld[1806162]: 2026-08-11  0:03:18 0 [Note] InnoDB: Completed initialization of buffer pool
Aug 11 00:03:19 db2201 mysqld[1806162]: 2026-08-11  0:03:19 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Aug 11 00:03:19 db2201 mysqld[1806162]: 2026-08-11  0:03:19 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=36391680665684
Aug 11 00:03:21 db2201 mysqld[1806162]: 2026-08-11  0:03:21 0 [Note] InnoDB: End of log at LSN=36392066792903
Aug 11 00:03:21 db2201 mysqld[1806162]: 2026-08-11  0:03:21 0 [Note] InnoDB: To recover: 89208 pages
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: Last binlog file './db2201-bin.000768', position 782084474
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: 128 rollback segments are active.
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file ful>
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: log sequence number 36392066792903; transaction id 20921060860
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] InnoDB: Loading buffer pool(s) from /srv/sqldata.s5/ib_buffer_pool
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] Plugin 'FEEDBACK' is disabled.
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] Recovering after a crash using db2201-bin
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] Starting table crash recovery...
Aug 11 00:03:24 db2201 mysqld[1806162]: 2026-08-11  0:03:24 0 [Note] Crash table recovery finished.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Note] Server socket created on IP: '0.0.0.0', port: '3315'.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Note] Server socket created on IP: '::', port: '3315'.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Note] Server socket created on IP: '0.0.0.0', port: '3335'.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Note] Server socket created on IP: '::', port: '3335'.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Warning] 'proxies_priv' entry '@% root@db2101' ignored in --skip-name-resolve mode.
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 2 [Note] Event Scheduler: scheduler thread started with id 2
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break w>
Aug 11 00:03:26 db2201 mysqld[1806162]: 2026-08-11  0:03:26 0 [Note] /opt/wmf-mariadb1011/bin/mysqld: ready for connections.
Aug 11 00:03:26 db2201 mysqld[1806162]: Version: '10.11.18-MariaDB-log'  socket: '/run/mysqld/mysqld.s5.sock'  port: 3315  MariaDB Server
Aug 11 00:03:26 db2201 systemd[1]: Started mariadb@s5.service - mariadb database server.
Aug 11 00:03:55 db2201 mysqld[1806162]: 2026-08-11  0:03:55 0 [Note] InnoDB: Buffer pool(s) load completed at 260811  0:03:55

Event Timeline

I am going to restart replication so it can catch up, but I will likely rebuild the host after I check for possible bad hw source of it (memory or storage), as it is very coincidental both instances have crashed withing days.

This is a few days old:

Description: A critical diagnostic event occurred in the memory device at A8. Contact your service provider for assistance in replacing the device. (Extended ID: 0x4E41).
[Tue Aug  4 17:48:49 2026] {6}[Hardware Error]:   DIMM location: not present. DMI handle: 0x0000
[Tue Aug  4 17:48:49 2026] EDAC MC3: 1 UE memory read error on CPU_SrcID#0_MC#3_Chan#1_DIMM#0 (channel:1 slot:0 page:0x15e95e1 offset:0x240 grain:32 -  err_code:0x0000:0x009f  SystemAddress:0x15e95e1240 ProcessorSocketId:0x0 MemoryControllerId:0x3 ChannelAddress:0x2ad2bc240 ChannelId:0x1 RankAddress:0x15695e240 PhysicalRankId:0x0 DimmSlotId:0x0 Row:0x9b4f Column:0x48 Bank:0x3 BankGroup:0x0 ChipSelect:0x0 ChipId:0x0)

DIMM A8 DRAM DDR4 32 GB Presence Detected|Critical Error Dual Rank 2933 MT/s

@ DC ops, could you help us with this hw issue? Would it be possible to get a replacement if server is still under warranty or otherwise a spare stick. If would also be not a huge issue loss if we lost a few GB of ram by removing this and its dual ranked one either, as a last option, if not too far from replacement.

server is still under warranty. starting the process to getting it replaced.

While we wait for vendor servicing, x1 had redundancy and it was not the active host, so we are good there. s5 didn't, so I am doing this transfer to set up a new alternative host as the active backup source for that section. FYI @Marostegui :

cumin2003:~$ transfer.py --type=decompress dbprov2005.codfw.wmnet:/srv/backups/snapshots/ongoing/snapshot.s5.2026-08-05--20-00-02.tar.gz db2250.codfw.wmnet:/srv/sqldata.s5
2026-08-11 13:56:21  INFO: About to transfer /srv/backups/snapshots/ongoing/snapshot.s5.2026-08-05--20-00-02.tar.gz from dbprov2005.codfw.wmnet to ['db2250.codfw.wmnet']:['/srv/sqldata.s5'] (157515218051 bytes).

Thanks btw to both @Marostegui and @cezmunsta for the help debugging. Hopefully this is fixed or workarounded before my time off.

Change #1324577 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Setup db2250 additional s5 instance to replace db2201

https://gerrit.wikimedia.org/r/1324577

Change #1324577 merged by Jcrespo:

[operations/puppet@production] mariadb: Setup db2250 additional s5 instance to replace db2201

https://gerrit.wikimedia.org/r/1324577

Change #1324672 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Pool db2250:s5 to replace db2201:s5 for backups

https://gerrit.wikimedia.org/r/1324672

jcrespo renamed this task from db2201's both mysql instances crashed in the last few days to db2201 memory failure (was: both mysql instances crashed in the last few days).Wed, Aug 12, 10:50 AM

Change #1324672 merged by Jcrespo:

[operations/puppet@production] dbbackups: Pool db2250:s5 to replace db2201:s5 for backups

https://gerrit.wikimedia.org/r/1324672

Change #1324690 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Reenable notifications for db2201

https://gerrit.wikimedia.org/r/1324690

Change #1324690 abandoned by Jcrespo:

[operations/puppet@production] dbbackups: Reenable notifications for db2201

Reason:

This was already merged.

https://gerrit.wikimedia.org/r/1324690

Change #1324690 restored by Jcrespo:

[operations/puppet@production] dbbackups: Reenable notifications for db2201

https://gerrit.wikimedia.org/r/1324690

Change #1324690 merged by Jcrespo:

[operations/puppet@production] dbbackups: Disable notifications for db2201

https://gerrit.wikimedia.org/r/1324690

dell is sending us a replacement stick. delivery is taking a lot longer than i'd like.

@Jhancock.wm No worries. Host has notifications disabled, service has been migrated away and data is assumed corrupted- don't wait for permission to service it- just go ahead and shutdown the host at your convenience- the host will be reimaged to trixie afterwards deleting the os and its data.

@jcrespo replacement has been completed. all yours!

Jhancock.wm claimed this task.