TZ: UTC +1/+2
User Details
User Details
- User Since
- Sep 1 2016, 6:48 AM (369 w, 6 d)
- Availability
- Busy Busy until Oct 16.
- IRC Nick
- marostegui
- LDAP User
- Marostegui
- MediaWiki User
- MArostegui (WMF) [ Global Accounts ]
Fri, Sep 29
Fri, Sep 29
• Marostegui added a comment to T347318: db2109 crashed.
Excellent news!! Thank you so much
Thu, Sep 28
Thu, Sep 28
• Marostegui added a comment to T344036: Productionize db12[34-49].
They all look good and with the correct HW specs.
• Marostegui changed the status of T344036: Productionize db12[34-49], a subtask of T342166: Q1:rack/setup/install db12[34-49], from Stalled to Open.
• Marostegui added a comment to T342166: Q1:rack/setup/install db12[34-49].
Thank you!!
Wed, Sep 27
Wed, Sep 27
• Marostegui added a comment to T347381: Support maintain-dbusers on the new network layout.
• Marostegui added a comment to T343408: Productionize pc1015, p1016, pc2015 and pc2016.
• Marostegui added a comment to T343408: Productionize pc1015, p1016, pc2015 and pc2016.
pc1 has been entirely running 10.6 for months now
• Marostegui added a comment to T347381: Support maintain-dbusers on the new network layout.
Grants added across all the replicas, this is how they look now:
root@clouddb1013.eqiad.wmnet[(none)]> show grants for 'labsdbadmin'@'10.64.148.21'; +-----------------------------------------------------------------------------------------------------------------------+ | Grants for labsdbadmin@10.64.148.21 | +-----------------------------------------------------------------------------------------------------------------------+ | GRANT `labsdbuser` TO `labsdbadmin`@`10.64.148.21` WITH ADMIN OPTION | | GRANT SUPER ON *.* TO `labsdbadmin`@`10.64.148.21` IDENTIFIED BY PASSWORD '*xx' | | GRANT SELECT, INSERT, UPDATE ON `mysql`.* TO `labsdbadmin`@`10.64.148.21` | | GRANT SELECT, SHOW VIEW ON `%wik%`.* TO `labsdbadmin`@`10.64.148.21` | | GRANT SELECT, SHOW VIEW ON `%\_p`.* TO `labsdbadmin`@`10.64.148.21` WITH GRANT OPTION | +-----------------------------------------------------------------------------------------------------------------------+ 5 rows in set (0.001 sec)
• Marostegui added a comment to T343408: Productionize pc1015, p1016, pc2015 and pc2016.
Can we please go for MariaDB 10.6 on those hosts?
Adding mariadb::package: 'wmf-mariadb106' on their yaml files should be enough (pc1015.yaml etc)
• Marostegui added a comment to T342164: Q1:rack/setup/install pc101[56].
Thank you!
• Marostegui updated subscribers of T341121: Decommission dbproxy10[12-17].
@ABran-WMF this could be your first decommission :)
• Marostegui moved T343408: Productionize pc1015, p1016, pc2015 and pc2016 from Blocked to Ready on the DBA board.
• Marostegui moved T346454: Master and candidate master of s5 and s8 in eqiad are in the same row from Triage to Blocked on the DBA board.
• Marostegui added a comment to T347318: db2109 crashed.
@Jhancock.wm were you able to see anything?
• Marostegui added a comment to T343408: Productionize pc1015, p1016, pc2015 and pc2016.
Btw I just ran this:
[08:45:24] marostegui@cumin1001:~$ sudo cumin 'pc[1015-1016]'.eqiad.wmnet 'lvextend -L+1T /dev/mapper/tank-data ; xfs_growfs /srv'
2 hosts will be targeted:
pc[1015-1016].eqiad.wmnet
OK to proceed on 2 hosts? Enter the number of affected hosts to confirm or "q" to quit: 2
===== NODE GROUP =====
(2) pc[1015-1016].eqiad.wmnet
----- OUTPUT of 'lvextend -L+1T /... xfs_growfs /srv' -----
Size of logical volume tank/data changed from <7.56 TiB (1981022 extents) to <8.56 TiB (2243166 extents).
Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data isize=512 agcount=32, agsize=63392704 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0
data = bsize=4096 blocks=2028566528, imaxpct=5
= sunit=64 swidth=256 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 2028566528 to 2297001984
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [00:00<00:00, 2.93hosts/s]
FAIL | | 0% (0/2) [00:00<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'lvextend -L+1T /... xfs_growfs /srv'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.• Marostegui added a project to T347381: Support maintain-dbusers on the new network layout: Data-Persistence (work done).
I have created the users requested on that patch, even though I have -1 it as it needs something else anyways.
Does that labsadmin user also needs a password? Keep in mind it is being assigned to a role, if so, please let me know where it is
• Marostegui changed the status of T343408: Productionize pc1015, p1016, pc2015 and pc2016, a subtask of T342163: Q1:rack/setup/install pc201[56], from Stalled to Open.
• Marostegui changed the status of T343408: Productionize pc1015, p1016, pc2015 and pc2016, a subtask of T342164: Q1:rack/setup/install pc101[56], from Stalled to Open.
• Marostegui changed the status of T343408: Productionize pc1015, p1016, pc2015 and pc2016 from Stalled to Open.
@Ladsgroup @ABran-WMF this can now go
• Marostegui updated subscribers of T347449: Degraded RAID on dbstore1005.
[2574910.962214] megaraid_sas 0000:18:00.0: scanning for scsi0... [2574910.962794] megaraid_sas 0000:18:00.0: 1244 (749109359s/0x0001/CRIT) - VD 00/0 is now DEGRADED [2575154.297980] megaraid_sas 0000:18:00.0: 1246 (749109605s/0x0004/CRIT) - Enclosure PD 20(c None/p1) phy bad for slot 1 [2576648.220845] Process accounting resumed [2577134.294399] megaraid_sas 0000:18:00.0: 1255 (749111585s/0x0004/CRIT) - Enclosure PD 20(c None/p1) phy bad for slot 4
Mon, Sep 25
Mon, Sep 25
Downtimed it for a week
• Marostegui closed T345509: db1128 mariadb crashed, a subtask of T344309: Compile and package MariaDB 11.0.3 10.6.15, 10.4.31 , as Resolved.
• Marostegui added a comment to T345509: db1128 mariadb crashed.
I have enabled notifications and repooled the host
Fri, Sep 22
Fri, Sep 22
• Marostegui added a comment to T347140: Switchover s7 master (db1181 -> db1136).
This was a test
• Marostegui added a comment to T345509: db1128 mariadb crashed.
This host seems stable, no crashes since it was recloned and downgraded, if it keeps like that by Monday, I will repool it
Thu, Sep 21
Thu, Sep 21
This is done
• Marostegui closed T345409: Database post-switchover tasks, a subtask of T345263: September 2023 Datacenter Switchover, as Resolved.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui changed the status of T345409: Database post-switchover tasks, a subtask of T345263: September 2023 Datacenter Switchover, from Stalled to Open.
• Marostegui moved T345409: Database post-switchover tasks from Blocked to In progress on the DBA board.
Wed, Sep 20
Wed, Sep 20
• Marostegui edited projects for T346375: [Spike] Consider the impact of dark mode and font size on user preferences table, added: Data-Persistence; removed DBA.
• Marostegui added a comment to T346464: Experiment with InnoDB buffer pool size on clouddb1019.eqiad.wmnet.
Whatever is decided, I'd prefer if we don't create configuration snowflakes. All hosts serving the same type of queries (web or analytics) should have the same configuration, otherwise we are going to run into problems in the long term.
Ideally we shouldn't even have differences between web and analytics roles, but I guess we can make an exception here, if we are really aware of the tradeoffs they might have too.
Tue, Sep 19
Tue, Sep 19
• Marostegui added a comment to T345509: db1128 mariadb crashed.
It's been a few hours since I downgraded+recloned and so far no crashes (with 10.4.31 it'd crash a lot faster), so I've updated MariaDB's bug with that information accordingly.
Anyways, I am not repooling this host until next week, just to make _fully_ sure it won't crash again.
• Marostegui updated subscribers of T298315: Deleting Ns:104 in it:voy.
I don't know if @Ladsgroup can help with this or we need Platform Engineering for it.
• Marostegui added a comment to T298315: Deleting Ns:104 in it:voy.
This would need to be done with a script via MW, we do not execute commands directly on the DB.
• Marostegui added a comment to T345509: db1128 mariadb crashed.
It cashed again, so I've downgraded it and started the recloning process
• Marostegui edited projects for T346679: Surface Temporary user information to Cloud Wiki Replicas, added: Data-Services, cloud-services-team, Data-Engineering; removed Patch-For-Review, Schema-change-in-production, DBA.
• Marostegui added a comment to T346679: Surface Temporary user information to Cloud Wiki Replicas.
The views part is handled by either cloud-services-team and/or Data-Engineering
Mon, Sep 18
Mon, Sep 18
• Marostegui added a comment to T346454: Master and candidate master of s5 and s8 in eqiad are in the same row.
• Marostegui added a comment to T345509: db1128 mariadb crashed.
I gave it one more chance as it looked like the table was corrupted (again):
mysql:root@localhost [enwiki]> check table user_properties quick; +------------------------+-------+----------+------------------------------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +------------------------+-------+----------+------------------------------------------------------------------------------+ | enwiki.user_properties | check | Warning | InnoDB: Index 'up_property' contains 270351545 entries, should be 270351543. | | enwiki.user_properties | check | error | Corrupt | +------------------------+-------+----------+------------------------------------------------------------------------------+ 2 rows in set (6 min 47.202 sec)
• Marostegui added a comment to T345509: db1128 mariadb crashed.
It crashed again a bit later than when I started replication yesterday:
Sep 17 10:07:25 db1128 mysqld[2681127]: 2023-09-17 10:07:25 0x7f219c360700 InnoDB: Assertion failure in file /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc line 219 Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: Failing assertion: !cursor->index->is_committed() Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: We intentionally generate a memory trap. Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: If you get repeated assertion failures or crashes, even Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: immediately after the mysqld startup, there may be Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: corruption in the InnoDB tablespace. Please refer to Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ Sep 17 10:07:25 db1128 mysqld[2681127]: InnoDB: about forcing recovery.
It was the same table again:
Sep 17 10:07:51 db1128 mysqld[2681127]: Some pointers may be invalid and cause the dump to abort. Sep 17 10:07:51 db1128 mysqld[2681127]: Query (0x7ec274397e85): INSERT /* MediaWiki\User\UserOptionsManager::saveOptionsInternal */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (3527085,'echo-subscriptions-push-ge-newcomer',''),(3527085,'ipinfo-beta-feature-enable',''),(3527085,'usecodemirror-colorblind','')
Sun, Sep 17
Sun, Sep 17
• Marostegui added a comment to T345509: db1128 mariadb crashed.
All the optimizes finished, so I have restarted replication. Let's see if it crashes again.
Fri, Sep 15
Fri, Sep 15
• Marostegui added a comment to T337446: Rebuild sanitarium hosts.
• Marostegui added a comment to T346454: Master and candidate master of s5 and s8 in eqiad are in the same row.
Not sure if by move you mean physically or logically. The latter is the best/easiest way to do so. Just converting a different host into candidate master
• Marostegui added a comment to T345831: Migrate DB grants for testreduce1001 to testreduce1002.
Yeah, misc clusters aren't part of the switchover.
Per T345509#9166433 I am not going to push 10.4.31 to the repo, so this ticket is done as 10.6.31 has been pushed. We can do 11 series later, as it is not a huge priority now to keep testing those minors.
Thu, Sep 14
Thu, Sep 14
I have been debugging this with MariaDB and it looks like we've simply hit this bug: https://jira.mariadb.org/browse/MDEV-32132.
It is not specific from this version, it is related to the change_buffer (which we had enabled in the past - and it is now disabled by default in MariaDB 10.5 and higher).
Essentially if we ran a DROP+CREATE index (which we did) and there were changes pending on the buffer, that could have potentially corrupted the table.
• Marostegui updated the task description for T345409: Database post-switchover tasks.
• Marostegui closed T345408: Database pre-switchover tasks, a subtask of T345263: September 2023 Datacenter Switchover, as Resolved.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui added a comment to P52505 (An Untitled Masterwork).
(gdb) print *ibuf
$6 = {size = 1, max_size = 6176768, seg_size = 185535, empty = true, free_list_len = 185533, height = 1, index = 0x55573c63ae58,
n_merges = {m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}},
n_merged_ops = {{m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}, {
m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}, {
m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}}, n_discarded_ops = {
{m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}, {
m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}, {
m_counter = {<std::__atomic_base<unsigned long>> = {static _S_alignment = 8, _M_i = 0}, <No data fields>}}}}• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui added a comment to P52505 (An Untitled Masterwork).
(gdb) set max-value-size 1000000
(gdb) set print elements 1000000
(gdb) frame 17
#17 0x0000555555f5403d in Query_log_event::do_apply_event (this=0x7fa0e8014fa8, rgi=0x7fa0e8000c40,
query_arg=0x7fa0e8a79625 "INSERT /* MediaWiki\\User\\UserOptionsManager::saveOptionsInternal */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (20542576,'advancedsearch-disable-local-exception',''),(20542576,'betafeatures-auto-enroll-local-exception',''),(20542576,'ccmeonemails-local-exception',''),(20542576,'centralnotice-display-campaign-type-advocacy',''),(20542576,'centralnotice-display-campaign-type-advocacy-local-exception',''),(20542576,'centralnotice-display-campaign-type-article-writing-local-exception',''),(20542576,'centralnotice-display-campaign-type-event-local-exception',''),(20542576,'centralnotice-display-campaign-type-fundraising',''),(20542576,'centralnotice-display-campaign-type-fundraising-local-exception',''),(20542576,'centralnotice-display-campaign-type-governance-local-exception',''),(20542576,'centralnotice-display-campaign-type-photography-local-exception',''),(20542576,'cirrussearch-pref-completion-profile-local-exception',''),(20542576,'compact-language-links-local-exception',''),(20542576,'date-local-exception',''),(20542576,'diffonly-local-exception',''),(20542576,'disablemail-local-exception',''),(20542576,'discussiontools-autotopicsub-local-exception',''),(20542576,'discussiontools-betaenable-local-exception',''),(20542576,'discussiontools-newtopictool-createpage-local-exception',''),(20542576,'discussiontools-newtopictool-local-exception',''),(20542576,'discussiontools-replytool-local-exception',''),(20542576,'discussiontools-sourcemodetoolbar-local-exception',''),(20542576,'discussiontools-topicsubscription-local-exception',''),(20542576,'discussiontools-visualenhancements-local-exception',''),(20542576,'echo-cross-wiki-notifications',''),(20542576,'echo-cross-wiki-notifications-local-exception',''),(20542576,'echo-dont-email-read-notifications-local-exception',''),(20542576,'echo-email-format-local-exception',''),(20542576,'echo-email-frequency-local-exception',''),(20542576,'echo-notifications-page-linked-title-muted-list-local-exception',''),(20542576,'echo-subscriptions-email-ge-newcomer',''),(20542576,'echo-subscriptions-email-login-fail',''),(20542576,'echo-subscriptions-email-login-success',''),(20542576,'echo-subscriptions-push-dt-subscription-archiving',''),(20542576,'echo-subscriptions-push-ge-newcomer',''),(20542576,'echo-subscriptions-web-ge-newcomer',''),(20542576,'editfont-local-exception',''),(20542576,'email-allow-new-users-local-exception',''),(20542576,'enotifminoredits-local-exception',''),(20542576,'enotifwatchlistpages-local-exception',''),(20542576,'extendwatchlist-local-exception',''),(20542576,'forceeditsummary-local-exception',''),(20542576,'gadget-ReferenceTooltips',''),(20542576,'gadget-extra-toolbar-buttons',''),(20542576,'gender-local-exception',''),(20542576,'growthexperiments-help-panel-tog-help-panel-local-exception',''),(20542576,'growthexperiments-homepage-enable-local-exception',''),(20542576,'hidecategorization',''),(20542576,'hidecategorization-local-exception',''),(20542576,'hideminor-local-exception',''),(20542576,'imagesize-local-exception',''),(20542576,'ipinfo-beta-feature-enable-local-exception',''),(20542576,'language-local-exception',''),(20542576,'math-local-exception',''),(20542576,'multimediaviewer-enable-local-exception',''),(20542576,'norollbackdiff-local-exception',''),(20542576,'previewonfirst-local-exception',''),(20542576,'previewontop',''),(20542576,'previewontop-local-exception',''),(20542576,'rcdays-local-exception',''),(20542576,'rcenhancedfilters-disable-local-exception',''),(20542576,'rclimit-local-exception',''),(20542576,'rcshowwikidata-local-exception',''),(20542576,'requireemail-local-exception',''),(20542576,'revisionslider-disable-local-exception',''),(20542576,'searchlimit-local-exception',''),(20542576,'showhiddencats-local-exception',''),(20542576,'skin-local-exception',''),(20542576,'skin-responsive-local-exception',''),(20542576,'thumbsize-local-exception',''),(20542576,'timecorrection-local-exception',''),(20542576,'twocolconflict-local-exception',''),(20542576,'underline-local-exception',''),(20542576,'usebetatoolbar-local-exception','1'),(20542576,'usecodemirror-colorblind',''),(20542576,'usecodemirror-colorblind-local-exception',''),(20542576,'useeditwarning-local-exception',''),(20542576,'uselivepreview-local-exception',''),(20542576,'usenewrc-local-exception',''),(20542576,'vector-limited-width-local-exception',''),(20542576,'visualeditor-betatempdisable-local-exception',''),(20542576,'visualeditor-newwikitext-local-exception',''),(20542576,'watchcreations-local-exception',''),(20542576,'watchdefault-local-exception',''),(20542576,'watchlistdays-local-exception',''),(20542576,'watchlisthideanons-local-exception',''),(20542576,'watchlisthideliu-local-exception',''),(20542576,'watchlisthideminor-local-exception',''),(20542576,'watchlisthideown-local-exception',''),(20542576,'watchlistunwatchlinks-local-exception',''),(20542576,'watchmoves-local-exception',''),(20542576,'wikilove-enabled-local-exception',''),(20542576,'wlenhancedfilters-disable-local-exception',''),(20542576,'wllimit-local-exception',''),(20542576,'wlshowwikibase-local-exception',''),(20542576,'usebetatoolbar',''),(20542576,'echo-notifications-page-linked-title-muted-list-local-exception-local-exception','')", q_len_arg=<optimized out>) at /root/mariadb-10.4.31/sql/sql_class.h:227
227 in /root/mariadb-10.4.31/sql/sql_class.hGreat - I am closing this for now. Reopen if you need something else.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui added a comment to P52505 (An Untitled Masterwork).
(gdb) print/x (*offsets)[1]@16
$4 = {0x2, 0x8006, 0x24, 0x28, 0xbd2c, 0x7fb8, 0x0, 0x4807, 0xbd2c, 0x7fb8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4960}• Marostegui added a comment to P52505 (An Untitled Masterwork).
$3 = {0x2000, 0x7fa000000004, 0x100a000000304, 0x7fff9c753ae0, 0x7fff9c7548b0, 0x9bcfc00000304, 0x2, 0x2, 0x3, 0x7facef0f8f90, 0x0,
0x1, 0x0, 0x2, 0x7fa0e8a6a228, 0x0}• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui added a comment to P52505 (An Untitled Masterwork).
(gdb) print *cursor.index.fields@cursor.index.n_fields
$2 = {{col = 0x7fa0e8a7eb58, name = {m_name = 0x7fa0e8a7ec58 "up_property"}, prefix_len = 0, fixed_len = 0}, {col = 0x7fa0e8a7eb38,
name = {m_name = 0x7fa0e8a7ec50 "up_user"}, prefix_len = 0, fixed_len = 4}}
(gdb) print/x offsets[1]@offsets[1]+16
Non-integral right operand for "@" operator.• Marostegui added a comment to P52505 (An Untitled Masterwork).
(gdb) frame 3
#3 0x0000555555b06f0b in row_ins_sec_index_entry_by_modify (mtr=0x7fff9c754950, thr=0x7fa0e8a7ac50, entry=<optimized out>, heap=0x7fa0e8a7c990, offsets_heap=<optimized out>, offsets=0x7fff9c7539f8, cursor=0x7fff9c753a70, mode=2, flags=0)
at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:219
219 /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc: No such file or directory.
(gdb) print *cursor
$1 = {index = 0x7fa0e8a6a228, page_cur = {index = 0x0, rec = 0x7fc48aeb9edc "enotifwatchlistpages-local-exception\001\071tp$\004\005`", offsets = 0x0, block = 0x7fc48ac1a2a0}, purge_node = 0x0, left_block = 0x0, thr = 0x7fa0e8a7ac50, flag = BTR_CUR_BINARY,
tree_height = 4, up_match = 1, up_bytes = 0, low_match = 2, low_bytes = 0, n_fields = 9992387443228972, n_bytes = 140325174312999, fold = 140735818319936, path_arr = 0x555556270dec
<btr_cur_search_to_nth_level(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, char const*, unsigned int, mtr_t*, unsigned long)+3180>, rtr_info = 0x0}• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui added a comment to P52505 (An Untitled Masterwork).
#0 0x00007ffff7671ce1 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff765b537 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000555555b1a8c0 in ut_dbg_assertion_failed (expr=expr@entry=0x55555661ade8 "!cursor->index->is_committed()", file=file@entry=0x55555661ac68 "/root/mariadb-10.4.31/storage/innobase/row/row0ins.cc", line=line@entry=219)
at /root/mariadb-10.4.31/storage/innobase/ut/ut0dbg.cc:60
#3 0x0000555555b06f0b in row_ins_sec_index_entry_by_modify (mtr=0x7fff9c754950, thr=0x7fa0e8a7ac50, entry=<optimized out>, heap=0x7fa0e8a7c990, offsets_heap=<optimized out>, offsets=0x7fff9c7539f8, cursor=0x7fff9c753a70, mode=2, flags=0)
at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:219
#4 row_ins_sec_index_entry_low (flags=<optimized out>, mode=<optimized out>, index=0x7fa0e8a6a228, offsets_heap=<optimized out>, heap=<optimized out>, entry=<optimized out>, trx_id=<optimized out>, thr=<optimized out>)
at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3149
#5 0x00005555561d8717 in row_ins_sec_index_entry (index=0x7fa0e8a6a228, entry=0x7fa0e8053500, thr=0x7fa0e8a7ac50, check_foreign=<optimized out>) at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3361
#6 0x00005555561d8f65 in row_ins_index_entry (thr=0x7fa0e8a7ac50, entry=<optimized out>, index=<optimized out>) at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3409
#7 row_ins_index_entry_step (thr=0x7fa0e8a7ac50, node=0x7fa0e8a77960) at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3576
#8 row_ins (thr=<optimized out>, node=<optimized out>) at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3713
#9 row_ins_step (thr=thr@entry=0x7fa0e8a7ac50) at /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc:3856
#10 0x00005555561ea1e5 in row_insert_for_mysql (mysql_rec=<optimized out>, prebuilt=0x7fa0e8a76dc8, ins_mode=ROW_INS_NORMAL) at /root/mariadb-10.4.31/storage/innobase/row/row0mysql.cc:1395
#11 0x000055555612d8ef in ha_innobase::write_row (this=0x7fa0e8a6b810, record=0x7fa0e8a69e00 "\376pt9\001$enotifwatchlistpages-local-exception") at /root/mariadb-10.4.31/storage/innobase/handler/ha_innodb.cc:8169
#12 0x0000555555e4aea6 in handler::ha_write_row (this=0x7fa0e8a6b810, buf=0x7fa0e8a69e00 "\376pt9\001$enotifwatchlistpages-local-exception") at /root/mariadb-10.4.31/sql/handler.cc:6850
#13 0x0000555555bf8a8d in write_record (thd=thd@entry=0x7fa0e8001538, table=table@entry=0x7fa0e8a75788, info=info@entry=0x7fff9c7558c0) at /root/mariadb-10.4.31/sql/sql_insert.cc:2085
#14 0x0000555555c0342f in mysql_insert (thd=thd@entry=0x7fa0e8001538, table_list=<optimized out>, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=true) at /root/mariadb-10.4.31/sql/sql_insert.cc:1084
#15 0x0000555555c2fc6c in mysql_execute_command (thd=0x7fa0e8001538) at /root/mariadb-10.4.31/sql/sql_parse.cc:4613
#16 0x0000555555c36af3 in mysql_parse (thd=0x7fa0e8001538, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7fff9c757430, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false)
at /root/mariadb-10.4.31/sql/sql_parse.cc:8010
#17 0x0000555555f5403d in Query_log_event::do_apply_event (this=0x7fa0e8014fa8, rgi=0x7fa0e8000c40,
query_arg=0x7fa0e8a79625 "INSERT /* MediaWiki\\User\\UserOptionsManager::saveOptionsInternal */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (20542576,'advancedsearch-disable-local-exception',''),(2054257"..., q_len_arg=<optimized out>)
at /root/mariadb-10.4.31/sql/sql_class.h:227
#18 0x0000555555b7ca24 in Log_event::apply_event (rgi=0x7fa0e8000c40, this=<optimized out>) at /root/mariadb-10.4.31/sql/log_event.h:1492
#19 apply_event_and_update_pos_apply (ev=ev@entry=0x7fa0e8014fa8, thd=thd@entry=0x7fa0e8001538, rgi=rgi@entry=0x7fa0e8000c40, reason=<optimized out>) at /root/mariadb-10.4.31/sql/slave.cc:3819
#20 0x0000555555b84c93 in apply_event_and_update_pos (ev=ev@entry=0x7fa0e8014fa8, thd=thd@entry=0x7fa0e8001538, rgi=rgi@entry=0x7fa0e8000c40) at /root/mariadb-10.4.31/sql/slave.cc:3981
#21 0x0000555555b869a2 in exec_relay_log_event (serial_rgi=0x7fa0e8000c40, rli=0x55573c769da8, thd=0x7fa0e8001538) at /root/mariadb-10.4.31/sql/slave.cc:4340
#22 handle_slave_sql (arg=arg@entry=0x55573c7683e0) at /root/mariadb-10.4.31/sql/slave.cc:5526
#23 0x00005555560cbc02 in pfs_spawn_thread (arg=0x55573c819df8) at /root/mariadb-10.4.31/storage/perfschema/pfs.cc:1869
#24 0x00007ffff7b2dea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#25 0x00007ffff7734a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6• Marostegui added a comment to P52505 (An Untitled Masterwork).
mysql:root@localhost [enwiki]> show create table user_properties\G
*************************** 1. row ***************************
Table: user_properties
Create Table: CREATE TABLE `user_properties` (
`up_user` int(10) unsigned NOT NULL,
`up_property` varbinary(255) NOT NULL DEFAULT '',
`up_value` blob DEFAULT NULL,
PRIMARY KEY (`up_user`,`up_property`),
KEY `up_property` (`up_property`)
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED
1 row in set (0.000 sec)• Marostegui added a comment to P52505 (An Untitled Masterwork).
Sep 11 04:51:45 db1128 mysqld[3056275]: 2023-09-11 04:51:45 0x7ed350f16700 InnoDB: Assertion failure in file /root/mariadb-10.4.31/storage/innobase/row/row0ins.cc line 219 Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: Failing assertion: !cursor->index->is_committed() Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: We intentionally generate a memory trap. Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: If you get repeated assertion failures or crashes, even Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: immediately after the mysqld startup, there may be Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: corruption in the InnoDB tablespace. Please refer to Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ Sep 11 04:51:45 db1128 mysqld[3056275]: InnoDB: about forcing recovery. Sep 11 04:51:45 db1128 mysqld[3056275]: 230911 4:51:45 [ERROR] mysqld got signal 6 ; Sep 11 04:51:45 db1128 mysqld[3056275]: This could be because you hit a bug. It is also possible that this binary Sep 11 04:51:45 db1128 mysqld[3056275]: or one of the libraries it was linked against is corrupt, improperly built, Sep 11 04:51:45 db1128 mysqld[3056275]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 11 04:51:45 db1128 mysqld[3056275]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 11 04:51:45 db1128 mysqld[3056275]: We will try our best to scrape up some info that will hopefully help Sep 11 04:51:45 db1128 mysqld[3056275]: diagnose the problem, but since we have already crashed, Sep 11 04:51:45 db1128 mysqld[3056275]: something is definitely wrong and this may fail. Sep 11 04:51:45 db1128 mysqld[3056275]: Server version: 10.4.31-MariaDB-log source revision: 2aea9387497cecb5668ef605b8f80886f9de812c Sep 11 04:51:45 db1128 mysqld[3056275]: key_buffer_size=134217728 Sep 11 04:51:45 db1128 mysqld[3056275]: read_buffer_size=131072 Sep 11 04:51:45 db1128 mysqld[3056275]: max_used_connections=42 Sep 11 04:51:45 db1128 mysqld[3056275]: max_threads=2010 Sep 11 04:51:45 db1128 mysqld[3056275]: thread_count=10 Sep 11 04:51:45 db1128 mysqld[3056275]: It is possible that mysqld could use up to Sep 11 04:51:45 db1128 mysqld[3056275]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4753830 K bytes of memory Sep 11 04:51:45 db1128 mysqld[3056275]: Hope that's ok; if not, decrease some variables in the equation. Sep 11 04:51:45 db1128 mysqld[3056275]: Thread pointer: 0x7ed1f4010368 Sep 11 04:51:45 db1128 mysqld[3056275]: Attempting backtrace. You can use the following information to find out Sep 11 04:51:45 db1128 mysqld[3056275]: where mysqld died. If you see no messages after this, something went Sep 11 04:51:45 db1128 mysqld[3056275]: terribly wrong... Sep 11 04:51:45 db1128 mysqld[3056275]: stack_bottom = 0x7ed350f15760 thread_stack 0x30000 Sep 11 04:51:45 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(my_print_stacktrace+0x2e)[0x56415f21070e] Sep 11 04:51:45 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(handle_fatal_signal+0x55d)[0x56415ec92c7d] Sep 11 04:51:45 db1128 mysqld[3056275]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f323b318140] Sep 11 04:51:46 db1128 mysqld[3056275]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f323ae50ce1] Sep 11 04:51:46 db1128 mysqld[3056275]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f323ae3a537] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0x5c68c0)[0x56415e9708c0] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0x5b2f0b)[0x56415e95cf0b] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0xc84717)[0x56415f02e717] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0xc84f65)[0x56415f02ef65] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0xc961e5)[0x56415f0401e5] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0xbd98ef)[0x56415ef838ef] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_ZN7handler12ha_write_rowEPKh+0x326)[0x56415eca0ea6] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x19d)[0x56415ea4ea8d] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0xb0f)[0x56415ea5942f] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_Z21mysql_execute_commandP3THD+0x191c)[0x56415ea85c6c] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x223)[0x56415ea8caf3] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(_ZN15Query_log_event14do_apply_eventEP14rpl_group_infoPKcj+0x6dd)[0x56415edaa03d] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0x628a24)[0x56415e9d2a24] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(handle_slave_sql+0x1672)[0x56415e9dc9a2] Sep 11 04:51:46 db1128 mysqld[3056275]: /opt/wmf-mariadb104/bin/mysqld(+0xb77c02)[0x56415ef21c02] Sep 11 04:51:46 db1128 mysqld[3056275]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f323b30cea7] Sep 11 04:51:46 db1128 mysqld[3056275]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f323af13a2f] Sep 11 04:51:46 db1128 mysqld[3056275]: Trying to get some variables. Sep 11 04:51:46 db1128 mysqld[3056275]: Some pointers may be invalid and cause the dump to abort. Sep 11 04:51:46 db1128 mysqld[3056275]: Query (0x7ed1f4061655): INSERT /* MediaWiki\User\UserOptionsManager::saveOptionsInternal */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (395179,'echo-subscriptions-push-ge-newcomer',''),(395179,'skin','timeless'),(395179,'twocolconflict',''),(395179,'usecodemirror-colorblind','') Sep 11 04:51:46 db1128 mysqld[3056275]: Connection ID (thread ID): 250653 Sep 11 04:51:46 db1128 mysqld[3056275]: Status: NOT_KILLED Sep 11 04:51:46 db1128 mysqld[3056275]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on Sep 11 04:51:46 db1128 mysqld[3056275]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains Sep 11 04:51:46 db1128 mysqld[3056275]: information that should help you find out what is causing the crash. Sep 11 04:51:46 db1128 mysqld[3056275]: Writing a core file... Sep 11 04:51:46 db1128 mysqld[3056275]: Working directory at /srv/sqldata Sep 11 04:51:46 db1128 mysqld[3056275]: Resource Limits: Sep 11 04:51:46 db1128 mysqld[3056275]: Limit Soft Limit Hard Limit Units Sep 11 04:51:46 db1128 mysqld[3056275]: Max cpu time unlimited unlimited seconds Sep 11 04:51:46 db1128 mysqld[3056275]: Max file size unlimited unlimited bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max data size unlimited unlimited bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max stack size 8388608 unlimited bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max core file size 0 0 bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max resident set unlimited unlimited bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max processes 2058221 2058221 processes Sep 11 04:51:46 db1128 mysqld[3056275]: Max open files 200001 200001 files Sep 11 04:51:46 db1128 mysqld[3056275]: Max locked memory 65536 65536 bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max address space unlimited unlimited bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max file locks unlimited unlimited locks Sep 11 04:51:46 db1128 mysqld[3056275]: Max pending signals 2058221 2058221 signals Sep 11 04:51:46 db1128 mysqld[3056275]: Max msgqueue size 819200 819200 bytes Sep 11 04:51:46 db1128 mysqld[3056275]: Max nice priority 0 0 Sep 11 04:51:46 db1128 mysqld[3056275]: Max realtime priority 0 0 Sep 11 04:51:46 db1128 mysqld[3056275]: Max realtime timeout unlimited unlimited us Sep 11 04:51:46 db1128 mysqld[3056275]: Core pattern: /var/tmp/core/core.%h.%e.%p.%t Sep 11 04:51:46 db1128 mysqld[3056275]: Kernel version: Linux version 5.10.0-25-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.191-1 (2023-08-16) Sep 11 04:51:46 db1128 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui updated the task description for T345408: Database pre-switchover tasks.
• Marostegui moved T345408: Database pre-switchover tasks from Ready to In progress on the DBA board.
Wed, Sep 13
Wed, Sep 13
• Marostegui added a comment to T345509: db1128 mariadb crashed.
Yeah I saw it
• Marostegui reopened T345509: db1128 mariadb crashed, a subtask of T344309: Compile and package MariaDB 11.0.3 10.6.15, 10.4.31 , as Open.
• Marostegui reopened T345509: db1128 mariadb crashed as "Open".
• Marostegui added a comment to T345211: Re-create querysampler database ID for Wiki Replicas clouddb databases.
great! can I close this?
• Marostegui added a comment to T345211: Re-create querysampler database ID for Wiki Replicas clouddb databases.
Yeah, it needs to be reloaded. I haven't touched it cause we do not own them and the alert has been there for 12 days according to Icinga, that's why I created the task for WMCS, as they own those hosts.
This was done
• Marostegui closed T339326: Compile MariaDB 10.6 on bookworm, a subtask of T339185: Test MariaDB + Debian bookworm on databases, as Resolved.
I am going to decline the task for now per @Umherirrender's comment. Also, if we need to delete something, I'd prefer if it is done via a maintenance script using MW - we tend to avoid deleting records on the database's prompt unless it is super critical/an emergency etc.
• Marostegui added a comment to T344309: Compile and package MariaDB 11.0.3 10.6.15, 10.4.31 .
Pushed 10.6.15 to the repo