Page MenuHomePhabricator
Create Task
Maniphest T315274

Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active; a caller might have failed to call endAtomic() or cancelAtomic().
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active; a caller might have failed to call endAtomic() or cancelAtomic().
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.23/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1694)
#0 /srv/mediawiki/php-1.39.0-wmf.23/includes/libs/rdbms/lbfactory/LBFactory.php(324): Wikimedia\Rdbms\LoadBalancer->approvePrimaryChanges(array, string)
#1 /srv/mediawiki/php-1.39.0-wmf.23/includes/MediaWiki.php(671): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, array)
#2 /srv/mediawiki/php-1.39.0-wmf.23/includes/api/ApiMain.php(901): MediaWiki::preOutputCommit(DerivativeContext)
#3 /srv/mediawiki/php-1.39.0-wmf.23/includes/api/ApiMain.php(846): ApiMain->executeActionWithErrorHandling()
#4 /srv/mediawiki/php-1.39.0-wmf.23/api.php(90): ApiMain->execute()
#5 /srv/mediawiki/php-1.39.0-wmf.23/api.php(45): wfApiMain()
#6 /srv/mediawiki/w/api.php(3): require(string)
#7 {main}
Impact
Notes

https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Database_error — user report on en.wiki at 2022-08-16T04:10:00.000Z

[c42d457a-4fe1-4b6f-b69f-0cf0ad0ab0b9] 2022-08-16 04:08:56: Fatal exception of type "Wikimedia\Rdbms\DBTransactionError"

Details

Request URL
https://en.wikipedia.org/w/api.php
SubjectRepoBranchLines +/-
SqlBagOStuff: use cancelAtomic()mediawiki/corewmf/1.39.0-wmf.25+19 -12
SqlBagOStuff: use cancelAtomic()mediawiki/coremaster+19 -12
objectcache: Add trace to SqlBagOStuff DBError loggingmediawiki/coremaster+4 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaaronT252951 ResourceLoader DepStore lock acquired twice?
 ResolvedjijikiT267581 Phase out "redis_sessions" cluster and away from memcached cluster
 ResolvedaaronT88445 MediaWiki active/active datacenter investigation and work (tracking)
 ResolvedKrinkleT270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
 ResolvedtstarlingT267270 Determine multi-dc strategy for CentralAuth
 ResolvedaaronT278392 Storage solution for cross-datacenter tokens
 ResolvedPRODUCTION ERRORtstarlingT311590 ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}
 ResolvedtstarlingT313496 Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask
 ResolvedtstarlingT91820 Create HTTP verb and sticky cookie DC routing in VCL
ResolvedaaronT91816 Add code to enable setting sticky DC cookies for POST requests
 ResolvedaaronT121440 Dedicated post-edit cache busting cookie to prevent stale reads (session consistency)
 ResolvedKrinkleT113916 Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
 ResolvedaaronT212129 Move MainStash out of Redis to a simpler multi-dc aware solution
 ResolvedEevansT222851 Improve Echo seentime code for multi-DC access
 ResolvedaaronT229062 Look into a simple way to have global keys with db-replicated
 ResolvedKrinkleT254634 Determine and implement multi-dc strategy for ChronologyProtector
Unknown Object (Task)
 ResolvedPapaulT267041 (Need By: 2020-11-29) rack/setup/install db214[234]
 Resolved• MarosteguiT269324 Productionize x2 databases
 ResolvedaaronT274174 Add modtoken field and flags to objectcache table
 ResolvedKrinkleT288998 Significant ParserCache space increase after 2021-08-12 (1.37.0-wmf.18 regression)
 ResolvedtstarlingT306118 Notify DBA prior to sending db traffic to x2
 ResolvedtstarlingT315271 db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash
 ResolvedPRODUCTION ERRORjcrespoT315274 Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active; a caller might have failed to call endAtomic() or cancelAtomic().
 ResolvedtstarlingT315995 Document how to disable x2 per DC
 ResolvedKrinkleT270225 Finish session storage to actually meet multi-DC requirements
 ResolvedEevansT222990 Audit session storage to determine max age of un-GC'd sessions
 ResolvedtstarlingT279664 Progressive Multi-DC roll out
 ResolvedLadsgroupT314486 Add support for multidc to auto_schema
 ResolvedtstarlingT314750 Clean up testwiki experiments (Aug 2022)
 ResolvedVgutierrezT315434 multi-dc.lua ATS script failing in production
 ResolvedtstarlingT134809 App servers <=> mariadb SSL/TLS for cross-datacenter writes
 ResolvedaaronT136218 Audit mysql database class and hhvm binding support of SSL
 ResolvedjcrespoT111654 Set up TLS for MariaDB replication
 ResolvedjcrespoT120122 Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
 Resolved• CmjohnsonT120689 es1019 and its management interface are unresponsive
 ResolvedGehelT128077 Create a PKI that can be used by Puppet and for general purpose certificates
 ResolvedjcrespoT151995 Rolling restart of external storage servers for TLS certificate update
 ResolvedjcrespoT152029 Rolling restart of parsercache servers for TLS certificate update
 ResolvedjcrespoT152188 Restart pending mysql hosts with old TLS cert
 Resolved• MarosteguiT152364 db1047 out of disk space, eventlogging_sync spam
 DuplicateNoneT152595 Implement TLS expiration/validation checking for MariaDB certificates
 ResolvedjcrespoT156005 Reimage db1065 and db1066
 ResolvedaaronT171071 Perform testing for TLS effect on connection rate
 ResolvedjcrespoT175672 Evaluate proxysql and native php-mysql client for TLS connection between Apache and mariadb
ResolvedtstarlingT196378 Investigate solutions for MySQL connection pooling
 ResolvedtstarlingT313578 Make OAuth work in Multi-DC active/active mode
 ResolvedaaronT320995 Create blog entry about multi-dc
 ResolvedKrinkleT329692 Story idea for Blog: Around the world: how we serve traffic from multiple data centers

Event Timeline

TheresNoTime created this task.Aug 16 2022, 4:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 16 2022, 4:26 AM
TheresNoTime added a parent task: T315271: db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash.Aug 16 2022, 4:27 AM
TheresNoTime updated the task description. (Show Details)Aug 16 2022, 4:33 AM
Dzahn subscribed.Aug 16 2022, 4:48 AM

04:46 < cwhite> Explicit transaction still active errors have stopped

TheresNoTime added a comment.EditedAug 16 2022, 4:50 AM

fwiw reporting user mentioned above is indeed now able to save their edit (perma)

colewhite closed this task as Resolved.Aug 16 2022, 4:53 AM
colewhite assigned this task to jcrespo.
colewhite subscribed.

Recovery appears to correlate with restoration of X2 primary db in eqiad. Optimistically resolving.

tstarling reopened this task as Open.Aug 16 2022, 6:09 AM
tstarling triaged this task as High priority.
tstarling added subscribers: aaron, tstarling.

Any kind of problem with x2 was supposed to cause graceful degradation in MediaWiki. The remaining issue is the fact that replication failure leads to an uncaught exception from approvePrimaryChanges().

tstarling added a project: Performance-Team.Aug 16 2022, 6:11 AM
RhinosF1 subscribed.Aug 16 2022, 7:52 AM
gerritbot added a comment.Aug 16 2022, 11:41 PM

Change 823763 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] objectcache: Add trace to SqlBagOStuff DBError logging

https://gerrit.wikimedia.org/r/823763

gerritbot added a project: Patch-For-Review.Aug 16 2022, 11:41 PM
gerritbot added a comment.Aug 17 2022, 2:50 AM

Change 823763 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Add trace to SqlBagOStuff DBError logging

https://gerrit.wikimedia.org/r/823763

gerritbot added a comment.Aug 17 2022, 2:55 AM

Change 823791 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] SqlBagOStuff: use cancelAtomic()

https://gerrit.wikimedia.org/r/823791

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.26; 2022-08-22).Aug 17 2022, 3:00 AM
gerritbot added a comment.Aug 17 2022, 4:49 PM

Change 823791 merged by jenkins-bot:

[mediawiki/core@master] SqlBagOStuff: use cancelAtomic()

https://gerrit.wikimedia.org/r/823791

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2022, 5:30 PM
jcrespo added a subscriber: lmata.Aug 18 2022, 12:05 PM

Preliminary incident report created at https://wikitech.wikimedia.org/wiki/Incidents/2022-08-16_x2_databases_replication_breakage CC @lmata

To concrete gaps in my knowledge that I left blank:

  • Who called Amir?
  • Detection- I wasn't around at that time (was it purely page-based, were user reports fundamental, was logstash/mw error rate what alerted SREs?

CC @colewhite

jcrespo added a project: Wikimedia-Incident.Aug 18 2022, 12:06 PM
lmata added a project: SRE-OnFire.Aug 18 2022, 1:54 PM

@jcrespo thank you for the report! looking forward to our review later, much appreciated.

jcrespo added a comment.Aug 18 2022, 2:43 PM

I checked operations logs and I have answers to my previous questions:

Who called Amir?

He was woken up by the manual page by @colewhite.

Detection- I wasn't around at that time (was it purely page-based, were user reports fundamental, was logstash/mw error rate what alerted SREs?

Icinga was taken seriously right away and procedure was followed. @TheresNoTime brought later user reports to the attention of SREs (which is helpful to have another way to verify user impact) but SREs were already working by then.

Updating report.

thcipriani moved this task from Untriaged to Aug 2022 on the Wikimedia-production-error board.Aug 18 2022, 3:58 PM
jcrespo moved this task from Backlog to Pending Review & Scorecard on the SRE-OnFire board.Aug 18 2022, 6:51 PM
gerritbot added a comment.Aug 19 2022, 1:07 AM

Change 824434 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@wmf/1.39.0-wmf.25] SqlBagOStuff: use cancelAtomic()

https://gerrit.wikimedia.org/r/824434

gerritbot added a project: Patch-For-Review.Aug 19 2022, 1:07 AM
gerritbot added a comment.Aug 19 2022, 1:24 AM

Change 824434 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.25] SqlBagOStuff: use cancelAtomic()

https://gerrit.wikimedia.org/r/824434

Stashbot added a comment.Aug 19 2022, 1:31 AM

Mentioned in SAL (#wikimedia-operations) [2022-08-19T01:30:59Z] <tstarling@deploy1002> Synchronized php-1.39.0-wmf.25/includes/libs/rdbms/database/DBConnRef.php: fix potential mainstash exception file 1 T315274 (duration: 03m 21s)

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2022, 1:31 AM
Stashbot added a comment.Aug 19 2022, 1:38 AM

Mentioned in SAL (#wikimedia-operations) [2022-08-19T01:37:59Z] <tstarling@deploy1002> Synchronized php-1.39.0-wmf.25/includes/objectcache/SqlBagOStuff.php: fix potential mainstash exception file 2 T315274 (duration: 03m 30s)

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.25; 2022-08-15); removed MW-1.39-notes (1.39.0-wmf.26; 2022-08-22).Aug 19 2022, 2:00 AM
tstarling closed this task as Resolved.Aug 19 2022, 2:03 AM

My reason for reopening this task is resolved. The patch has been deployed, so a similar failure in future should not lead to user-visible exceptions.

jcrespo mentioned this in T306118: Notify DBA prior to sending db traffic to x2.Aug 23 2022, 9:58 AM
tstarling mentioned this in T316847: Production test of x2 failure modes.Sep 1 2022, 1:04 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL