Page MenuHomePhabricator

Catalyst the first edit using each tag is saved without its tag
Closed, ResolvedPublic2 Estimated Story PointsBUG REPORT

Description

This was discovered while working on getting the RevisionSlider extension selenium tests on Catalyst. RevisionSlider has a test that passes every time on retry, but fails on the initial run each time. This is because the tag is not correctly applied the first time it's used on a new wiki. On retries, the tag isn't new any more, so it works.

Steps to replicate the issue (include links if applicable):

  1. Create a new Catalyst env (for example the mediawiki chart with Vector, like the catalyst-daily jobs) and log in as admin
  2. Test A, the wiki's first blanking:
    • a. Open Test_A?action=edit, type one, and save.
    • b. Edit it again, change the text to two, and save.
    • c. Edit it again, delete all the text, and save. The page is now blank.
  3. Test B, the second blanking, do the same three edits on Test_B?action=edit.

What happens?:
The first edit using each tag is saved, but without adding its tag

Also, Special:RecentChanges does not show the blanking of the first page

What should have happened instead?:
The tag should display the first time, as well as all subsequent times that it should be applied.
Also, the edit should appear in Special:RecentChanges.

Other information (browser name/version, screenshots, etc.):

gif of testTest A, no blanking tagTest B showing the blanking tagFirst blanking not showing in Special:RecentChanges
Screen Recording 2026-09-30 at 2.18.25 PM.gif (589×575 px, 2 MB)
Screenshot 2026-09-30 at 2.49.11 PM.png (2,604×1,180 px, 301 KB)
Screenshot 2026-09-30 at 2.47.24 PM.png (2,084×1,088 px, 293 KB)
Screenshot 2026-09-30 at 2.44.32 PM.png (2,506×1,900 px, 499 KB)

Likely cause:
envdb (MariaDB 11.8.2) has innodb_snapshot_isolation on by default. With it on, MediaWiki loses the tag and the Recent changes entry for the first edit that uses each tag. The edit itself is saved. This is why RevisionSlider's daily test fails on its first try. Link to mariaDB doc that shows defaults

Production (MariaDB 10.11.16) and Jenkins (search log for 10.11.18-MariaDB) run older versions where it is off by default, so they don't hit this.

To confirm on catalyst: run SELECT @@global.innodb_snapshot_isolation; on envdb (1 = on).

Potential fix, if the above is confirmed to be true: add innodb_snapshot_isolation=OFF to myCnf in catalyst-api helm/env-db/values.yaml.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
envDB: set `innodb_snapshot_isolation=off`repos/test-platform/catalyst/catalyst-api!219jnucheT439780main
mariaDB: set `innodb_snapshot_isolation=off` in standalone DBrepos/test-platform/catalyst/ci-charts!163jnucheT439780main
mariaDB: set `innodb_snapshot_isolation=off`repos/abstract-wiki/aw-ci-chart!60jnucheT439780main
Customize query in GitLab

Event Timeline

vaughnwalters updated the task description. (Show Details)
vaughnwalters updated the task description. (Show Details)

One additional note, this is an issue only in Catalyst now because it is the only (as far as I can see?) env running 11.6.2 or later, which is when this behavior became default. The catalyst env I tested this on was on 11.8.2-MariaDB. BUT, as we upgrade jenkins, betacluster, and prod to anything 11.6.2 or later then this will likely also present in those envs.

This appears to be raising a valid issue, so probably it needs fixing as opposed to disabling the behaviour that causes the issue? I would have thought MediaWiki should support innodb_snapshot_isolation being enabled (even if production does not / will not do this)

This appears to be raising a valid issue, so probably it needs fixing as opposed to disabling the behaviour that causes the issue? I would have thought MediaWiki should support innodb_snapshot_isolation being enabled (even if production does not / will not do this)

Ah... see this ticket too T388337. Maybe short term fix here is to disable the behavior though, so test env shows what production would do.

I could reproduce the issue locally, innodb_snapshot_isolation is definitely the trigger:

  • v11.5.2: Does not reproduce
  • v11.6.2: Reproduces
  • v11.6.2 + innodb_snapshot_isolation=off: Does not reproduce

After reading about the issue that innodb_snapshot_isolation is trying to address, it seems MW must be failing to properly lock/serialize some of its writes — I'd be curious to know which concrete transactions are doing that. At any rate, T388337 already tracks the MW side of things, so I think it makes sense for us to restore the previous behavior in Catalyst.

Form our side, I've turned off the behavior in all relevant locations where a Catalyst environment uses a MariaDB.

I'm also still curious about what was the smoking gun that pointed to innodb_snapshot_isolation as the trigger. @vaughnwalters was it the Error 1020 that you were seeing in the logs or something else?

I'm also still curious about what was the smoking gun that pointed to innodb_snapshot_isolation as the trigger. @vaughnwalters was it the Error 1020 that you were seeing in the logs or something else?

Yep, exactly, it was the Error 1020. But I couldn't verify it a prod env and wasn't sure what we would want to do about it either way. So ... I guess mariaDB is not a drop in replacement for MySQL then now?

I'm also still curious about what was the smoking gun that pointed to innodb_snapshot_isolation as the trigger. @vaughnwalters was it the Error 1020 that you were seeing in the logs or something else?

Yep, exactly, it was the Error 1020. But I couldn't verify it a prod env and wasn't sure what we would want to do about it either way. So ... I guess mariaDB is not a drop in replacement for MySQL then now?

Yeah, that actually got me thinking because MariaDB forked from MySQL more than a decade ago and it would be surprising if this is the first change they introduce that breaks that guarantee.

And sure enough, MariaDB hasn't been a drop-in replacement for a while now 😅 https://mariadb.com/docs/release-notes/community-server/about/compatibility-and-differences/mariadb-vs-mysql-compatibility

jnuche moved this task from In progress to Done on the Catalyst (Luka Ijo Pimeja Jan) board.

Verified that the issue does not reproduce anymore in prod for Patchdemo and CI