Page MenuHomePhabricator

SQLite's flexible typing causing incompatibility issues
Open, Needs TriagePublic

Description

So, first of all, I'm by no means an expert of SQLite, and I just learned about SQLite's flexible typing.

A few days ago I noticed that this AbuseFilter test was failing on SQLite (see e.g. this console output), whereas it used to pass on MySQL.

It took me a decent amount of time (and tests using the CI) to find out the cause. The abuse_filter_log table definition for SQLite has:

CREATE TABLE /*$wgDBprefix*/abuse_filter_log (
	...
	afl_filter varbinary(64) NOT NULL DEFAULT '',

the test does

INSERT INTO abuse_filter_log ( ... afl_filter ... ) values ( ... 1 ...

i.e. it's inserting a numeric 1. Then we do something like

SELECT  1  FROM unittest_abuse_filter_log WHERE afl_filter = '1'

and this cannot find the row inserted above! Instead, it works as expected if you remove the quotes from '1'.


The docs above don't explicitly mention what happens if you try to insert an integer in a varbinary column, but apparently, no cast is taking place (unlike when inserting strings in an integer field). Moreover, SELECT queries are not flexibly typed, which explains why the test was failing. Needles to say, this is a big drift from MySQL and a potential source of sneaky issues.

I don't know whether this has already caused problems in the past, and/or it's already being mitigated somehow, but I think it's very important that we do something about that, even just putting a warning on mw.org.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
eprodromou subscribed.

Putting a warning on Mediawiki.org seems like the right thing to do here. I'm untagging CPT, but tag us if you think there's more work to be done on our side.

See T255390 for a couple actual examples; there might be more, any row using the BLOB data type (at least, but lots of columns use that) is potentially affected. This feature can actually cause data corruption (sort of), per T255390#6223722:

And this is why the issue with flexible typing is (IMHO) worrying: a single, bad INSERT will require all future SELECTs to account for both possibilities.

You can use the wrong type in INSERTs, but SELECTs must match the exact type stored in the row. So, for instance:

CREATE TABLE foo(
  id integer primary key autoincrement,
  bar BLOB
  );
  
INSERT INTO foo VALUES (null,42);
INSERT INTO foo VALUES (null,'42');
SELECT * FROM foo
--> row 1 and 2
SELECT * FROM foo WHERE bar = 42;
--> Only row 1 is returned
SELECT * FROM foo WHERE bar = '42';
--> Only row 2 is returned
Daimona renamed this task from SQLite's flexible typing causing incompatibility issues (at least in WMF CI) to SQLite's flexible typing causing incompatibility issues.Aug 17 2022, 5:24 PM