Page MenuHomePhabricator

Change table schema to cope with new BIGINT UNSIGNED ids
Closed, ResolvedPublic

Description

Let's do this work jointly with T366542:

While testing MR 46, I suddenly started having a new mismatched schema failure :

error: instance type (number) does not match any allowed primitive type (allowed: ["integer"])
    level: "error"
    schema: {"loadingURI":"#","pointer":"/properties/revision/properties/rev_id"}
    instance: {"pointer":"/revision/rev_id"}
    domain: "validation"
    keyword: "type"
    found: "number"
    expected: ["integer"]
error: instance type (number) does not match any allowed primitive type (allowed: ["integer"])
    level: "error"
    schema: {"loadingURI":"#","pointer":"/properties/revision/properties/rev_parent_id"}
    instance: {"pointer":"/revision/rev_parent_id"}
    domain: "validation"
    keyword: "type"
    found: "number"
    expected: ["integer"]

Which is weird, because I had not hit that before. I chased it down in Spark, and found that it was coming from the analytic replicas, and in Spark the schema coming from our replica queries now looks like so:

root
 |-- page_id: long (nullable = true)
 |-- page_namespace: integer (nullable = true)
 |-- page_title: string (nullable = true)
 |-- user_id: decimal(20,0) (nullable = true)        <----------
 |-- user_text: string (nullable = true)
 |-- user_is_visible: integer (nullable = true)
 |-- revision_id: decimal(20,0) (nullable = true)        <----------
 |-- revision_parent_id: decimal(20,0) (nullable = true)        <----------
 |-- mw_revision_timestamp: binary (nullable = true)
 |-- revision_is_minor_edit: integer (nullable = true)
 |-- revision_comment: string (nullable = true)
 |-- revision_comment_is_visible: integer (nullable = true)
 |-- revision_sha1: string (nullable = true)
 |-- revision_size: long (nullable = true)
 |-- slot_role_name: string (nullable = true)
 |-- slot_content_model: string (nullable = true)
 |-- slot_content_sha1: string (nullable = true)
 |-- slot_content_size: long (nullable = true)
 |-- revision_content_is_visible: integer (nullable = true)

None of the highligthed columns used to be decimal(20,0); they were longs, just like page_id.

It turns out that my test wiki, itwiki, was being migrated as part of T367856: Cleanup revision table schema. TL;DR here is that these ALTERs are being run on all wikis:

ALTER TABLE /*_*/revision
  CHANGE rev_id rev_id BIGINT UNSIGNED AUTO_INCREMENT NOT NULL,
  CHANGE rev_comment_id rev_comment_id BIGINT UNSIGNED NOT NULL,
  CHANGE rev_actor rev_actor BIGINT UNSIGNED NOT NULL,
  CHANGE rev_parent_id rev_parent_id BIGINT UNSIGNED DEFAULT NULL;

A BIGINT UNSIGNED translates in Spark to a DECIMAL(20,0), and in Iceberg there is no support for BIGINT UNSIGNED, but there is support for DECIMAL(20,0). Therefore, it looks like we will need to change our schema for all these ids.

Event Timeline

the page_change schema will eventually have to change as well as the ids are currrently set like so:

rev_id:
  description: The (database) revision ID.
  type: integer
  maximum: 9007199254740991
  minimum: 1

That maximum is now outdated. In fact, it was already outdated:

    9007199254740991      <<<<<< declared maximum on page_change
 9223372036854775807      <<<<<< declared maximum of MariaDB BIGINT
18446744073709551615      <<<<<< declared maximum of MariaDB UNSIGNED BIGINT

MariaDB BIGINT docs.

CC @Ottomata, @gmodena

Hm,

In our JSONSchema converter, we convert integer types to DataTypes.LongType

Spark's LongType is min -9223372036854775808 and max 9223372036854775807. No UNSIGNED support it seems.

9007199254740991      <<<<<< declared maximum on page_change

The 9007199254740991 value comes from Number.MAX_SAFE_INTEGER and is added to the jsonschema maximum via the jsonschema-tools enforcedNumericBounds config

IIUC, while it is possible to represent numbers larger than MAX_SAFE_INTEGER in JSON, they may lose precision while (de)serializing.

> JSON.stringify(9223372036854775807)
'9223372036854776000'

> JSON.stringify(18446744073709551615)
'18446744073709552000'

It will probably not be wise for us to support integers larger than MAX_SAFE_INTEGER :)

But, note that MAX_SAFE_INTEGER is way larger than max UNSIGNED INT (the prior MariaDB rev_id max).

MAX_SAFE_INTEGER: 9007199254740991
UNSIGNED INT max:       4294967295

I'm not sure how fast rev_id grows, buuuut I think it is unlikely that we will encounter a problem anytime soon ;)

A BIGINT UNSIGNED translates in Spark to a DECIMAL(20,0), and in Iceberg there is no support for BIGINT UNSIGNED, but there is support for DECIMAL(20,0). Therefore, it looks like we will need to change our schema for all these ids.

Instead of using a DECIMAL(20,0), could we just use BIGINT / long type? This is what other tables, e.g. event.mediawiki_page_change.v1 will have for rev_id fields.

Sure, max BIGINT is smaller than max UNSIGNED BIGINT, but max BIGINT is bigger than JavaScript Number.MAX_SAFE_INTEGER, so if we did ever encounter rev_ids this large, we will have to deal with it in JSON world before we deal with it in Hive/Spark/Iceberg world ;)

Instead of using a DECIMAL(20,0), could we just use BIGINT / long type?

BIGINT is what we currently have in the datalake table schema.

But I am now explicitly casting the incoming DECIMAL(20,0) down to a BIGINT via CAST(rev_id AS UNSIGNED). This CAST happens at the MariaDB level.

I understand there is a lot of runway for this, as the max rev_id for, say, wikidatawiki, is:

MariaDB [wikidatawiki]> select max(rev_id) from revision;
+-------------+
| max(rev_id) |
+-------------+
|  2274147444 |
+-------------+
1 row in set (0.001 sec)

But I wanted to open this task to track, as it is a schema change that is now not reflected at the datalake table nor the jsonschema. A future (AI?) maintainer would probably want to know the historical context of the bug they're trying to fix. :D

but max BIGINT is bigger than JavaScript Number.MAX_SAFE_INTEGER

We can always just keep a STRING.

We can always just keep a STRING.

🤮

It looks like JavaScript BigInt objects may be useful for this problem in the future: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt.

And yes,BigInts eventually serialize to JSON as a STRING. Don't shoot the messenger; I didn't invent JavaScript. A 5 year old did.

xcollazo claimed this task.

Nope, we are done here.

For a future reader: the decision was to document this (T379703#10340808), but not to change code.