Let's do this work jointly with T366542:
In T368755#10315174, @xcollazo wrote:While testing MR 46, I suddenly started having a new mismatched schema failure :
error: instance type (number) does not match any allowed primitive type (allowed: ["integer"]) level: "error" schema: {"loadingURI":"#","pointer":"/properties/revision/properties/rev_id"} instance: {"pointer":"/revision/rev_id"} domain: "validation" keyword: "type" found: "number" expected: ["integer"] error: instance type (number) does not match any allowed primitive type (allowed: ["integer"]) level: "error" schema: {"loadingURI":"#","pointer":"/properties/revision/properties/rev_parent_id"} instance: {"pointer":"/revision/rev_parent_id"} domain: "validation" keyword: "type" found: "number" expected: ["integer"]Which is weird, because I had not hit that before. I chased it down in Spark, and found that it was coming from the analytic replicas, and in Spark the schema coming from our replica queries now looks like so:
root |-- page_id: long (nullable = true) |-- page_namespace: integer (nullable = true) |-- page_title: string (nullable = true) |-- user_id: decimal(20,0) (nullable = true) <---------- |-- user_text: string (nullable = true) |-- user_is_visible: integer (nullable = true) |-- revision_id: decimal(20,0) (nullable = true) <---------- |-- revision_parent_id: decimal(20,0) (nullable = true) <---------- |-- mw_revision_timestamp: binary (nullable = true) |-- revision_is_minor_edit: integer (nullable = true) |-- revision_comment: string (nullable = true) |-- revision_comment_is_visible: integer (nullable = true) |-- revision_sha1: string (nullable = true) |-- revision_size: long (nullable = true) |-- slot_role_name: string (nullable = true) |-- slot_content_model: string (nullable = true) |-- slot_content_sha1: string (nullable = true) |-- slot_content_size: long (nullable = true) |-- revision_content_is_visible: integer (nullable = true)None of the highligthed columns used to be decimal(20,0); they were longs, just like page_id.
It turns out that my test wiki, itwiki, was being migrated as part of T367856: Cleanup revision table schema. TL;DR here is that these ALTERs are being run on all wikis:
ALTER TABLE /*_*/revision CHANGE rev_id rev_id BIGINT UNSIGNED AUTO_INCREMENT NOT NULL, CHANGE rev_comment_id rev_comment_id BIGINT UNSIGNED NOT NULL, CHANGE rev_actor rev_actor BIGINT UNSIGNED NOT NULL, CHANGE rev_parent_id rev_parent_id BIGINT UNSIGNED DEFAULT NULL;A BIGINT UNSIGNED translates in Spark to a DECIMAL(20,0), and in Iceberg there is no support for BIGINT UNSIGNED, but there is support for DECIMAL(20,0). Therefore, it looks like we will need to change our schema for all these ids.