Page MenuHomePhabricator

"chemical" major mime type was never added to production database
Closed, ResolvedPublic

Description

It seems T68412: Updater applies patches after fresh install of MW where tables.sql already has been patched (2014) was never applied to production. For example, filearchive in enwiki is:

|  fa_major_mime        | enum('unknown','application','audio','image','text','video','message','model','multipart')

While locally it's:

fa_major_mime        | enum('unknown','application','audio','image','text','video','message','model','multipart','chemical')

This has caused massive drift reports:

[
    "filearchive fa_major_mime field-size-mismatch": {
        "s1": [
            "db1083:enwiki",
            "db1084:enwiki",
            "db1169:enwiki",
            "db1118:enwiki",
            "db1163:enwiki",
            "db1106:enwiki",
            "db1164:enwiki",
            "db1119:enwiki",
            "db1134:enwiki"
        ],
        "s2": [
            "db1122:svwiki",
            "db1129:svwiki",
            "db1076:svwiki",
            "db1074:svwiki"
        ],
        "s3": [
            "db1123:aswikibooks",
            "db1166:aswikibooks",
            "db1157:aswikibooks",
            "db1175:aswikibooks",
            "db1112:aswikibooks"
        ],
        "s6": [
            "db1131:ruwiki",
            "db1168:ruwiki",
            "db1085:ruwiki",
            "db1173:ruwiki"
        ],
        "s7": [
            "db1086:cawiki",
            "db1127:cawiki",
            "db1136:cawiki",
            "db1079:cawiki"
        ],
        "s8": [
            "db1104:wikidatawiki",
            "db1172:wikidatawiki",
            "db1109:wikidatawiki",
            "db1114:wikidatawiki",
            "db1126:wikidatawiki",
            "db1111:wikidatawiki",
            "db1087:wikidatawiki"
        ]
    },
    "image img_major_mime field-size-mismatch": {
        "s1": [
            "db1083:enwiki",
            "db1084:enwiki",
            "db1169:enwiki",
            "db1118:enwiki",
            "db1163:enwiki",
            "db1106:enwiki",
            "db1164:enwiki",
            "db1119:enwiki",
            "db1134:enwiki",
            "db1135:enwiki"
        ],
        "s2": [
            "db1122:svwiki",
            "db1129:svwiki",
            "db1076:svwiki",
            "db1074:svwiki"
        ],
        "s3": [
            "db1123:aswikibooks",
            "db1166:aswikibooks",
            "db1157:aswikibooks",
            "db1175:aswikibooks",
            "db1112:aswikibooks"
        ],
        "s6": [
            "db1131:ruwiki",
            "db1168:ruwiki",
            "db1085:ruwiki",
            "db1173:ruwiki"
        ],
        "s7": [
            "db1086:cawiki",
            "db1174:cawiki",
            "db1127:cawiki",
            "db1136:cawiki",
            "db1079:cawiki"
        ],
        "s8": [
            "db1104:wikidatawiki",
            "db1172:wikidatawiki",
            "db1109:wikidatawiki",
            "db1114:wikidatawiki",
            "db1126:wikidatawiki",
            "db1111:wikidatawiki",
            "db1087:wikidatawiki"
        ]
    },
    "oldimage oi_major_mime field-size-mismatch": {
        "s1": [
            "db1083:enwiki",
            "db1084:enwiki",
            "db1169:enwiki",
            "db1118:enwiki",
            "db1163:enwiki",
            "db1106:enwiki",
            "db1164:enwiki",
            "db1119:enwiki",
            "db1134:enwiki",
            "db1135:enwiki"
        ],
        "s2": [
            "db1122:svwiki",
            "db1129:svwiki",
            "db1076:svwiki",
            "db1074:svwiki"
        ],
        "s3": [
            "db1123:aswikibooks",
            "db1166:aswikibooks",
            "db1157:aswikibooks",
            "db1175:aswikibooks",
            "db1112:aswikibooks"
        ],
        "s6": [
            "db1131:ruwiki",
            "db1168:ruwiki",
            "db1085:ruwiki",
            "db1173:ruwiki"
        ],
        "s7": [
            "db1086:cawiki",
            "db1174:cawiki",
            "db1127:cawiki",
            "db1136:cawiki",
            "db1079:cawiki"
        ],
        "s8": [
            "db1104:wikidatawiki",
            "db1172:wikidatawiki",
            "db1109:wikidatawiki",
            "db1114:wikidatawiki",
            "db1126:wikidatawiki",
            "db1087:wikidatawiki"
        ]
    },
]

I assume wikis created after 2014 wouldn't have this drift. The project for this (T18491: Support for Chemical Markup Language) is stalled so no rush in fixing this but I hope it gets picked up again. It's a cool project :(

The patches for it can be found in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/135756/13/maintenance/archives/patch-img_major_mime-chemical.sql (and onwards) but I'm not 100% sure it would be work after all these years.

Also another reason why we should avoid using enums (T119173: RFC: Discourage use of MySQL's ENUM type)

ALTERs needed:

ALTER TABLE filearchive  CHANGE  fa_major_mime fa_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE image  CHANGE  img_major_mime img_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE oldimage CHANGE  oi_major_mime oi_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';

Needed on:

  • s4
    • eqiad
    • codfw
  • s5
    • eqiad T277354#7543202
    • codfw
      • cebwiki
      • dewiki
      • enwikivoyage
      • mgwiktionary
      • mhwiktionary
      • muswiki
      • shwiki
      • srwiki
  • s6 T277354#7532985
    • eqiad
    • codfw
      • frwiki
      • jawiki
      • ruwiki

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2021-12-14T11:36:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T277354)', diff saved to https://phabricator.wikimedia.org/P18216 and previous config saved to /var/cache/conftool/dbconfig/20211214-113615-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T12:21:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T277354)', diff saved to https://phabricator.wikimedia.org/P18219 and previous config saved to /var/cache/conftool/dbconfig/20211214-122129-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T12:21:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18220 and previous config saved to /var/cache/conftool/dbconfig/20211214-122137-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T12:23:31Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18221 and previous config saved to /var/cache/conftool/dbconfig/20211214-122331-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T13:08:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18224 and previous config saved to /var/cache/conftool/dbconfig/20211214-130845-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T13:08:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18225 and previous config saved to /var/cache/conftool/dbconfig/20211214-130853-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T13:10:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18226 and previous config saved to /var/cache/conftool/dbconfig/20211214-131047-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2021-12-14T13:56:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T277354)', diff saved to https://phabricator.wikimedia.org/P18229 and previous config saved to /var/cache/conftool/dbconfig/20211214-135601-marostegui.json

End of schema change on db2103,dbstore1003:3311,db1184,db1169,db1164,db1163,db1140:3311,db1139:3311,db1135,db1134,db1133,db1128,db1119,db1106,db1105:3311,db1099:3311

Marostegui updated the task description. (Show Details)

This is all done! It was very good to have this schema change to polish the script to automate schema changes.
s3 and s1 were done entirely with the script (again, not the primary masters).

Ran the drift reports, this showed up again. It seems s4 also needs this.

Interesting, I thought it was not needed there.

ALTERs needed:

ALTER TABLE filearchive CHANGE fa_major_mime fa_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE image CHANGE img_major_mime img_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE oldimage CHANGE oi_major_mime oi_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';

This also removes the NOT NULL from img_major_mime and oi_major_mime which is in code and results in schema drift (fa_major_mime is nullable in code)

Nullability in an ENUM that has a default value should be moot but I will check when the holidays is over.

Running the alter on s4 codfw master.

ALTERs needed:

ALTER TABLE filearchive CHANGE fa_major_mime fa_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE image CHANGE img_major_mime img_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';
ALTER TABLE oldimage CHANGE oi_major_mime oi_major_mime ENUM('unknown','application','audio','image','text','video','message','model','multipart','chemical') default 'unknown';

This also removes the NOT NULL from img_major_mime and oi_major_mime which is in code and results in schema drift (fa_major_mime is nullable in code)

And: T277354#7532988

Running the alter on s4 eqiad hosts

Mentioned in SAL (#wikimedia-operations) [2022-01-03T09:30:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18269 and previous config saved to /var/cache/conftool/dbconfig/20220103-093003-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T11:14:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18275 and previous config saved to /var/cache/conftool/dbconfig/20220103-111457-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T12:00:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18284 and previous config saved to /var/cache/conftool/dbconfig/20220103-120011-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T13:42:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18289 and previous config saved to /var/cache/conftool/dbconfig/20220103-134227-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T15:27:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18299 and previous config saved to /var/cache/conftool/dbconfig/20220103-152710-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T16:12:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18303 and previous config saved to /var/cache/conftool/dbconfig/20220103-161224-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T16:12:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18304 and previous config saved to /var/cache/conftool/dbconfig/20220103-161232-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T18:07:43Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18311 and previous config saved to /var/cache/conftool/dbconfig/20220103-180743-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T18:52:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18317 and previous config saved to /var/cache/conftool/dbconfig/20220103-185257-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T18:53:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18318 and previous config saved to /var/cache/conftool/dbconfig/20220103-185305-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T20:36:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18319 and previous config saved to /var/cache/conftool/dbconfig/20220103-203654-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T21:22:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18322 and previous config saved to /var/cache/conftool/dbconfig/20220103-212209-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T21:22:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18323 and previous config saved to /var/cache/conftool/dbconfig/20220103-212216-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-03T23:24:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18324 and previous config saved to /var/cache/conftool/dbconfig/20220103-232433-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T00:09:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18327 and previous config saved to /var/cache/conftool/dbconfig/20220104-000947-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T01:51:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18332 and previous config saved to /var/cache/conftool/dbconfig/20220104-015125-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T03:35:55Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18333 and previous config saved to /var/cache/conftool/dbconfig/20220104-033555-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T04:21:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18337 and previous config saved to /var/cache/conftool/dbconfig/20220104-042109-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T04:21:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18338 and previous config saved to /var/cache/conftool/dbconfig/20220104-042116-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T07:37:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18343 and previous config saved to /var/cache/conftool/dbconfig/20220104-073745-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T08:22:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18350 and previous config saved to /var/cache/conftool/dbconfig/20220104-082259-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T08:23:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18351 and previous config saved to /var/cache/conftool/dbconfig/20220104-082306-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T10:14:01Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18359 and previous config saved to /var/cache/conftool/dbconfig/20220104-101400-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T10:59:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18365 and previous config saved to /var/cache/conftool/dbconfig/20220104-105914-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T10:59:22Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18366 and previous config saved to /var/cache/conftool/dbconfig/20220104-105922-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T12:58:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18377 and previous config saved to /var/cache/conftool/dbconfig/20220104-125845-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T13:43:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18381 and previous config saved to /var/cache/conftool/dbconfig/20220104-134359-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T13:44:10Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18382 and previous config saved to /var/cache/conftool/dbconfig/20220104-134410-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T15:24:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18384 and previous config saved to /var/cache/conftool/dbconfig/20220104-152416-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2022-01-04T16:09:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18388 and previous config saved to /var/cache/conftool/dbconfig/20220104-160930-marostegui.json

s4 slaves are done (clouddb* hosts are still running the change) - tomorrow I will go for the primary master.

Change 742423 merged by jenkins-bot:

[mediawiki/core@master] installer: Set \"unknown\" for default value of *_major_mime in MySQL

https://gerrit.wikimedia.org/r/742423

Marostegui updated the task description. (Show Details)

s4 master done

Interesting, I thought it was not needed there.