Page MenuHomePhabricator
Create Task
Maniphest T313251

Collect and fix schema drifts of wmf-deployed extension tables in production
Open, Needs TriagePublic
Actions

Description

We are mostly done with core drifts, now it's time to take a look at extensions.

This is now possible with T261912: Convert WMF Deployed Extensions to Abstract Schema being done.

You just need to add paths to extension schema files in https://github.com/Ladsgroup/db-analyzor-tools/blob/master/abstract_paths.json and in valid_categories: https://github.com/Ladsgroup/db-analyzor-tools/blob/master/drift-tracker/app.py and then one run would add the list.

  • AbuseFilter T381523
  • AntiSpoof T381532
  • Babel
  • BetaFeatures
  • BounceHandler
  • CentralAuth
  • CentralNotice
  • CheckUser T321063
  • Cognate
  • ContentTranslation
  • CreditsSource T326156
  • DiscussionTools
  • Echo
  • EntitySchema
  • FlaggedRevs T313253
  • Flow
  • GeoData
  • GlobalBlocking T381502
  • GlobalPreferences
  • GlobalUsage
  • GrowthExperiments
  • LdapAuthentication
  • Linter
  • LiquidThreads
  • MachineVision
  • Math
  • Newsletter
  • OATHAuth
  • OAuth
  • OAuthRateLimiter
  • ORES
  • PageAssessments
  • PageTriage
  • ProofreadPage
  • PropertySuggester
  • ReadingLists
  • SecurePoll T381536
  • ShortUrl
  • TimedMediaHandler
  • Translate
  • UploadWizard
  • UrlShortener
  • WikiLove
  • Wikibase
  • WikibaseQualityConstraints
  • WikimediaEditorTasks

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT313251 Collect and fix schema drifts of wmf-deployed extension tables in production
 ResolvedLadsgroupT313253 Fix Flaggedrevs database schema drifts in production
 ResolvedLadsgroupT318950 Fix renamed indexes of flaggedrevs table in production
 ResolvedLadsgroupT318955 Drop fr_comment and fr_text from flaggedrevs table in production
 ResolvedLadsgroupT318959 Add fr_user index on flaggedrevs in production
 ResolvedLadsgroupT322618 Fix renamed indexes of flaggedrevs_tracking table in production
 ResolvedLadsgroupT323214 Fix unsigned drifts in flaggedrevs caused by 4c0b3c7b9b0
 ResolvedLadsgroupT323827 Finish timestamp schema changes in flaggedrevs
 ResolvedLadsgroupT323907 Make fr_user unsigned
 ResolvedLadsgroupT333918 Switchover s2 master (db1162 -> db1122)
 ResolvedLadsgroupT251191 FlaggedRevs has lots of database drifts but only in s1 and s5
 ResolvedLadsgroupT321063 Fix CheckUser database schema drifts in production
 ResolvedMarosteguiT321123 Drop old index cuc_user_time on cu_changes table for wmf wikis
 ResolvedMarosteguiT321127 Add index cuc_actor_ip_time to cu_changes on wmf wikis
 ResolvedMarosteguiT321126 Add column 'cul_actor' and index cul_actor_time to cu_log on wmf wikis
 ResolvedMarosteguiT321130 Add column cuc_private to cu_changes on wmf wikis
 ResolvedPapaulT322988 db2173 HW errors
 ResolvedMarosteguiT343174 Add missing column cuc_only_for_read_old to testcommonswiki
 ResolvedMarosteguiT343175 Remove old fields 'cuc_user' and 'cuc_user_text' as well as index 'cuc_user_ip_time' from a few production wikis
 ResolvedLadsgroupT356631 Column cu_log.cul_timestamp has an incorrect type and incorrectly has a default value set
 ResolvedMarosteguiT356634 Column cu_log.cul_type incorrectly has a default value set on s1
 ResolvedLadsgroupT326156 Fix CreditsSource drifts
 ResolvedLadsgroupT326662 Drop unused tables of CreditsSource
 OpenNoneT381502 Fix GlobalBlocking database schema drifts in production
 ResolvedDreamy_JazzT381517 Older wikis with GlobalBlocking installed have an incorrect type for gbw_expiry
 ResolvedLadsgroupT381521 Update type of gbw_expiry on the global_block_whitelist table
 OpenNoneT381877 Older wikis with GlobalBlocking installed have an incorrect type for gb_id
 OpenNoneT381523 Fix AbuseFilter database schema drifts in production
 OpenNoneT381532 Fix AntiSpoof database schema drifts in production
 ResolvedLadsgroupT381536 Fix SecurePoll database schema drifts in production

Event Timeline

Ladsgroup created this task.Jul 18 2022, 6:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2022, 6:19 PM
Ladsgroup moved this task from Inbox to Epic - Database on the Data-Persistence board.Jul 18 2022, 6:19 PM
Ladsgroup updated the task description. (Show Details)Jul 18 2022, 6:24 PM
Umherirrender updated the task description. (Show Details)Jul 18 2022, 8:01 PM
Daimona subscribed.Jul 19 2022, 12:21 AM
Umherirrender added a comment.Jul 29 2022, 5:08 PM

First collection with the abstract schema file names (as of now) and some dblist entries

Needs revisit on which databases the check needs to run. Not sure how to tag the tables only exists once (central) and some other special handling for databases (like x1)

I am not sure how big the overhead for new dblists is, but extensions not exists on all wiki possible needs its own list to make the check easier in the drift tracker (like Translate, ContentTranslation, LiquidThreads, WikiLove, ReadingLists, PageTriage, etc.)

"abusefilter": {
    "path": [
        "mediawiki/extensions/AbuseFilter/+/master/db_patches/tables.json"
    ]
},
"antispoof": {
    "path": [
        "mediawiki/extensions/AntiSpoof/+/master/sql/tables.json"
    ]
},
"babel": {
    "path": [
        "mediawiki/extensions/Babel/+/master/sql/tables.json"
    ]
},
"betafeatures": {
    "path": [
        "mediawiki/extensions/BetaFeatures/+/master/sql/tables.json"
    ]
},
"bouncehandler": {
    "path": [
        "mediawiki/extensions/BounceHandler/+/master/sql/tables.json"
    ]
},
"centralauth": {
    "path": [
        "mediawiki/extensions/CentralAuth/+/master/schema/tables.json"
    ]
},
"centralnotice": {
    "path": [
        "mediawiki/extensions/CentralNotice/+/master/sql/tables.json"
    ]
},
"checkuser": {
    "path": [
        "mediawiki/extensions/CheckUser/+/master/schema/tables.json"
    ]
},
"cognate": {
    "path": [
        "mediawiki/extensions/Cognate/+/master/sql/tables.json"
    ],
    "dblist": "wiktionary"
},
"contenttranslation": {
    "path": [
        "mediawiki/extensions/ContentTranslation/+/master/sql/tables.json"
    ]
},
"creditssource": {
    "path": [
        "mediawiki/extensions/CreditsSource/+/master/schema/tables.json"
    ],
    "dblist": "wikivoyage"
},
"discussiontools": {
    "path": [
        "mediawiki/extensions/DiscussionTools/+/master/sql/discussiontools_subscription.json"
    ]
},
"echo": {
    "path": [
        "mediawiki/extensions/Echo/+/master/sql/tables.json"
    ],
    "dblist": "echo"
},
"echo_central": {
    "path": [
        "mediawiki/extensions/Echo/+/master/sql/tables-sharedtracking.json"
    ]
},
"entityschema": {
    "path": [
        "mediawiki/extensions/EntitySchema/+/master/sql/tables.json"
    ],
    "dblist": "wikibaserepo"
},
"flow": {
    "path": [
        "mediawiki/extensions/Flow/+/master/sql/tables.json"
    ],
    "dblist": "flow"
},
"geodata": {
    "path": [
        "mediawiki/extensions/GeoData/+/master/sql/tables.json"
    ]
},
"globalblocking": {
    "path": [
        "mediawiki/extensions/GlobalBlocking/+/master/sql/tables-global_block_whitelist.json"
    ]
},
"globalblocking_central": {
    "path": [
        "mediawiki/extensions/GlobalBlocking/+/master/sql/tables-globalblocks.json"
    ]
},
"globalpreferences": {
    "path": [
        "mediawiki/extensions/GlobalPreferences/+/master/sql/tables.json"
    ]
},
"globalusage": {
    "path": [
        "mediawiki/extensions/GlobalUsage/+/master/sql/tables.json"
    ]
},
"growthexperiments": {
    "path": [
        "mediawiki/extensions/GrowthExperiments/+/master/sql/growthexperiments_link_recommendations.json",
        "mediawiki/extensions/GrowthExperiments/+/master/sql/growthexperiments_link_submissions.json",
        "mediawiki/extensions/GrowthExperiments/+/master/sql/growthexperiments_mentee_data.json",
        "mediawiki/extensions/GrowthExperiments/+/master/sql/growthexperiments_mentor_mentee.json"
    ],
    "dblist": "growthexperiments"
},
"ldapauthentication": {
    "path": [
        "mediawiki/extensions/LdapAuthentication/+/master/schema/tables.json"
    ]
},
"linter": {
    "path": [
        "mediawiki/extensions/Linter/+/master/sql/tables.json"
    ]
},
"liquidthreads": {
    "path": [
        "mediawiki/extensions/LiquidThreads/+/master/sql/tables.json"
    ]
},
"machinevision": {
    "path": [
        "mediawiki/extensions/MachineVision/+/master/sql/tables.json"
    ]
},
"math": {
    "path": [
        "mediawiki/extensions/Math/+/master/sql/mathlatexml.json",
        "mediawiki/extensions/Math/+/master/sql/mathoid.json"
    ]
},
"newsletter": {
    "path": [
        "mediawiki/extensions/Newsletter/+/master/sql/tables.json"
    ]
},
"oathauth": {
    "path": [
        "mediawiki/extensions/OATHAuth/+/master/sql/tables.json"
    ]
},
"oauth": {
    "path": [
        "mediawiki/extensions/OAuth/+/master/schema/tables.json"
    ]
},
"oauthratelimiter": {
    "path": [
        "mediawiki/extensions/OAuthRateLimiter/+/master/schema/tables.json"
    ]
},
"ores": {
    "path": [
        "mediawiki/extensions/ORES/+/master/sql/tables.json"
    ]
},
"pageassessments": {
    "path": [
        "mediawiki/extensions/PageAssessments/+/master/db/tables.json"
    ],
    "dblist": "pageassessments"
},
"pagetriage": {
    "path": [
        "mediawiki/extensions/PageTriage/+/master/sql/tables.json"
    ]
},
"proofreadpage": {
    "path": [
        "mediawiki/extensions/ProofreadPage/+/master/sql/tables.json"
    ],
    "dblist": "wikisource"
},
"propertysuggester": {
    "path": [
        "mediawiki/extensions/PropertySuggester/+/master/sql/tables.json"
    ],
    "dblist": "wikibaserepo"
},
"readinglists": {
    "path": [
        "mediawiki/extensions/ReadingLists/+/master/sql/tables.json"
    ]
},
"securepoll": {
    "path": [
        "mediawiki/extensions/SecurePoll/+/master/sql/tables.json"
    ]
},
"shorturl": {
    "path": [
        "mediawiki/extensions/ShortUrl/+/master/schemas/tables.json"
    ]
},
"timedmediahandler": {
    "path": [
        "mediawiki/extensions/TimedMediaHandler/+/master/sql/tables.json"
    ]
},
"translate": {
    "path": [
        "mediawiki/extensions/Translate/+/master/sql/tables.json"
    ]
},
"uploadwizard": {
    "path": [
        "mediawiki/extensions/UploadWizard/+/master/sql/tables.json"
    ]
},
"urlshortener": {
    "path": [
        "mediawiki/extensions/UrlShortener/+/master/schemas/table.json"
    ]
},
"wikibasequalityconstraints": {
    "path": [
        "mediawiki/extensions/WikibaseQualityConstraints/+/master/sql/tables.json"
    ]
},
"wikilove": {
    "path": [
        "mediawiki/extensions/WikiLove/+/master/patches/tables.json"
    ]
},
"wikimediaeditortasks": {
    "path": [
        "mediawiki/extensions/WikimediaEditorTasks/+/master/sql/tables.json"
    ]
},
Lucas_Werkmeister_WMDE added a subtask: T321063: Fix CheckUser database schema drifts in production.Oct 18 2022, 2:23 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Ladsgroup added a subtask: T326156: Fix CreditsSource drifts.Jan 3 2023, 3:38 PM
Ladsgroup updated the task description. (Show Details)
Ladsgroup closed subtask T326156: Fix CreditsSource drifts as Resolved.Jan 30 2023, 7:42 AM
Ladsgroup updated the task description. (Show Details)
Ladsgroup closed subtask T313253: Fix Flaggedrevs database schema drifts in production as Resolved.Apr 4 2023, 8:11 AM
Ladsgroup updated the task description. (Show Details)
Umherirrender unsubscribed.Jun 30 2024, 7:34 PM
Ladsgroup closed subtask T321063: Fix CheckUser database schema drifts in production as Resolved.Nov 29 2024, 11:04 AM
Ladsgroup updated the task description. (Show Details)
Dreamy_Jazz updated the task description. (Show Details)Dec 4 2024, 3:07 PM
Dreamy_Jazz updated the task description. (Show Details)Dec 4 2024, 6:49 PM
Dreamy_Jazz subscribed.Dec 4 2024, 8:45 PM

Working on getting the drifts for extensions that Trust and Safety Product Team either own or have recently been working on (i.e AbuseFilter).

Dreamy_Jazz updated the task description. (Show Details)Dec 4 2024, 8:56 PM
Dreamy_Jazz updated the task description. (Show Details)Dec 5 2024, 10:13 AM
Ladsgroup closed subtask T381536: Fix SecurePoll database schema drifts in production as Resolved.Thu, Dec 12, 1:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits