Page MenuHomePhabricator
Create Task
Maniphest T259374

Convert extensions to using abstract schema
Open, Needs TriagePublic
Actions

Description

Seeing as T191231: RFC: Abstract schemas and schema changes has been approved, rather than doing tasks like T244898: Notifications ('Echo') should work with postgres, T244899: CentralAuth should work with postgres and T199544: Make AbuseFilter work on PostgreSQL and SQLite (epic), it seems prudent to convert the current schemas to the abstract syntax and use that to generate the PG schema, rather than building a schema manually ourselves

The TLDR guide for this:

  • Write a tables.json for the schema (see maintenance/tables.json in mw core)
  • Run maintenance/generateSchemaSql.php on the tables.json you created to generate the sql files
  • git add the new files. Wire them in as normal in your LoadExtensionSchemaUpdates subscriber

More information can be seen at https://www.mediawiki.org/wiki/Manual:Schema_changes

Details

SubjectRepoBranchLines +/-
Update UrlShortener to use Abstract schemamediawiki/extensions/UrlShortenermaster+99 -26
Convert Babel to use abstract schemamediawiki/extensions/Babelmaster+87 -20
Convert Linter to abstract schemamediawiki/extensions/Lintermaster+128 -20
Convert ShortUrl to use abstract schemamediawiki/extensions/ShortUrlmaster+80 -14
Convert BetaFeatures to use abstract schemamediawiki/extensions/BetaFeaturesmaster+62 -10
Convert TimedMediaHandler to abstract schemamediawiki/extensions/TimedMediaHandlermaster+165 -32
Convert WikiLove to use abstract schemamediawiki/extensions/WikiLovemaster+193 -29
Convert ProofreadPage to use abstract schemamediawiki/extensions/ProofreadPagemaster+108 -17
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLadsgroupT191231 RFC: Abstract schemas and schema changes
 OpenNoneT259374 Convert extensions to using abstract schema
 OpenNoneT261057 Allow extensions to validate that their tables.json file when transformed matches the SQL output
 ResolvedNoneT261911 Convert Bundled Extensions to Abstract Schema
 ResolvedNoneT261912 Convert WMF Deployed Extensions to Abstract Schema
 ResolvedJdforrester-WMFT259375 Migrate Echo to Abstract Schema
 ResolvedUmherirrenderT306473 Drop foreign keys from echo table echo_push_subscription
 ResolvedUmherirrenderT259376 Migrate CentralAuth to Abstract Schema
 ResolvedUmherirrenderT300576 Standardise type for timestamp columns in CentralAuth extension
 ResolvedUmherirrenderT259377 Migrate AbuseFilter to Abstract Schema
 ResolvedDaimonaT220791 afl_filter should be split in afl_filter_id and afl_global
 Resolved• MarosteguiT234052 Add abuse_filter_log.afl_filter_id and afl_global columns
 Resolved• BstormT234615 Wikireplicas changes for abuse_filter_log including two new columns
 ResolvedDaimonaT269712 Migrate afl_filter to afl_filter_id and afl_global
 ResolvedUrbanecmT269713 Run the MigrateAflFilter script for AbuseFilter
 Resolved• MarosteguiT291719 Remove abuse_filter_log.afl_filter column and adjust schema consequently from Wikimedia production
 ResolvedrookT291806 Remove afl_filter column from the views
 DuplicateNoneT259411 Migrate Wikibase to the abstract schema
 ResolvedUmherirrenderT268535 Convert AntiSpoof to AbstractSchema
 InvalidNoneT268536 Convert Babel to AbstractSchema
 InvalidNoneT268537 Convert BetaFeatures to AbstractSchema
 ResolvedReedyT268538 Convert BounceHandler to AbstractSchema
 ResolvedUmherirrenderT268539 Convert CentralNotice to AbstractSchema
 ResolvedUmherirrenderT310447 Standardise type for timestamp columns in CentralNotice extension
 ResolvedUmherirrenderT268540 Convert CheckUser to AbstractSchema
 ResolvedUmherirrenderT300575 Standardise default for cu_changes.cuc_timestamp on CheckUser extension
 ResolvedUmherirrenderT268542 Convert Cognate to use AbstractSchema changes
 ResolvedUmherirrenderT268543 Convert ContentTranslation to AbstractSchema
 ResolvedUmherirrenderT310448 Standardise type for timestamp columns in ContentTranslation extension
 InvalidNoneT268545 Convert DonationInterface to AbstractSchema
 ResolvedReedyT268546 Convert EntitySchema to AbstractSchema
 ResolvedReedyT268591 Make it possible for abstract schema dbs to not have a PK
 InvalidNoneT268547 Convert EventLogging to AbstractSchema
 ResolvedUmherirrenderT268548 Convert FlaggedRevs to AbstractSchema
 ResolvedUmherirrenderT310437 Standardise type for timestamp columns in FlaggedRevs extension
 ResolvedUmherirrenderT268549 Convert Flow to AbstractSchema
 ResolvedUmherirrenderT310420 Standardise type for timestamp column in Flow extension
 ResolvedJdforrester-WMFT268550 Convert GeoData to AbstractSchema
 ResolvedReedyT268551 Convert GlobalBlocking to AbstractSchema
 ResolvedReedyT268552 Convert GlobalPreferences to AbstractSchema
 ResolvedReedyT268553 Convert GlobalUsage to AbstractSchema
 InvalidNoneT268554 Convert GWToolset to AbstractSchema
 DeclinedNoneT268555 Convert Jade to AbstractSchema
 ResolvedReedyT268556 Convert LdapAuthentication to AbstractSchema
 InvalidNoneT268557 Convert Linter to AbstractSchema
 ResolvedUmherirrenderT268559 Convert LiquidThreads to AbstractSchema
 ResolvedUmherirrenderT310423 Standardise type for timestamp columns in LiquidThreads extension
 ResolvedUmherirrenderT268560 Convert MachineVision to AbstractSchema
 ResolvedPhysikerweltT268561 Convert Math to AbstractSchema
 InvalidNoneT268562 Convert MediaModeration to AbstractSchema
 ResolvedUmherirrenderT268563 Convert Newsletter to AbstractSchema
 ResolvedReedyT268564 Convert OATHAuth to AbstractSchema
 ResolvedUmherirrenderT268565 Convert OAuth to AbstractSchema
 ResolvedUmherirrenderT300446 Standardise type for timestamp columns in OAuth extension
 ResolvedUmherirrenderT268566 Convert ORES to AbstractSchema
 ResolvedReedyT268567 Convert PageAssessment to AbstractSchema
 ResolvedUmherirrenderT268568 Convert PageTriage to AbstractSchema
 ResolvedUmherirrenderT310449 Standardise type for timestamp columns in PageTriage extension
 InvalidNoneT268569 Convert ProofreadPage to AbstractSchema
 ResolvedReedyT268570 Convert PropertySuggester to AbstractSchema
 ResolvedUmherirrenderT268571 Convert ReadingLists to AbstractSchema
 ResolvedSTranT268572 Convert SecurePoll to AbstractSchema
 ResolvedUmherirrenderT272966 Migrate SecurePoll tables that need schema changes to AbstractSchema
 ResolvedTchandersT272965 Migrate SecurePoll tables that have parity to AbstractSchema
 ResolvedUmherirrenderT310444 Standardise type for timestamp columns in SecurePoll extension
 InvalidNoneT268573 Convert ShortUrl to AbstractSchema
 InvalidNoneT268574 Convert Thanks to AbstractSchema
 InvalidNoneT268575 Convert TimedMediaHandler to AbstractSchema
 ResolvedawightT268576 Convert Translate to AbstractSchema
 ResolvedReedyT268577 Convert UploadWizard to AbstractSchema
 InvalidNoneT268578 Convert UrlShortener to AbstractSchema
 ResolvedLadsgroupT268579 Convert WikibaseQualityConstraints to AbstractSchema
 InvalidNoneT268580 Convert WikiLove to AbstractSchema
 ResolvedUmherirrenderT268581 Convert WikimediaEditorTasks to AbstractSchema
 ResolvedUmherirrenderT298496 Convert CreditsSource to AbstractSchema
 OpenNoneT270882 Convert MathSearch to AbstractSchema
 ResolvedNoneT280240 Convert Wikispeech to using abstract schema
 OpenNoneT280241 Convert WikispeechSpeechDataCollector to using abstract schema

Event Timeline

Reedy created this task.Jul 31 2020, 5:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2020, 5:11 PM
Reedy added a parent task: T191231: RFC: Abstract schemas and schema changes.Jul 31 2020, 5:14 PM
Ladsgroup subscribed.Jul 31 2020, 5:16 PM
Reedy added a project: Platform Team Initiatives (Abstract Schema).Jul 31 2020, 5:22 PM
Reedy updated the task description. (Show Details)Jul 31 2020, 5:25 PM
Reedy updated the task description. (Show Details)
Ladsgroup added a comment.Aug 1 2020, 3:06 PM

An extra step would be to add a test to make sure the .sql files are indeed generated from the abstract schemas (so we don't end up people doing the old way). There is such test in core already, maybe we can sorta hook it up so they define it extension.json? What do you think?

Reedy added a comment.Aug 22 2020, 8:19 PM
In T259374#6353619, @Ladsgroup wrote:

An extra step would be to add a test to make sure the .sql files are indeed generated from the abstract schemas (so we don't end up people doing the old way). There is such test in core already, maybe we can sorta hook it up so they define it extension.json? What do you think?

I think minimal possible "effort" to have this test in an extension would be a win, yeah

gerritbot added a comment.Aug 22 2020, 8:20 PM

Change 621905 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/BetaFeatures@master] Convert BetaFeatures to use abstract schema

https://gerrit.wikimedia.org/r/621905

gerritbot added a project: Patch-For-Review.Aug 22 2020, 8:20 PM
gerritbot added a comment.Aug 22 2020, 8:34 PM

Change 621909 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/ProofreadPage@master] Convert ProofreadPage to use abstract schema

https://gerrit.wikimedia.org/r/621909

gerritbot added a comment.Aug 22 2020, 8:48 PM

Change 621912 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/Linter@master] Convert Linter to abstract schema

https://gerrit.wikimedia.org/r/621912

gerritbot added a comment.Aug 22 2020, 9:00 PM

Change 621915 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/Babel@master] Convert Babel to use abstract schema

https://gerrit.wikimedia.org/r/621915

gerritbot added a comment.Aug 22 2020, 9:23 PM

Change 621917 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/TimedMediaHandler@master] Convert TimedMediaHandler to abstract schema

https://gerrit.wikimedia.org/r/621917

gerritbot added a comment.Aug 22 2020, 9:35 PM

Change 621919 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/ShortUrl@master] Convert ShortUrl to use abstract schema

https://gerrit.wikimedia.org/r/621919

gerritbot added a comment.Aug 22 2020, 9:45 PM

Change 621922 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/OATHAuth@master] Update OATHAuth to Abstract Schema

https://gerrit.wikimedia.org/r/621922

gerritbot added a comment.Aug 22 2020, 10:12 PM

Change 621924 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikiLove@master] Convert WikiLove to use abstract schema

https://gerrit.wikimedia.org/r/621924

gerritbot added a comment.Aug 22 2020, 10:20 PM

Change 621909 merged by jenkins-bot:
[mediawiki/extensions/ProofreadPage@master] Convert ProofreadPage to use abstract schema

https://gerrit.wikimedia.org/r/621909

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.6; 2020-08-25).Aug 22 2020, 11:00 PM
Reedy mentioned this in T261057: Allow extensions to validate that their tables.json file when transformed matches the SQL output.Aug 22 2020, 11:26 PM
gerritbot added a comment.Aug 24 2020, 12:27 PM

Change 621905 merged by jenkins-bot:
[mediawiki/extensions/BetaFeatures@master] Convert BetaFeatures to use abstract schema

https://gerrit.wikimedia.org/r/621905

gerritbot added a comment.Aug 24 2020, 12:27 PM

Change 621924 merged by jenkins-bot:
[mediawiki/extensions/WikiLove@master] Convert WikiLove to use abstract schema

https://gerrit.wikimedia.org/r/621924

gerritbot added a comment.Aug 24 2020, 12:31 PM

Change 621917 merged by jenkins-bot:
[mediawiki/extensions/TimedMediaHandler@master] Convert TimedMediaHandler to abstract schema

https://gerrit.wikimedia.org/r/621917

gerritbot added a comment.Aug 24 2020, 3:08 PM

Change 621912 merged by jenkins-bot:
[mediawiki/extensions/Linter@master] Convert Linter to abstract schema

https://gerrit.wikimedia.org/r/621912

gerritbot added a comment.Aug 24 2020, 3:08 PM

Change 621919 merged by jenkins-bot:
[mediawiki/extensions/ShortUrl@master] Convert ShortUrl to use abstract schema

https://gerrit.wikimedia.org/r/621919

Jdforrester-WMF subscribed.Aug 24 2020, 5:23 PM
Reedy added subtasks: T261911: Convert Bundled Extensions to Abstract Schema, T261912: Convert WMF Deployed Extensions to Abstract Schema.Sep 3 2020, 3:06 AM
Reedy removed subtasks: T259411: Migrate Wikibase to the abstract schema, T259377: Migrate AbuseFilter to Abstract Schema, T259376: Migrate CentralAuth to Abstract Schema, T259375: Migrate Echo to Abstract Schema.
gerritbot added a comment.Sep 19 2020, 10:58 AM

Change 621915 merged by jenkins-bot:
[mediawiki/extensions/Babel@master] Convert Babel to use abstract schema

https://gerrit.wikimedia.org/r/621915

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.10; 2020-09-22); removed MW-1.36-notes (1.36.0-wmf.6; 2020-08-25).Sep 19 2020, 11:00 AM
gerritbot added a comment.Oct 12 2020, 7:02 AM

Change 633472 had a related patch set uploaded (by Ammarpad; owner: Ammarpad):
[mediawiki/extensions/UrlShortener@master] Update UrlShortener to use Abstract schema

https://gerrit.wikimedia.org/r/633472

gerritbot added a comment.Oct 12 2020, 10:33 AM

Change 633472 merged by jenkins-bot:
[mediawiki/extensions/UrlShortener@master] Update UrlShortener to use Abstract schema

https://gerrit.wikimedia.org/r/633472

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.13; 2020-10-12); removed MW-1.36-notes (1.36.0-wmf.10; 2020-09-22).Oct 12 2020, 11:00 AM
Tgr subscribed.Oct 18 2020, 6:36 PM
Reedy removed projects: MW-1.36-notes (1.36.0-wmf.13; 2020-10-12), Patch-For-Review.Nov 24 2020, 12:00 AM
Reedy added a project: Goal.Nov 24 2020, 12:08 AM
Reedy mentioned this in T280240: Convert Wikispeech to using abstract schema.Apr 15 2021, 10:18 AM
Reedy mentioned this in T280241: Convert WikispeechSpeechDataCollector to using abstract schema.
Reedy added a subtask: T280240: Convert Wikispeech to using abstract schema.
Reedy added a subtask: T280241: Convert WikispeechSpeechDataCollector to using abstract schema.
kalle closed subtask T280240: Convert Wikispeech to using abstract schema as Resolved.Jun 10 2021, 7:40 AM
Ltrlg subscribed.Nov 22 2021, 11:23 AM
Umherirrender added a subtask: T298496: Convert CreditsSource to AbstractSchema.Jan 3 2022, 10:46 PM
Umherirrender mentioned this in T237839: Wire abstract schema and schema changes for extension.Jun 15 2022, 7:04 PM
Jdforrester-WMF closed subtask T298496: Convert CreditsSource to AbstractSchema as Resolved.Jul 13 2022, 3:52 PM
Jdforrester-WMF closed subtask T261911: Convert Bundled Extensions to Abstract Schema as Resolved.Jul 13 2022, 5:46 PM
Jdforrester-WMF closed subtask T261912: Convert WMF Deployed Extensions to Abstract Schema as Resolved.Jul 13 2022, 7:14 PM
Umherirrender mentioned this in T261912: Convert WMF Deployed Extensions to Abstract Schema.Jul 14 2022, 10:57 PM
Sbailey mentioned this in T327806: Fix linter table linter_params definition to not be nullable.Jan 24 2023, 5:53 PM
Ladsgroup removed a project: Platform Team Initiatives (Abstract Schema).Oct 6 2023, 7:53 PM

The initiative is over and these are improvements that are welcome but not really any blocker to call the project done specially given the fact that all wmf-deployed extensions have been migrated.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL