Change 830576 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Add Makefile generating SQL files from JSON files

https://gerrit.wikimedia.org/r/830576

I like this and has been in back of my mind on how to improve this but I hope we can find a solution that covers more than just Wikibase. Yesterday I had similar fun when doing schema change on core. One thing to keep in mind is that now we have --type=all that automatically discovers and puts the files in all three supported RDBMSes (it's not perfect, specially for core where archives are all over the place). Using type=all should simplify your makefile a lot at least.

In T317180#8217052, @Ladsgroup wrote:

One thing to keep in mind is that now we have --type=all that automatically discovers and puts the files in all three supported RDBMSes (it's not perfect, specially for core where archives are all over the place). Using type=all should simplify your makefile a lot at least.

I don’t think --type=all works for us, at least not while it puts files into $dir/$type/$name.sql whereas we have schema changes under $dir/$type/archives/$name.sql :/

(Edit: Hm, but maybe we can still use it for the tables at least. I’ll check that.)

(Edit: Hm, but maybe we can still use it for the tables at least. I’ll check that.)

Yeah that works, updated the change: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/830576/3..4/Makefile

I wonder if there could be extension.json sections that specify how to map JSON paths to SQL paths, one for generateSchemaSql.php and one for generateSchemaChangeSql.php? (And the MediaWiki core mappings would be hard-coded in the maintenance scripts.)

(I’d also say that figuring this out nicely for all extensions and core doesn’t need to block a solution for Wikibase. We can merge the Makefile and later replace it with the better solution.)

In T317180#8217209, @Lucas_Werkmeister_WMDE wrote:

In T317180#8217052, @Ladsgroup wrote:

One thing to keep in mind is that now we have --type=all that automatically discovers and puts the files in all three supported RDBMSes (it's not perfect, specially for core where archives are all over the place). Using type=all should simplify your makefile a lot at least.

I don’t think --type=all works for us, at least not while it puts files into $dir/$type/$name.sql whereas we have schema changes under $dir/$type/archives/$name.sql :/

We definitely should strive to standardize the paths. The script can also simply check for existence of archives/ directory, it would also help with core's case.

Makefiles not helpful on windows, as far as I know.

Math has written its own maintenance script to generate all types - https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Math/+/refs/heads/master/maintenance/MathGenerateSql.php
DiscussionTools is storing the commands in composer.json - https://gerrit.wikimedia.org/g/mediawiki/extensions/DiscussionTools/+/88f8e534eb4aa07931ca3b9246003d6caae8f9fb/composer.json#19

In T317180#8217307, @Lucas_Werkmeister_WMDE wrote:

I wonder if there could be extension.json sections that specify how to map JSON paths to SQL paths, one for generateSchemaSql.php and one for generateSchemaChangeSql.php? (And the MediaWiki core mappings would be hard-coded in the maintenance scripts.)

That could be T237839

Three different ways to the do same thing is exactly what I want to avoid 😅

In T317180#8217639, @Ladsgroup wrote:

The script can also simply check for existence of archives/ directory, it would also help with core's case.

Could generateSchemaChangeSQL.php just always add archives/ to the path? Are there any extensions that keep schema changes directly in mysql/postgres/sqlite directories, right next to table definitions?

Probably for small extensions but at the end, it's better to make it consistent then "next to each other". Let's make it go to archives/ and change extensions that doesn't follow this. I wish we could make core also go with mysql/ directory but that would mean a lot of changes in core.

The initiative is over and these are improvements that are welcome but not really any blocker to call the project done.