Page MenuHomePhabricator

Separate "application" and "project" concerns
Closed, DeclinedPublic

Description

Problems

  1. There is no separation of concerns between my project (i.e. my wiki with it's own custom skins and extensions) and the MediaWiki application.
  2. It is not clear what is user-editable and what is not. The only things that are user-editable are:
    • cache
    • extensions
    • images
    • skins
    • composer.local.json
    • composer.lock
  3. If you modify any other folder or files, you may have difficulty upgrading.
  4. It is not clear if bundled extensions should be upgraded with MediaWiki or if they ought to be upgraded independently. If it's the former than the extensions the user has installed are mixed in with the bundled extensions.
  5. If a user uses Composer's CLI, then composer.json (and composer.lock) will be modified. A user will have to merge the changes when upgrading.
  6. There is no easy way to upgrade MediaWiki with Composer.
  7. Users who use Composer should be able to exclude MediaWiki from their VCS repository (like you can with /vendor)
  8. If a user executes composer create-project mediawiki/core it works properly, but there is no way to upgrade using a similar method.
  9. Because the composer.lock file is shared between the user's project and the application, we are pinning versions in composer.json instead of the proper place which is composer.lock.

Solutions

  1. Move most of MediaWiki into a /core folder.
    • This would involve moving anything that is not "user-customizable" into that folder (i.e. /includeswould become /core/includes
    • The /core folder would have it's own composer.json and a read-only subtree split would be created (and maintained with the Ci server).
    • The entry points would remain to maintain backwards compatibility, but would be modified to be very simple (i.e. just doing a require_once to a path inside /core). Since these are "user-customizable" the user would be able to alter them if they wanted /core in some other directory.
    • To preserve other file paths, we could create relative symlinks that are optional (and would be deleted in a future version).
    • If bundled extensions ought to be upgraded independently of MediaWiki, then they can be installed in /extensions. If not, then we would need to support two directories: /core/extensions that the user ought not modify, and /extensions that the user is free to modify.
  2. Create a mediawiki-app "wrapper"
    • Would include a starting point composer.json that would require mediawiki/core and put it into the webroot.
    • Would include a web root (/html) and inside of that web root would be core (where MediaWiki would go). Would also include slim entry points / symlinks. The reason we would have to put it in a subfolder is because MediaWiki must be in the webroot, but you cannot put extensions or skins (which also need to be in the webroot) within MediaWiki (which will be blown out on upgrade).
    • Users who use the zip/tar would not benefit from this, unless we decided that the zip/tar should use the wrapper as well.

Event Timeline

bd808 subscribed.

Tagging as an RfC because this is a big change to the source tree that would possibly have far reaching impact. Tangentally related to T166010: The Great Namespaceization Effort at least in that moving all the things should take into account PSR-0/PSR-4 changes that are also under discussion.

Logic from top-level *.php files should be moved to classes, not unstructured PHP files in a harder-to-find location.

TBH this fees like pretending MW core is a component that can just be added to your project, without doing anything to actually make that true.

Logic from top-level *.php files should be moved to classes, not unstructured PHP files in a harder-to-find location.

I agree, that's the ideal, I just thought this should be as simple as possible on the first go-around.

TBH this fees like pretending MW core is a component that can just be added to your project, without doing anything to actually make that true.

What do you mean? The next steps are in T166956: Cannot use Composer's CLI to manage a project's dependencies

What "user customization" would this actually enable? Besides the previously-rejected idea about trying to allow installing extensions and skins via composer, that is.

I note that if you don't want extensions, skins, and uploaded images to be subdirectories of the MediaWiki install, that's easily accomplished by setting appropriate configuration settings (e.g. $wgExtensionDirectory, $wgExtensionAssetsPath, $wgStylePath, $wgStyleDirectory, $wgLocalStylePath, $wgUploadPath, and $wgUploadDirectory). If that's done, I believe an upgrade could replace the whole MediaWiki directory as long as you copy over your LocalSettings.php (and StartProfiler.php, if you use that).

Technically a user should be able to modify anything in the root directory (or anything outside of /core after this change) without the changes being blown out after an upgrade.

This issue does not go against the previously rejected idea because loading extensions/skins with Composer would be completely optional.

This change eases using Composer with MediaWiki.

It's good to know that the file paths are already configurable, that should make this easier!

In other words, you also see no advantages to this change besides the "install extensions/skins with composer" thing?

In other words, you also see no advantages to this change besides the "install extensions/skins with composer" thing?

From the description:

This makes it easier to upgrade MediaWiki (most of the time just replace the /core folder). It simplifies what a user can modify and ought not modify. It also would give users the ability to upgrade MediaWiki core with Composer (T166956). And even completely exclude the /core folder from their repository altogether.

:)

Do we really want to imply to the user that they can just replace the core/ directory? What if we add a new php entry point? Will this make (tarball) users less likely to upgrade skins/ and extensions/?

What types of users do you envision would want to upgrade mediawiki (core) via composer. Ae we talking end users (e.g. current tarball or debian package users), developers (current git users), users who want to use mediawiki as a framework for their new app (currently dont exist)? Someone else?

If we can identify the target group of this proposal, id like to hear from some users in that group to see if they think this would be useful.

/

Do we really want to imply to the user that they can just replace the core/ directory?

I think making upgrades easier for users is always a bonus.

What if we add a new php entry point?

Ideally, everything should use the same entry point, but if that's really required, it could either be in /core or if it's in the root, we can instruct the users to copy it (which is what they would be doing now).

Will this make (tarball) users less likely to upgrade skins/ and extensions/?

I've updated the description to handle that. I forgot that the "core" extensions/skins were not in the repository. I think those should be moved into the /core folder which will leave empty directories for user customization.

What types of users do you envision would want to upgrade mediawiki (core) via composer. Ae we talking end users (e.g. current tarball or debian package users), developers (current git users), users who want to use mediawiki as a framework for their new app (currently dont exist)? Someone else?

In my mind the typical use case is the end user who wants to manage all of the dependencies of their project the same way (i.e. with Composer). If I'm an end user that is also a developer, I will probably be making some custom extensions for our project. I should be able to modify the project's composer.json with new dependencies. LIkewise I should be able to exclude the /core folder from the repo completely (like the /vendor folder).

As a new developer to MediaWiki, I expected to be able to setup MediaWiki this way (like Drupal, Symfony, Laravel, etc.). I was surprised that you can't. This was fine, but I do find it a little frustrating that you can't use Composer's CLI without modifying a "core" file (composer.json) and then having to later merge that file with the changes when upgrading.

If we can identify the target group of this proposal, id like to hear from some users in that group to see if they think this would be useful.

Completely agree! I suppose the target group(s) is end users who have developers (or people comfortable with using the command line).

I think this proposal is framed as way more work than it actually is. The current setup of having most of MediaWiki in a "core" folder is already possible, and I run that as my main development setup every day:

km@km-tp ~/p/g/mediawiki> ls
core/  extensions/  libs/  php/  services/  skins/  tools/

I have MediaWiki core in a folder named "core", extensions and skins in their folders, etc.

LocalSettings.php
$wgExtensionDirectory = "$IP/../extensions";
$wgStyleDirectory = "$IP/../skins";

I think the idea of having LocalSettings.php and images/ directory (stuff that is specific to the user's install) in a separate directory is a great idea (the Debian package already implements this separation btw), but proposing it by asking everything to be renamed is a huge stretch.

Instead I think to reach your end goal this task is as simple as 1) document the settings necessary to move all other directories that aren't shipped with core in a separate directory and 2) make mediawiki/core.git installable via composer which I believe you've already filed as a task somewhere. Is there anything I'm missing?

I am in the same boat as @Legoktm. With the couple settings in LocalSettings.php one can effectively split mediawiki/core to a standalone directory. That is what I have been doing for ages on my local development machines. That solves the problem of having skins and extensions outside of the core tree, which seems to be the underlying problem statement for this task.

I can well imagine people create a project by doing:

composer require mediawiki/core
composer require mediawiki/extension-foo-bar

Which would result in something like:

/composer.json
/composer.lock
/index.php
/LocalSettings.php
/vendor/mediawiki/core/
/vendor/mediawiki/extension-foo-bar/

That is better suited for T166956 and I think we might want to decline this task to move files around. Can always be revisited later if it ends up being a solution.

So maybe it's best to create something like https://roots.io/bedrock/ or https://github.com/drupal-composer/drupal-project for now?

I was under the impression that MediaWiki (and extensions/skins) had to be in the webroot. Is that not the case?

I was under the impression that MediaWiki (and extensions/skins) had to be in the webroot. Is that not the case?

Not at all.

There are two "paths" that need consideration: the filesystem directory where files are actually located, and the URL path where a browser will fetch content.

The URL path is defined by $wgScriptPath, by default everything is assumed to be under that path. Extensions, skins, and uploads can use different paths by setting $wgExtensionAssetsPath, $wgStylePath+$wgLocalStylePath, and $wgUploadPath, or you could just configure your webserver to serve the correct directories at the default paths.

The usual install would place the MediaWiki core directory under the webroot matching $wgScriptPath (e.g. if $wgScriptPath is "/w" and your webroot is "/var/www/html", MediaWiki would be at "/var/www/html/w"). But it's easy enough to not do that: you could configure your webserver to serve that path from a different directory outside of your normal webroot, or you could have stub entry points in that directory that load MediaWiki's actual entry points from elsewhere (as WMF does, for example).

Inside MediaWiki, the base directory is generally automatically detected and stored in the global $IP, although you can override this with an environment setting MW_INSTALL_PATH if you need to for some reason.

Extensions, skins, and uploads default to directories under $IP, but can easily use different directories by setting $wgExtensionDirectory, $wgStyleDirectory, and $wgUploadDirectory. There's also $wgCacheDirectory, which has no default in MediaWiki itself but the installer will generate a LocalSettings.php that sets it to $IP/cache.

That leaves only one essential user-controlled file in $IP: LocalSettings.php. And even that can be moved by defining the PHP constant MW_CONFIG_FILE (or MW_CONFIG_CALLBACK). You might do that, for example, by using [[http://www.php.net/manual/en/ini.core.php#ini.auto-prepend-file|PHP's auto-prepend-file setting]], or by having your own entry points that define the constant before loading MediaWiki's entry points.

I understand you could put it in a sub directory, what I am asking is that, typically /vendor and all it's subdirectories are not web accessible (i.e. if your webroot is /var/www/html then the vendor would be /var/www/vendor).

If that would not work (I don't think it would?) then we'd have to work with Composer Installers to add a mediawiki-core type that would put that type into /core (rather than /vendor/mediawiki/mediawiki). We'd then just need to update the root composer.json to have a type mediawiki-core.

The way that everything in MediaWiki is typically in the webroot is a whole other mess that your currently proposed reorganization really would not fix. A "typical" modern application layout might be something like:

mediawiki
 ├─ docs
 ├─ extensions
 ├─ public_html [or www] : the webroot content and the only directory expected to be exposed to anyone
 ├─ skins
 ├─ src : PSR4 organized classes
 ├─ test : PSR4 organized tests
 └─ vendor : Not actually in the repo but generated by Composer for deployments

The top-level composer.json file has a bit of a twisted history. You can see the timeline in composer.json's revision history. It was originally introduced in rMWdda24874f026: composer.json to register MediaWiki in Packagist pretending that MediaWiki was an installable library. This was done by Jeroen DeDauw who was at the time personally working towards making some extensions installable via Composer. A few months later he seems to have discovered the same issues being surfaced here and reversed direction with rMWd6e69d774d61: Make it possible to install extensions using Composer.

In May 2014 we brought the top level composer.json back with psr/log as the sole library. This of course broke things for anyone following Jeroen's lead in using Composer for extension/skin management leading to T67188: composer.json should be useable by WMF, core and extensions. composer-merge-plugin came out of the work of the Librarization project where we made a determined push to drag MediaWiki core kicking and screaming into the world of vendored libraries and Composer. This was at the time the best compromise we could find for allowing people to continue to use Composer to manage locally deployed extensions and skins while still letting MediaWiki do something sane with code management.

In T467: RfC: Extension management with Composer we looked into the possibility of making the extension/skin work pioneered by Jeroen the default and preferred method of managing our plugins. The upstream discussion at https://github.com/composer/composer/issues/4109 was fundamentally the end of T467. I love Composer, but I really do reject the idea that MediaWiki core, skin, and extensions should be installed or upgraded using it. MediaWiki is an application, not a library or a framework. Composer is a library management tool, not a plugin management tool.


I think MediaWiki could and probably should have a well supported plugin management tool that can be used for initial code installations and updates. MediaWiki-Vagrant's git-update command shows that this can be done in an opinionated deploy. The question left open is who would build and maintain that tool, and how opinionated would it need to be. That is a related but orthogonal discussion to this one.

The way that everything in MediaWiki is typically in the webroot is a whole other mess that your currently proposed reorganization really would not fix. A "typical" modern application layout might be something like:

mediawiki
 ├─ docs
 ├─ extensions
 ├─ public_html [or www] : the webroot content and the only directory expected to be exposed to anyone
 ├─ skins
 ├─ src : PSR4 organized classes
 ├─ test : PSR4 organized tests
 └─ vendor : Not actually in the repo but generated by Composer for deployments

I agree, but not trying to reorganize everything right now, just move most everything down a directory. If we are going to reorganize more, I would split non-webroot code from webroot code first. Since everything we have is in the webroot, then it doesn't really make a difference. Regardless the application needs to be separate from the project.


The top-level composer.json file has a bit of a twisted history. You can see the timeline in composer.json's revision history. It was originally introduced in rMWdda24874f026: composer.json to register MediaWiki in Packagist pretending that MediaWiki was an installable library. This was done by Jeroen DeDauw who was at the time personally working towards making some extensions installable via Composer. A few months later he seems to have discovered the same issues being surfaced here and reversed direction with rMWd6e69d774d61: Make it possible to install extensions using Composer.

This is a similar thing that happened in Drupal. Ultimately the problem is, is that /core is an instalable "library" (not literal) and and root is a creatable "project". Until we make the split between the two, it will always not be correct.

In May 2014 we brought the top level composer.json back with psr/log as the sole library. This of course broke things for anyone following Jeroen's lead in using Composer for extension/skin management leading to T67188: composer.json should be useable by WMF, core and extensions. composer-merge-plugin came out of the work of the Librarization project where we made a determined push to drag MediaWiki core kicking and screaming into the world of vendored libraries and Composer. This was at the time the best compromise we could find for allowing people to continue to use Composer to manage locally deployed extensions and skins while still letting MediaWiki do something sane with code management.

I agree that unless we're willing to make /core a dependency of the root, it is the best solution. But it is far from perfect.

In T467: RfC: Extension management with Composer we looked into the possibility of making the extension/skin work pioneered by Jeroen the default and preferred method of managing our plugins. The upstream discussion at https://github.com/composer/composer/issues/4109 was fundamentally the end of T467. I love Composer, but I really do reject the idea that MediaWiki core, skin, and extensions should be installed or upgraded using it. MediaWiki is an application, not a library or a framework. Composer is a library management tool, not a plugin management tool.

I agree that MediaiWiki is an application, not a library or a framework. But Drupal is an application (not a library nor framework) and they have made it work well, and I think we should look at what they are doing and see if the ideas implemented there could work for us.

I think MediaWiki could and probably should have a well supported plugin management tool that can be used for initial code installations and updates. MediaWiki-Vagrant's git-update command shows that this can be done in an opinionated deploy. The question left open is who would build and maintain that tool, and how opinionated would it need to be. That is a related but orthogonal discussion to this one.

I really do believe that Composer is that tool, but our application has to be structured in such a way that it works (sadly). It would be really great if that wasn't the case, but unless we split the concerns ("core" which is an application vs. "root" which is your project) it wont ever work as intended.

This split does not mean that things will change for non-Composer users. As you can see in Drupal, with the split, things work exactly the same as they did before (if you go and just download the .tar.gz)

dbarratt renamed this task from Move most of MediaWiki within a /core folder to Separate "application" and "project" concerns by moving most of MediaWiki within a /core folder.Aug 9 2017, 5:40 PM
daniel subscribed.

As per Tim's suggestion at the August 9 TechCom meeting, I'm putting this into our IRC queue.

This RFC has been scheduled for a public discussion on IRC tonight, September 6.

As always, the discussion will take place in the IRC channel
#wikimedia-office on Wednesday 21:00 UTC (2pm PDT, 23:00 CEST).

Drupal is an application (not a library nor framework)

Drupal styles itself as a content management framework. It is probably somewhere halfway between an app and a real framework.

IMO some of the proposed changes are useful, for reasons completely unrelated to Composer.

As a site admin of a small-but-not-completely amateurish wiki, I have two expectations of the directory structure:

  • anything that comes from upstream should be possible to abstract with the minimal number of symlinks. I want to do the update by setting up and testing the new version advance, then halting traffic, rewriting the symlink, running the DB update script and reenabling traffic. Right now I have to set up separate symlinks for every file except LocalSettings.php in the application root, plus most of the directories (but not all; I have to figure out which). It's a bit of a pain. Having all of that behind a single core directory would make my life easier. I do want skins, extensions and vendor in there, though - those also come from upstream.
  • I want to minimize the amount of files accessible directly through Apache, without setting up a bunch of rewrite rules (probably the worst part of the current installation process). If there was a way to move the core directory completely out of the webroot, that would make things a lot nicer. (Having a version-controlled public_html directory inside MediaWiki does not work well with cheap hosting options. Granted, the alternative of having to copy assets over into the webroot is also not great, but sort of unavoidable when extensions/skins have their own assets.) Yeah you can sort of already do this by setting a dozen globals and constants (some of which are even documented), but it would be an improvement.

Drupal styles itself as a content management framework. It is probably somewhere halfway between an app and a real framework.

Valid. Their mission statement says:

The Drupal mission is to build the best open source content management framework—one that represents the newest ideas and best practices in community publishing, knowledge management, and software design. We believe in open source, innovation, globalism, and collaboration.

If there was a way to move the core directory completely out of the webroot, that would make things a lot nicer.

I completely agree, but outside of the scope of this issue. Let's first get everything inside a single folder then let's move it out of the webroot.

  • I want to minimize the amount of files accessible directly through Apache, without setting up a bunch of rewrite rules (probably the worst part of the current installation process). If there was a way to move the core directory completely out of the webroot, that would make things a lot nicer. (Having a version-controlled public_html directory inside MediaWiki does not work well with cheap hosting options. Granted, the alternative of having to copy assets over into the webroot is also not great, but sort of unavoidable when extensions/skins have their own assets.) Yeah you can sort of already do this by setting a dozen globals and constants (some of which are even documented), but it would be an improvement.

I have this in nginx, only entry points and assets are accessible, everything else is a 404:

	location /wiki/ {
		rewrite ^/wiki/(?<pagename>.*)$ /index.php;
		fastcgi_pass unix:/run/php/php-fpm.sock;
		include /etc/nginx/fastcgi_params;
		fastcgi_param SCRIPT_FILENAME $document_root/index.php;
		fastcgi_param PATH_INFO $pagename;
		fastcgi_param QUERY_STRING $query_string;
		break;
	}
	location ~ ^/(index|load|api|thumb|opensearch_desc)\.php$ {
		fastcgi_pass unix:/run/php/php-fpm.sock;
		include /etc/nginx/fastcgi_params;
		fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
	}
	location ~ ^/resources/(assets|lib|src) {
		try_files $uri 404;
	}
	location ~ ^/(skins|extensions)/.+\.(css|js|gif|jpg|jpeg|png|svg)$ {
		try_files $uri 404;
	}
	location / {
		return 404;
	}

I completely agree, but outside of the scope of this issue. Let's first get everything inside a single folder then let's move it out of the webroot.

Sure, but it's a use case (you need to put things into a core folder before you can move it).

IRC meeting notes: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-09-06-21.01.html

Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-09-06-21.01.log.html

Minutes:

  • '''Separate "application" and "project" concerns by moving most of MediaWiki within a /core folder''' (DanielK_WMDE__, 21:01:53)
    • ''LINK:'' https://phabricator.wikimedia.org/T167038 (DanielK_WMDE__, 21:02:08)
    • prior to extension.json, extensions could be in any directory you wanted. Now there is a requirement that they're all in the same directory ($wgExtensionDirectory) but they can still be symlinked in or something (legoktm, 21:14:16)
    • instead of changing the structure of the mediawiki repo, we could introduce a new repo that has the new top level structure, and would pull in the mediawiki (core) code in a sub-folder (DanielK_WMDE__, 21:15:38)
    • <Vulpix> separate bundled extensions from own extensions (in different directories) would be great <Vulpix> legoktm: when upgrading, it would be easier to unpack the whole tar and then it's clear which extensions I have to move from the old folder to the new without overwriting the bundled ones (legoktm, 21:17:17)
    • <TimStarling> have a top level installer which uses git to pull core and a standard set of extensions (DanielK_WMDE__, 21:19:55)
    • <legoktm> I think composer is tangential to this whole discussion, tgr said on phabricator "IMO some of the proposed changes are useful, for reasons completely unrelated to Composer." which I agree with (DanielK_WMDE__, 21:22:01)
    • if we can't move entry points, we can move config files (DanielK_WMDE__, 21:24:30)
    • <TimStarling> ok, so you would move them to ../images and ../config/LocalSettings.php? (legoktm, 21:25:47)
    • <TimStarling> if we really need to support shared hosting, have an installer that takes the FTP username and password, it can download everything with git and upload it via FTP (DanielK_WMDE__, 21:38:31)
    • < bd808> for the pure tarball use-case (if it really exists) something could be built in Cloud Services that gives a web interface to pick things to put in the tarball and then generates it on the fly (tgr, 21:39:45)
    • <TimStarling> I think we will pick the bits we like out of this RFC and the surrounding discussion <TimStarling> but we're not going to implement it as proposed (DanielK_WMDE__, 21:54:34)

I have this in nginx, only entry points and assets are accessible, everything else is a 404:

Yeah, it's possible but unnecessarily complicated. An install process of "copy MediaWiki to directory outside public_html, copy entry points to public_html, set MW directory location" seems like a lot less headache then messing with webserver config.

I think I got everything from our discussion, please let me know if I missed anything! :)

dbarratt renamed this task from Separate "application" and "project" concerns by moving most of MediaWiki within a /core folder to Separate "application" and "project" concerns.Sep 7 2017, 10:53 PM

After revisiting the IRC discussion, TechCom decided to decline this RFC as proposed. However, we acknowledge that the problems listed are worth looking into. Also, the discussion turned up some interesting proposals and alternatives.

Overall, the RFC seems to propose a change to the directory structure, listing perceived benefits. It would perhaps be better to investigate possible solutions to each problem.