Page MenuHomePhabricator

Discuss future of the Arrays extension
Open, Needs TriagePublic

Assigned To
None
Authored By
MGChecker
Sep 4 2018, 10:27 PM
Referenced Files
None
Tokens
"Love" token, awarded by Dinoguy1000."Love" token, awarded by Kghbln."Love" token, awarded by Jayprakash12345.

Description

Currently, the quite popular Arrays extension isn't actively maintained and poses some weird behavior, as well as questionable implementation in some cases. Considering this, we should collect all issues in one place and decide what should be done about that afterwards, maybe leading to an Arrays 3.0 release.

While the parser behavior should't radically change, I think virtually everything should be up to discussion here, especially removing obsolete features and doing backwards incompatible changes if necessary.

List of suggestions (most of these should probably become tasks sooner or later):

  • Meta
    • This extension should have one or two active maintainers
    • Add a maintenance script to do most incompatible changes
    • Add further parser tests
  • Behavior changes
    • Drop this ugly options parameter and use named params instead
    • Ignore empty values at beginning and end of arrays
    • Add option to remove empty values from arrays
    • Stop unique from removing empty values from arrays
  • Bugfixes
    • Handle multibyte values properly
  • Feature requests
  • Removing old behavior
    • Deprecating and removing Compatibility mode, as it makes code messy and development harder
  • Documentation
    • is horrible and basically hasn't changed since 2009
    • should be clear and on a subpage (Help:Extension:Arrays)
    • isn't localized
  • Other
    • Create vagrant role
    • Update coding style

List of people interested in modernizing this extension

  • @MGChecker - some experience with PHP arrays and with page state parser function extensions. I could be a maintainer if we find noone better suited.
  • You?

Event Timeline

The documentation for Arrays is in horrible shape and desperately needs rewritten IMO. In particular, the function options have never been clearly documented (for example, the #arrayprint syntax documentation shows that options can be provided, but literally nothing else is said about it)! On a behavioral note, requiring a delimiter to be specified for #arraydefine to follow options passed to it is just stupid.

The most visible behavior change, I think, would be having #arraydefine automatically ignore empty elements at the start or end of input (there is probably an argument for having it keep empty elements in the middle of input, and there should be a behavior switch to keep all empty elements, since there may be situations where that's actually wanted).

A feature request I've always wanted is a shorthand function for defining and printing an array in one stroke, without requiring creation of an array variable, similar to #arraymap of Extension:Page Forms (in fact, it might be worth implementing an exact duplicate of #arraymap and coordinating with @Yaron_Koren, Ext:Page Forms' author, to have the extensions somehow share the functionality when both are installed).

On a more general note, a Phabricator project for this extension should be requested.

An important bugfix which has been needed for years is making the extension multi-byte character safe; see for example this topic where I originally raised the issue four years ago, and this more recent topic where someone else ran into the same or a similar issue. (This should really have a separate Phab ticket.)

The documentation for Arrays is in horrible shape and desperately needs rewritten IMO. In particular, the function options have never been clearly documented (for example, the #arrayprint syntax documentation shows that options can be provided, but literally nothing else is said about it)! On a behavioral note, requiring a delimiter to be specified for #arraydefine to follow options passed to it is just stupid.

I agree. If it's simpler to understand the implementation than the documentation, you did something wrong. For me, this is one of the points why I don't like this extension.

The most visible behavior change, I think, would be having #arraydefine automatically ignore empty elements at the start or end of input (there is probably an argument for having it keep empty elements in the middle of input, and there should be a behavior switch to keep all empty elements, since there may be situations where that's actually wanted).

Where do these come from in normal use? I don't see how such elements would be created by accident.

A feature request I've always wanted is a shorthand function for defining and printing an array in one stroke, without requiring creation of an array variable, similar to #arraymap of Extension:Page Forms (in fact, it might be worth implementing an exact duplicate of #arraymap and coordinating with @Yaron_Koren, Ext:Page Forms' author, to have the extensions somehow share the functionality when both are installed).

It would be easy to do so.

  • Arrays wouldn't register its Hook if PageForms is installed.
  • Page Forms would check if Arrays is installed, and if yes, define this array before doing anything else.

However, it would be more elegant to use a library for code sharing like that.

On a more general note, a Phabricator project for this extension should be requested.

I can create one. I personally perfer projects as well instead of seriously considering Flow for coordinationg development.

@Dinoguy1000 Would you be interested in helping to organize a project similar to MediaWiki-extensions-Variables or to rewrite the documentation of this extension?

I'm not able to create Phabricator projects, as far as I know.

I can take a shot at rewriting the documentation, though I won't promise any particular level of quality. =D Probably I'll be basing it heavily on the documentation for ParserFunctions.

I invite people willing to use their skills to bring this extension up to date to add themselves to the task description, stating what they could do.

I'm not able to create Phabricator projects, as far as I know.

Well, I am. With organising I don't mean creation itself, but migrating complaints from Extension talk:Arrays to Phabricator and triaging bugs, once the project is created.

I can take a shot at rewriting the documentation, though I won't promise any particular level of quality. =D Probably I'll be basing it heavily on the documentation for ParserFunctions.

Can't be worse than before. I would prefer if you're writing it on a seperate help as I proposed, so we have a draft that we can link later on.

With organising I don't mean creation itself, but migrating complaints from Extension talk:Arrays to Phabricator and triaging bugs, once the project is created.

Aah, of course; sorry for the confusion. I can start on this later. =)

Currently, the quite popular Arrays extension isn't actively maintained

Relatedly, https://www.mediawiki.org/wiki/Code_stewardship_reviews came to my mind but Arrays is not deployed on WMF servers so it's out of scope. :-/

Do not forget the #arraymaptemplate parser function provided by the PageForms extension. Hmm, usually people using Arrays will most likely also have PageForms installed and vice versa. I personally think it will be great if this extension sees refactoring, maintenance etc. Adding new parser functions which are already provided by other extensions is probably unnecessarily complicating things at this point. I think this will be a second step if at all after the groundwork was done.

Do not forget the #arraymaptemplate parser function provided by the PageForms extension. Hmm, usually people using Arrays will most likely also have PageForms installed and vice versa. I personally think it will be great if this extension sees refactoring, maintenance etc. Adding new parser functions which are already provided by other extensions is probably unnecessarily complicating things at this point. I think this will be a second step if at all after the groundwork was done.

Is there some way to query more complicated things to wikiapiary? It shows me that both Arrays and PageForms are quite popular, but I don't see any possibility to check for Arrays AND PageForms or Arrays XOR PageForms.

Currently, the quite popular Arrays extension isn't actively maintained

Relatedly, https://www.mediawiki.org/wiki/Code_stewardship_reviews came to my mind but Arrays is not deployed on WMF servers so it's out of scope. :-/

I think it's a pity that there's no procedure in place to really maintain popular extensions on Gerrit if they aren't WMF-deployed. Many of these extensions are in desperate need to be updated to modern coding standards, to fix long-standing bugs, and to follow extension best practices. It's not fair to the hundreds of Wikis using them, that they are rotting as soon as the person who wrote the extensions gets off the radar. And it's not like the WMF wouldn't have the resorces to employ someone who is doing this for unmaintained extensions, gives advice to other third-party-extension maintainers and is reviewing there patches. Shouldn't be a big deal

With organising I don't mean creation itself, but migrating complaints from Extension talk:Arrays to Phabricator and triaging bugs, once the project is created.

Aah, of course; sorry for the confusion. I can start on this later. =)

Thank you very much!

One question regarding MediaWiki releases: A project like this requires breaking changes at some point, and it's desirable to provide these at a single point after most of the work is done to deliver a consistent experience with a clear upgrading path. Nevertheless nobody wants to do all things that have been proposed here in a single changeset.

However, there are automatically created MediaWiki releases twice a year, and we can't really control at which point of development this happens. Is there any way to control their behavior, for example by creating them earlier based on something else than current master?

Is there some way to query more complicated things to wikiapiary? It shows me that both Arrays and PageForms are quite popular, but I don't see any possibility to check for Arrays AND PageForms or Arrays XOR PageForms.

Not with the current data model. I have exported the results for each exension and merged them in LO Calc. The result is 100 though I believe it should be much more. Anyways this is already quite a high number.

I think it's a pity that there's no procedure in place to really maintain popular extensions on Gerrit if they aren't WMF-deployed. Many of these extensions are in desperate need to be updated to modern coding standards, to fix long-standing bugs, and to follow extension best practices. It's not fair to the hundreds of Wikis using them, that they are rotting as soon as the person who wrote the extensions gets off the radar. And it's not like the WMF wouldn't have the resorces to employ someone who is doing this for unmaintained extensions, gives advice to other third-party-extension maintainers and is reviewing there patches. Shouldn't be a big deal

That's a general issue and not specific to this extension. I do not really thing that WMF should maintain the extensions which somehow end up in their repo. However I do strongly believe that there must be an easy way to figure out if an extension is actively maintained or just rotting.

One question regarding MediaWiki releases: A project like this requires breaking changes at some point, and it's desirable to provide these at a single point after most of the work is done to deliver a consistent experience with a clear upgrading path. Nevertheless nobody wants to do all things that have been proposed here in a single changeset.

However, there are automatically created MediaWiki releases twice a year, and we can't really control at which point of development this happens. Is there any way to control their behavior, for example by creating them earlier based on something else than current master?

That's a general issue and not specific to this extension too. I am afraid that this is the way it is. When these branches are created it is assumed that they work with the respective branch, at least this is my impression and basically true for the WMF extension. For non-WMF extensions I personally see individual version tags as a way out or backports of current changes to the respective branches you would like to also see them. This is actually happening in some cases.

I think it's a pity that there's no procedure in place to really maintain popular extensions on Gerrit if they aren't WMF-deployed. Many of these extensions are in desperate need to be updated to modern coding standards, to fix long-standing bugs, and to follow extension best practices. It's not fair to the hundreds of Wikis using them, that they are rotting as soon as the person who wrote the extensions gets off the radar. And it's not like the WMF wouldn't have the resorces to employ someone who is doing this for unmaintained extensions, gives advice to other third-party-extension maintainers and is reviewing there patches. Shouldn't be a big deal

That's a general issue and not specific to this extension. I do not really thing that WMF should maintain the extensions which somehow end up in their repo. However I do strongly believe that there must be an easy way to figure out if an extension is actively maintained or just rotting.

However, I don't consider it satisfactory, that even the most popular non-WMF extensions can just end up unmaintained, potentially leading to them becoming unstable. But you are right, this train of thought is leading us nowhere.

One question regarding MediaWiki releases: A project like this requires breaking changes at some point, and it's desirable to provide these at a single point after most of the work is done to deliver a consistent experience with a clear upgrading path. Nevertheless nobody wants to do all things that have been proposed here in a single changeset.

However, there are automatically created MediaWiki releases twice a year, and we can't really control at which point of development this happens. Is there any way to control their behavior, for example by creating them earlier based on something else than current master?

That's a general issue and not specific to this extension too. I am afraid that this is the way it is. When these branches are created it is assumed that they work with the respective branch, at least this is my impression and basically true for the WMF extension. For non-WMF extensions I personally see individual version tags as a way out or backports of current changes to the respective branches you would like to also see them. This is actually happening in some cases.

I agree, however I wan't sure if there's some mechanism in place here that we could use and I didn't know about.

Since I was just checking this for the pre 2.0 documentation page on MediaWiki.org, it's probably worth noting here that literally none of the wikis WikiApiary lists as using Arrays, are listed as using a version older than 2.0, though I don't know offhand if this actually has any significance for removing the compatibility mode and its config variable.

Since I was just checking this for the pre 2.0 documentation page on MediaWiki.org, it's probably worth noting here that literally none of the wikis WikiApiary lists as using Arrays, are listed as using a version older than 2.0, though I don't know offhand if this actually has any significance for removing the compatibility mode and its config variable.

There are some plans to expose extension settings to the public in the future. Right now, the only way to create compatibility mode statistics I can imagine is actually using the parse API to check for certain behaviors in hundreds of wikis… Yay.

So the holdup for refactoring this extension is the uncertainty of the number of users of the Arrays extension using a configuration parameter which was intended to ease the transition to a newer version which was introduced seven or eight years ago? As far as I am concerned there was plenty of time to migrate and my suggestion is to ditch this. Tough titties for those who did not migrate yet I'd say. :)

Nothing prevents you from making a proper non-alpha 2.0 release (long overdue) with the current code, maintain that to a certain degree and at the same time work on a 3.0 with all the refactoring and ditching compatibility mode.

However I do strongly believe that there must be an easy way to figure out if an extension is actively maintained or just rotting.

This is a highly subjective matter. On the one hand, there are a couple of improvements that would add value to the extension, on the other hand, it's still functioning and despite being a popular extensions there do not seem to be serious bugs (I'd consider multibyte chars a new feature as it was never designed for this).
In the end it's up to the community to decide and it simply needs someone to step up and take ownership.

So the holdup for refactoring this extension is the uncertainty of the number of users of the Arrays extension using a configuration parameter which was intended to ease the transition to a newer version which was introduced seven or eight years ago? As far as I am concerned there was plenty of time to migrate and my suggestion is to ditch this. Tough titties for those who did not migrate yet I'd say. :)

Normally I'd say you're right, but parser changes are difficult. You would at least have to specify a deprecation period of 12-18 months; if a larger count of active wikis is affected (we could probably check this with the parse API), we should provide a migration script as well.

One great advantage we have here is that a substantial parser tests file exists. However, it doesn't check compatibility mode works as it should, and I can't guarantee that the file is comprehensive. That's the huge problem with parser functions: Most changes can lead to a behavior change in some weird edge cases. Dealing with them can be hard for large wikis. This raises the question:

Is anyone here aware of a large wiki with comprehensive use of the Array extension and complicated parser functions? We could use a bot to identify differences in parser output (using action=parse in the API of this wiki and on a testwiki with an improved Arrays software. (However, we should probably ask them about this as well. ^^)

I'm still ready to contribute and review code here, however I want to avoid a situation where I'm the only active maintainer here… Currently, I don't even get the registration migration patch merged for more than 6 months. This is frustrating and doesn't really put me in an optimistic mood for this project.

I still think that a hard exit is possible here. Of the 50 wiki I personally know about none uses the antediluvian mode.

Currently, I don't even get the registration migration patch merged for more than 6 months. This is frustrating and doesn't really put me in an optimistic mood for this project.

This is indeed really discouraging. @Legoktm is usually the "Allzweckwaffe" to ping in this case.