This might require further investigation.

From an initial look, an easy way might be to install an individual copy of xtools into each individual tool labs account. Then, turn off everything except the current tool. Then, we would have to turn the base xtools account into a fancy re-director.

Why do we need routing again? Can't we have a copy of the repo in each of the tools?

It would be fine to have a copy, and I think the only drawback would be URLs like /xtools-ec/ec/blah and /xtools-articleinfo/articleinfo/blah, so it'd be nice to rewrite these to keep the existing URLs (e.g. /xtools/articleinfo/blah).

But do we need to do this? The idea is to be more stable isn't it? That if one tool is abusing resources, the others won't be affected?

In T163283#3200480, @Samwilson wrote:

But do we need to do this? The idea is to be more stable isn't it? That if one tool is abusing resources, the others won't be affected?

Are you replying to me or the ticket?

I have a bigger question. Why are all these tools in the same repo? It would make so much more sense to abstract out the common underlying components into a class/group of classes and use that framework for building each tool. We seem to be just patching stuff together in a rickety way, just to make it work. This isn't scalable. If tomorrow, a developer wants to build a new tool, it would be so much easier for them to just take the library and build off on it instead of messing with the xtools' gigantic, not very comprehensible, repo.

How much extra work do you estimate it to be?

In T163283#3200603, @Niharika wrote:

In T163283#3200480, @Samwilson wrote:

But do we need to do this? The idea is to be more stable isn't it? That if one tool is abusing resources, the others won't be affected?

Are you replying to me or the ticket?

I have a bigger question. Why are all these tools in the same repo? It would make so much more sense to abstract out the common underlying components into a class/group of classes and use that framework for building each tool. We seem to be just patching stuff together in a rickety way, just to make it work. This isn't scalable. If tomorrow, a developer wants to build a new tool, it would be so much easier for them to just take the library and build off on it instead of messing with the xtools' gigantic, not very comprehensible, repo.

How much extra work do you estimate it to be?

Most of what you see in the repo is Symfony framework, which does not need to be touched. In order to create a new tool in future, all a developer has to do is create a controller and one or more views. Along with some basic framework (i18n and a couple configuration options), that's all that's needed. This can be evidenced by rXTR2937dba19438, which created a simple tool in a single commit. There are three "twig" views, and a single controller created there. I also made some style modifications to existing code. I grant you the documentation is not very good there yet, but I am working on improving that.

When I began the rewrite (before Community Tech was even involved), I opted for symfony because of the high re-usability of components (As new components can be added as "helpers" and easily included with one line of code). By splitting into individual tools, we are in fact removing that very re-usability. Unless we wanted to restart the rewrite from the ground up (no!), Symfony must be installed in every Tool Labs tool you create (This is a restriction of Composer). This creates a huge maintenance overhead for the volunteer developers, as every change must be deployed to a large number of tools.

I have a proposal: It appears this ticket is attempting to solve an issue that may or may not be an issue. As we move closer to the release of the modified xTools, let's apply a bit of analytics to the suite. We can identify bottlenecks within the software at that time, and split those bottlenecked components out into their own tool accounts if need be. Alternatively, they can be optimized to remove the bottlenecks. As such, I propose stalling this task until that time, as this is not a current blocker to the continued development of the suite.

I definitely don't think the tools should be broken out into different repos. They share a lot of code, and separating it would defy the point of Symfony, which I've come to enjoy. It looks like a big scary repo but that's the nature of a framework. Once you get it down it's easy peasy. We can also work to document how to add new tools, but those familiar with Symfony would probably figure it out as-is.

What this ticket is more about is whether we should give each tool it's own Tool Labs account.

In T163283#3200480, @Samwilson wrote:
But do we need to do this? The idea is to be more stable isn't it? That if one tool is abusing resources, the others won't be affected?

Exactamente. And I actually like @Matthewrbowker's idea at T163283#3192262. If we can get it to work that way that is most ideal, in my opinion. I don't want to end up with /xtools-adminstats/adminstats, or even just /xtools-adminstats, when we could have the more intuitive and backwards-compatible /xtools/adminstats.

In T163283#3200660, @Matthewrbowker wrote:

As we move closer to the release of the modified xTools, let's apply a bit of analytics to the suite.

I was planning on this :) We're already doing it for the old XTools, which hits a custom API I wrote in Ruby. Not a very good way of doing it, I just used something I already had set up so I could prove how popular the tools were for the Community Wishlist Survey. Anyway, for the rebirth we can make our own internal usage tracking system, as simple as recording the tool and the project, perhaps over time. It'd be cool to have some timeline usage metrics like we do with Pageviews Analysis: https://tools.wmflabs.org/pageviews/meta

In T163283#3200660, @Matthewrbowker wrote:

Symfony must be installed in every Tool Labs tool you create (This is a restriction of Composer). This creates a huge maintenance overhead for the volunteer developers, as every change must be deployed to a large number of tools.

This is true, but it's not quite that bad. The required disk space shouldn't be a problem, what's annoying is having to update each individual tool one by one (if necessary). That's what I've had to deal with for the Pageviews suite of apps. Not fun, but it works. If we really wanted we could write some sort of magical bash script to make us SXAT deployers (Set XTools Aflame Team). There is danger in this approach.

Honestly I'm OK looking into https://xtools.wmflabs.org. I believe we already have the instance itself set up, we just need to install everything. The big drawback of course is we lose free sysadmin support. I don't know about you all, but I'm not like one of those yuvipandas, who maintains probably the best example of a Labs instance. I'm not any kind of panda, actually, so my confidence level in ongoing support and maintenance of a full production Linux machine is not so great. I suppose it's still worth a try? Maybe to see if we can even get it up and running? From there we could do some load testing.

In T163283#3200732, @MusikAnimal wrote:

In T163283#3200660, @Matthewrbowker wrote:

Symfony must be installed in every Tool Labs tool you create (This is a restriction of Composer). This creates a huge maintenance overhead for the volunteer developers, as every change must be deployed to a large number of tools.

This is true, but it's not quite that bad. The required disk space shouldn't be a problem, what's annoying is having to update each individual tool one by one (if necessary). That's what I've had to deal with for the Pageviews suite of apps. Not fun, but it works. If we really wanted we could write some sort of magical bash script to make us SXAT deployers (Set XTools Aflame Team). There is danger in this approach.

That might be a dangerous option, especially since it needs to run an autonomous Composer install. If any new parameters are defined, we would have to handle Composer prompting for them.

Honestly I'm OK looking into https://xtools.wmflabs.org. I believe we already have the instance itself set up, we just need to install everything. The big drawback of course is we lose free sysadmin support. I don't know about you all, but I'm not like one of those yuvipandas, who maintains probably the best example of a Labs instance. I'm not any kind of panda, actually, so my confidence level in ongoing support and maintenance of a full production Linux machine is not so great. I suppose it's still worth a try? Maybe to see if we can even get it up and running? From there we could do some load testing.

There is a current testing instance running at http://xtools-dev.wmflabs.org (vm is xtools-dev02) . It's currently running code from three months(!!) ago, however; it should be fairly straightforward to point it to the new Git repo, run a pull, and update the dependencies. This could be used as a test platform if you'd like to explore the idea further.

A few random comments in favor of modular code bases and separate tool accounts for separate activites:

• Making small modules encourages clean interface design and reduces coupling. Its much more obvious when you are ending up with a "kitchen sink" class.
• Smaller codebases are easier for new contributors to understand.
• Making each "tool" (i.e. distinct activity) independent of the others makes it easier for someone to fork the tool and tweak it to do something related but slightly different.
• In a PHP project, composer can be used to great effect to leverage shared libraries and functionality. For an example, see https://github.com/wikimedia/wikimedia-slimapp and the apps built using it like https://github.com/wikimedia/wikimedia-wikimania-scholarships, https://github.com/bd808/SAL, and https://github.com/bd808/quips. (I still have not gotten around to converting https://github.com/wikimedia/wikimedia-iegreview, but it will remove a ton of code when I do.)
• Versioned libraries can be used to allow changes to roll out across the suite in a controlled manner. If it's all in one bundle then breaking api changes effect everything at once.
• The intent of tool accounts in Tool Labs is to make it easier for ownership of a project to be transferred to new people over time as the prior maintainers lose interest. This transfer is smoother when the projects are small and targeted rather than a large collection of things which are only loosely related.
• Putting all the tools under one account does not solve the problems of instability in one activity effecting all other activities.
• Putting all the tools under one account does not provide an easy way to delegate maintainership of sub-tools to different people.
• If everything is kept in one large bundle, three to five years from now there will be another group of people trying to rescue the things they need from the monolithic suite in the same way that all of you are trying to rescue the work of X today.

I think we're past the point of unbundling the suite, at least if we except to meet our quarterly goal. We should mind the enormous amount of work that has already been put into this, so let's please not take any steps backwards. It would be different if the XTools codebase were a big giant unorganized mess, like my Pageviews apps =P Instead, we're following the conventions provided by Symfony, a very popular framework. Documentation for it is ample, StackOverflow answers, etc. I agree the one big repo may be confusing for a new contributor, but that's assuming they are either unfamiliar with Symfony or we don't have good documentation (which we will work on). Forking and updating a single tool should be straightforward – for most things you want to do there are only a handful of files that would need adjusting. Now, if they want to fork for their own purposes and use only a single tool, then well, yeah, there's a lot of a unnecessary overhead. However XTools has always been seen as a "suite" of apps, one goes with the other, lots of inter-tool links, etc. We at least have config options to turn certain tools off should you want to.

I am ambivalent about Tool Labs versus Labs, but if we do go with the former, we definitely want separate accounts.

@bd808 Solely in terms of disk space and "overhead", how bad is it to have the full Symfony framework running on each dedicated Tool Labs account? I imagine not that bad...? Nevermind maintenance costs, having to run "composer install" on each, etc. – I first want to know if there would be any potential negative effects on the Tool Labs ecosystem.

In T163283#3201839, @bd808 wrote:

• If everything is kept in one large bundle, three to five years from now there will be another group of people trying to rescue the things they need from the monolithic suite in the same way that all of you are trying to rescue the work of X today.

This isn't quite true. Each tool in the old XTools is pretty much a single file. This is hideous because the views, backend logic, etc., are all in one place. No separation of concerns, good syntax highlighting, etc. The MVC is something we all really wanted for the rewrite. And again the "monolithic suite" is mostly Symfony. The "Articleinfo" tool, for instance, has a controller, two views (index and result page), a JS and CSS file. The shared functionality is something we want to be consistent across all tools, and even that only includes (for articleinfo) three PHP files, two shared Twig extensions, and two shared Twig macro files. Everything else you see is under the hood and can be ignored. Compare this to CopyPatrol, a single isolated tool, and you have just as many files to maintain. Not putting down CopyPatrol, you know I love that app (and SlimApp!), I'm just trying to justify that there isn't as much going on in the new XTools as it looks like. The bonus features Symfony gives us are pretty awesome, in my opinion, but that's coming from someone who's still mostly new PHP frameworks. I like it more than Rails, to say the least! :)

Sounds like this investigation has begun!

In T163283#3201891, @MusikAnimal wrote:

@bd808 Solely in terms of disk space and "overhead", how bad is it to have the full Symfony framework running on each dedicated Tool Labs account? I imagine not that bad...? Nevermind maintenance costs, having to run "composer install" on each, etc. – I first want to know if there would be any potential negative effects on the Tool Labs ecosystem.

The size of the PHP codebase is not a concern. Almost every tool we have running uses more disk for access.log and error.log than for the code itself.

In T163283#3201951, @MusikAnimal wrote:

And again the "monolithic suite" is mostly Symfony.

For me, the "monolithic suite" is placing N different end user desired activities under one codebase and delivered from one tool account. I don't care if each HTTP page has 200 support files (especially if they are mostly in a vendor directory managed by a package manager), I care that when someone wants to take an activity from the tool that does almost what they want and make a new tool by forking and modifying the codebase that they don't have to figure out how to disable N-1 other things.

The best tools (and apps generally) do one thing and do it well. If the URLs for your tool include subdirectories for completely different functionality then it is highly likely that you should instead be making a new tool account to host something different. Not to pick on anyone, but the best known example of this monolithic anti-pattern for tool deployment is https://tools.wmflabs.org/hay/. There is no good reason that https://tools.wmflabs.org/hay/directory/ and https://tools.wmflabs.org/hay/exturl/ are hosted from the same tool account. It is in my mind even worse that the tool account itself is named for the primary author of the tools because this promotes the idea that the code is 'owned' by one person. Creating software to support the work of the movement is not a great place to have ego and ownership involved. Nobody likes contributing to pages on-wiki that have a highly protective primary editor. The same is true for our software.

Yup, and I pulled that number myself with https://tools.wmflabs.org/musikanimal I have my regrets :( We definitely don't want to do that for XTools, and I think we're mostly at the consensus of using separate Tool Labs accounts (if we opt for Tool Labs). If someone wants to fork and turn off the other tools, it involves literally one single config file, where you put 0 next to the tools you don't want, and a 1 next to the ones you do. I think all the drawbacks we've discussed thus far regarding new contributors can be solved with good documentation. These config options are actually prompted when you first do composer install, so the developer doesn't have to even look for them.

Finally, it's important to bear in mind the "brand" of XTools. People expect the tools to all be under one roof, or at least associated in some way, and to be able to easily navigate between them. You may consider this an anti-pattern but the brand goes a very, very long way. People will use and trust it just because "xtools" is in the URL. I'm personally willing to make some technical sacrifices to maintain that impression, but again I think we can still accomplish this with separate Tool Labs accounts.

In T163283#3202109, @MusikAnimal wrote:

Finally, it's important to bear in mind the "brand" of XTools.

And that's also why I am really fond of https://xtools.wmflabs.org. In all honesty, you can't deny that that sounds legit??? We just need a splash page with blinking text and marquees, then we're profiting :)

So can someone summarize the implementation that was decided on?

In T163283#3419421, @kaldari wrote:

So can someone summarize the implementation that was decided on?

This ticket started as an investigation on how to get routing like /xtools/topedits to work when each individual XTool has it's own tool account. We've since decided on using VPS (Labs instance), so I think we're done here :) The routing of URL parameters was covered in T165612. "Forwarding" of the routes from old XTools to new hasn't been done, but shouldn't yet because we're first launching as a Beta.

@MusikAnimal: Does this implementation give us the stability improvements we were hoping for?

In T163283#3422033, @kaldari wrote:

@MusikAnimal: Does this implementation give us the stability improvements we were hoping for?

TBD! We need to do some load testing. Ultimately it should be no problem, we just have to figure out if we really need load balancing and/or a bigger box. See T169590#3422038