This came up most recently in the context of looking at https://gerrit.wikimedia.org/r/175394, as an example.

In T76245#793828, @hashar wrote:

In theory we could have a service that spin up a MediaWiki-Vagrant image on labs and apply whatever patchset, but I don't think it is worth the engineering effort. Much cheaper to have tester load the patch on their own Vagrant :-)

It's cheaper to have individuals spin up their own instances rather than having a shared instance? I don't follow.

We have quite a few test wikis and a beta cluster, so I don't think we need to reëstablish the virtue of having public test instances of MediaWiki in this task. That seems to be undisputed.

The question here is whether we can make it even easier to spin up a test MediaWiki instance so that we can quickly and easily demonstrate functionality of a particular proposed code change. I think it's at least an idea worth considering.

You're free to say "I don't want to do this," but you're not free to say "this shouldn't be done." Not so hastily and dismissively, anyway. Re-opening.

Sorry :-(

In T76245#793834, @MZMcBride wrote:

<snip>
It's cheaper to have individuals spin up their own instances rather than having a shared instance? I don't follow.

Yeah I meant on a local machine, a user can spin up MediaWiki Vagrant and quite easily apply a Gerrit change to it for local testing :-)

For a shared environment, we could end up with some kind of wikifarm that would install a dummy MediaWiki, apply the patch, install it and make the result available somewhere (like: alpha.wmflabs.org/12345,42/index.php ). That would need to write a tool / job to spin up such install and garbage collect the install once the patch is merged in. It is probably easy to do with just core + vendor + some skin.

Meanwhile, it seems to me that MediaWiki Vagrant fills the gap quite nicely and is already available. Hence cheaper in term of engineering/developer time.

EDIT: you might want to bring up the subject on wikitech-l and gather interested folks. Then write a small RFC we can implement :]

You can create MediaWiki instances reasonably easy with Labs-Vagrant. This isn't one-click nor particularly resource-efficient, but it works. Also, you get control of the complete environment, so in your testing you are never blocked by waiting for an admin to do this or that.

In T76245#793845, @hashar wrote:

You're free to say "I don't want to do this," but you're not free to say "this shouldn't be done." Not so hastily and dismissively, anyway. Re-opening.

Sorry :-(

No worries. I've been trying to push back against very quick Phabricator task closures. We want to be welcoming to ideas and discussion!

For a shared environment, we could end up with some kind of wikifarm that would install a dummy MediaWiki, apply the patch, install it and make the result available somewhere (like: alpha.wmflabs.org/12345,42/index.php ). That would need to write a tool / job to spin up such install and garbage collect the install once the patch is merged in. It is probably easy to do with just core + vendor + some skin.

I think there's a lot of value in sometimes having a shared environment where everyone can see exactly what we're talking about (and do manual browser testing!).

Meanwhile, it seems to me that MediaWiki Vagrant fills the gap quite nicely and is already available. Hence cheaper in term of engineering/developer time.

I'm trying to glue the pieces together. It sounds like maybe we could:

• have a dedicated Labs provisioned instance(?) such as alpha.wmflabs.org
• create some kind of wrapper Python or shell script that can:
  • clone MediaWiki-Vagrant and/or Labs-Vagrant
  • clone and apply the relevant Gerrit changeset
  • do some very simple Web server configuration to expose the MediaWiki instance to the public Internet

Would that work? Is that crazy?

EDIT: you might want to bring up the subject on wikitech-l and gather interested folks. Then write a small RFC we can implement :]

Yeah, probably. But I'm already severely in RFC debt. :-(

I know some places that have a set number of "qa instances" that people can "check out" (like a book from a library) and have a specific tag/revision be deployed there. Something like that could be worthwhile here, maybe?

But that is probably better for the Labs-Vagrant project instead of Beta Cluster (as BC is solely about master).

This kind of testing is exactly what I expect developers to be doing (and I do myself) in a local MediaWiki-Vagrant instance using git-review to pull the appropriate changes. When you need to share what you are seeing with others, you can use vagrant share or test in a Labs-Vagrant managed instance (like http://sulfinalization.wmflabs.org/).

The complexity of having a pool of shared servers that can be reserved for doing this kind of thing seems like a lot of machinery to keep working. I personally think that keeping beta running is hard enough without adding a "sort of beta but not really" environment that has shared ownership.

In T76245#799303, @bd808 wrote:

The complexity of having a pool of shared servers that can be reserved for doing this kind of thing seems like a lot of machinery to keep working. I personally think that keeping beta running is hard enough without adding a "sort of beta but not really" environment that has shared ownership.

+1000 *If* this is ever on the Release-Engineering-Team list of things to do, it's below maintenance of Beta Cluster.

So, I've thought about this a lot - essentially a 'Tool Labs' type setup for Mediawiki test setups.

Most promising way to do this efficiently would be to have a cluster of machines with a shared:

1. Database server
2. File storage for uploads
3. Redis/Memcached for caching (with prefixing)

And then run the code itself in a docker/LXC environment. This would allow us to setup a simple web interface that lets people pick vagrant roles they want, and provision a docker environment that runs that set of things. Since all 'state' is kept outside the docker environment, it can be restarted / re-created whenever we want. Providing shell access to the image would also not be too hard, and applying a gerrit change to it would also be fairly easy.

Now, the question is a simple matter of 'is this worth it?'. This would essentially replace the many labs-vagrant instances we have. Considering the lack of manpower available for betacluster issues themselves, the ease-of-use of vagrant share / labs-vagrant, I do not think we'll get to this anytime soon. If there's some volunteer interested in getting this done, all is well! But I do not believe WMF engineering/ops will prioritize this anytime :)

FWIW I think this is an excellent idea and I would actually like to see us do something like this, especially along the lines of what @yuvipanda described here.

If we were to have differential run some tests on a virtual machine, then we could simply let that machine linger for a while, eventually recycled if it goes unused. Then along with unit test results, the differential build could link to the live instance with that code applied and running. If a test fails you could log in and check out the problem. When someone comes along to review your revision they will have instant access to see the effects of your change without any extra effort. I think it would significantly streamline the code review process, at least for changes that are front-end-visible (that is, when the change does not require manual back-end testing)

There are about a hundred open changes in Gerrit for mediawiki/core that have been updated in the last week. Could anybody (with an LDAP account) log into any such machine and fiddle with it? How do people simultaneously logged in not interfere with each other's tests?

@scfc: presumably the author of the patch and any reviewers would be able to log in. I'm not sure if it would be appropriate to allow anyone at all to log in without having participated in the review somehow. Anyone could access the instance via http(s) but not necessarily ssh.

The situation currently, on the beta cluster we have many things happening that could interfere with eachother, so a bit of isolation would be an improvement.

Drupal has a tool kind of like this.. https://simplytest.me
you can specify "Drupal core" and under advanced options you can apply a patch or add an extension.

The instance only remains active for 30 minutes (and shows a timer).

In the time since I wrote this ticket, I started and ended a job that had this type of betas infrastructure. It was great. When someone filed a GitHub pull request, Jenkins would automatically spin up an instance of the application and address it at https://pr-1234.beta.example.com. While project managers and other non-technical folks found the most value out of being able to see the changes in a browser easily, developers also loved having their code be automatically built and spun up and easily demonstrable to others. We really should do this for Gerrit changesets if we haven't already.

I've made a small project that allows this. You can try it out here: https://patchdemo.wmflabs.org/

Paste a patch number from Gerrit and pick the base branch, and it will create a test wiki just for you. Be patient, it takes ~30 seconds to set up the new wiki.

Source code: https://github.com/MatmaRex/patchdemo

The project right now doesn't really have any anti-abuse mechanisms. The wikis are not isolated (not in virtual machines or anything), and I'm basically letting you execute arbitrary PHP code, so please don't mess with it, even though you could. If somebody breaks it, I'll just wipe it and won't bother setting it up again (feel free to run your own and password-protect it or something).

I hope this proof of concept at least motivates someone else to do it properly :)

Nice work @matmarex! Looking forward to playing with that. Adding a link to T241140 which is related as well.

@matmarex nice work. I am sure there will be many cases where this tool will be helpful. I already successfully tested one patch with it!

In T76245#5788579, @matmarex wrote:

I've made a small project that allows this. You can try it out here: https://patchdemo.wmflabs.org/

👍 Great tool!

Renamed task to find it easier with search.

@matmarex Many thanks for this tool!

Some feedback:

• Could you make the default password 1 char longer, so it's not asking to change it? Maybe the def username one word?
• Please add the PS number to the gerrit link in "Patches": just append /$PS.
• It would be really informative to see the commit msg (1st line of the 1st patch) next to the demo in the list of demos.
  • The list of patches and commit messages on the Main_Page.
• Is there a way to delete an old instance to save space (cleanup)? There is now, thanks!
  • Won't you run out of space (considering only typical installs)?
• Is there - or are you planning - a way to replace an old instance / update to newer patch?
• To demonstrate different patches to the same task it would be useful to have shared instances, where only the code pulled from git differs, the database and the vendor folder are the same. This would save space and spinup time too. It would not be able to demonstrate changes in the setup process / composer deps, but that's usually not what I (and probably others) would use it for.

In T76245#5973141, @Demian wrote:

• Could you make the default password 1 char longer, so it's not asking to change it? Maybe the def username one word?

https://github.com/MatmaRex/patchdemo/pull/33

• Please add the PS number to the gerrit link in "Patches": just append /$PS.

https://github.com/MatmaRex/patchdemo/pull/34

• It would be really informative to see the commit msg (1st line of the 1st patch) next to the demo in the list of demos.
  • The list of patches and commit messages on the Main_Page.

This would take a bit more work. Feel free to file an issue on GitHub, but I don't know if/when I would work on that (or submit a pull request). The second point is somewhat covered by this issue already: https://github.com/MatmaRex/patchdemo/issues/7.

• Won't you run out of space (considering only typical installs)?

Not really (although obviously the space isn't infinite). Although each wiki is a completely separate installation, the files are de-duplicated on the filesystem level using hardlinks.

A single installation of MediaWiki plus vendor dependencies, Parsoid and the various extensions takes ~930 MB. There are currently 15 wikis, so you would expect them to be using ~14 GB of space. But thanks to the de-duplication, they are actually using only 986 MB in total (not counting the databases, but they are mostly tiny since the wikis have barely any content).

In other words, every wiki after the first one "costs" less than 4 MB. There's currently ~11 GB free space on the labs instance I'm using, so it can host a few thousand wikis before I need to start worrying about clearing out old wikis or switching to a bigger instance.

• Is there - or are you planning - a way to replace an old instance / update to newer patch?

No, The de-duplication scheme with hardlinks actually makes it impractical to update existing wikis, because any change to a file inside one wiki's installation would also affect all other installations (all the identical files refer to the same "area on disk"). You'd have to unlink the file, then write a new one, then de-duplicate again, and that's just too much effort.

• To demonstrate different patches to the same task it would be useful to have shared instances, where only the code pulled from git differs, the database and the vendor folder are the same. This would save space and spinup time too. It would not be able to demonstrate changes in the setup process / composer deps, but that's usually not what I (and probably others) would use it for.

This sounds cool but it would make a few things more difficult. Deleting wikis would now have to consider whether the database is shared. You'd also need some expertise with wikis farms (which I don't have) to be confident in setting this up, e.g. I think that some database tables like 'objectcache' must not be shared or you'll get wrong versions of localisation messages on some of the wikis. I don't really want to implement or maintain that.

@matmarex The new update (patch titles) is awesome! Thank you.

Is there some reason this task isn't resolved?

I'm glad it's useful. I haven't resolved the task myself because I was hoping that leaving it open will motivate someone to work on a better tool. Patchdemo has some limitations rooted in the fact that the wikis it creates are not isolated from each other, and resolving that would require a different approach than I took. But I guess T241140 covers that now.

@matmarex: Patch demo is awesome, thank you so much!! I just used it with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/762099 and I was able to spin up a new wiki in 59 seconds and easily test my changes.