Page MenuHomePhabricator

Investigate and discuss reliable dev environment
Closed, ResolvedPublic

Description

We should take some time to check the available dev environments ( docker, vagrant, .... what else? ) and consider one of them as a standard we want to use for reliable, stable, reproducible testing and development on our local machines.

Background
The projects we're working on and need to maintain become more and more complex. The requirements to the environments that we need to setup for these projects grew accordingly. Both factors make increadingly harder for each member of the team to be able to develop and test under similar conditions. Therefore we should find a way to have stable, reliable development environments, that are easy to maintain and setup and whose settings could easily be shared for reproducible product testing and development.

Requirments

  • ...

Event Timeline

@awight @WMDE-Fisch here's a summary of our discussion, let me know if I left something out:

Goal

  • unified dev environment capable of running all WMDE Technical Wishes projects and their dependencies (e.g. multi-wiki, centralauth)
  • ability to easily (optimally one command) setup a completely configured dev environment for any one of our projects
    • installs any required dependencies
    • imports a set of prefilled data, pages, templates
    • handles any configuration (e.g. LocalSettings)
  • support for using any combination of projects at the same time (e.g. all projects, just TwoColConflict and FileImporter, etc.)
  • shareable (e.g. git repository)
  • highly reproducible for everyone
  • support for testing (e.g. unit, browser)
  • support for debugging (e.g. xdebug, logs)

Discussion

  • we agreed on using Docker over Vagrant, benefits of Docker:
    • lightweight
    • future is more certain
    • benefits of containerization
    • easier to expand upon and work with
  • setting our goals above
  • a little uncertainty over whether we should start fresh or expand upon mediawiki-docker-dev
  • look into using shell scripts and docker-compose overrides for configuring projects

Todo

  • Create a Gitlab repository
  • Decide on whether to start fresh or expand upon an existing solution
  • Use FileImporter as a test pilot

I'll experiment with this for a little bit, in https://gitlab.com/wmde/wmde-technicalwishes-docker-dev . Will do my changes as merge requests, so everyone should feel free to work in parallel!

Trying to fix basic things about the wiki so it becomes usable...

I'd like to hear other people's thoughts about persistence. Should we keep the database contents on the host filesystem, so that it's available across container instances, and provide a clean script to delete it? Or should the default be to rebuild a wiki from scratch, importing only a controlled set of fixture pages? I'm leaning towards the latter. If you want to save some pages, Special:Export them, this isn't a very common use case in my experience.

Also wondering about the admin frontend. Should we provide a Makefile to run the various scripts? Or something more modern? Or nothing at all, just bare scripts which can call each other as needed?

Trying to fix basic things about the wiki so it becomes usable...

I'd like to hear other people's thoughts about persistence. Should we keep the database contents on the host filesystem, so that it's available across container instances, and provide a clean script to delete it? Or should the default be to rebuild a wiki from scratch, importing only a controlled set of fixture pages? I'm leaning towards the latter. If you want to save some pages, Special:Export them, this isn't a very common use case in my experience.

I'm also leaning more towards rebuilding from scratch and a controlled set. You could also always add additional stuff if needed, but in my experience it should also be enough with a fixed set. We could extend that always as needed.

Also wondering about the admin frontend. Should we provide a Makefile to run the various scripts? Or something more modern? Or nothing at all, just bare scripts which can call each other as needed?

I like the idea of a Makefile. Lets start with the script. We would need that anyways. - Also open for a 2020 version of Makefile ... or just a script of scripts? :-D

I like the idea of a Makefile. Lets start with the script. We would need that anyways. - Also open for a 2020 version of Makefile ... or just a script of scripts? :-D

The scripts are getting out of hand already, I think I'll include something like a "functions.sh" for common subroutines. Not so worried about the Makefile either, the main thing on my mind is that the workflow is gross. If I design that part better, then running scripts won't be so annoying. Currently, it goes something like this:

./bin/serve

and in another window,

./bin/addsite dev
./bin/module-up CentralAuth

I don't like the asymmetry between addsite dev and the addsite centralauth hidden beneath the module-up machinery. The solution is probably that serve automatically creates the default site as needed.

Okay, that was fun! I'm off tomorrow and relinquish any priority claims :-)

Latest was https://gitlab.com/wmde/wmde-technicalwishes-docker-dev/-/merge_requests/3 , I've been looking at the multiwiki situation... Starting to think that we need to make some changes to core's installer. Everyone cheats it... But "for now", we might consider dumping just the database structure and running createAndPromote.php, as suggested in https://www.mediawiki.org/wiki/Manual:Wiki_family#Wikimedia_Method . Thinking we should calculate $wikiId in the minimal LocalSettings.php, then surround each LocalSettings.d/00-site-<sitename>.php with an if block...

awight claimed this task.

We're very happy with the environment we've created: https://gitlab.com/wmde/wmde-technicalwishes-docker-dev/