Page MenuHomePhabricator

Investigate the required effort of using wbstack as a test system
Open, Needs TriagePublic

Description

From @Addshore:

Adding to the wbstack test situation vs other test situation, the essential steps for having some sort of test site (without elastic) on wbstack would be:

  1. Make a new wbstack mediawiki 1.36 image, with latest of Wikibase (could be built nightly?)
  2. Setup the new DB Schemas for the new 1.36 system & hook them up to the platform
  3. Deploy a new service with the new mediawiki image (and thus federated properties)
  4. Manually deploy a site using the new service etc for testing fedey props

This would leave all wbstack sites in ye old situation, and just have 1 site that could be "continually deployed" to.
These steps would result in a test site that 1) doesnt have elastic, 2) OAuth for service such as quickstatements & cradle would continue to point at the old service 3) a small amount of aditional stuff might also be needed, like poking something to do with the query service.
IMO all of this would draw focus away from the focus of this hike, being feddy props, and thus might not be a good idea for this round to look at.
And if there were more unexpected things it would slow it down further etc

We would probably also want ElasticSearch to be installed per the requirements in the parent ticket.

Event Timeline

Making the new images
Manually creating a new image looks like it could be done but altering the wbstack/mediawiki repo and adding a tag.
We could also adjust the github action in order to regularly build another branch.
Automatically pulling the latest code to the "another branch" could also be done with another manual, or cron triggered action.

Setting up/provisioning 1.36 DBs
We'd need to make a 1.36 .sql file
We'd need to refactorin the api app/Jobs/ProvisionWikiDbJob.php to allow multiple "pools" of Wiki dbs, luckily there are already marked by version so this shouldn't be too hard
We'd also need to alter api app/Http/Controllers/WikiController.php to allow creation of "alpha" Wikis

Alternatively these above steps could be manual and we could just run the provision and controller logic once.

Deploying an alpha service
We can reuse the existing "alpha" image tag in the mediawiki helm charts. We just need to point this at our newly tagged mediawiki image (see: https://github.com/wbstack/deploy/blob/main/charts/mediawiki/values.yaml#L13)
We and make sure all traffic to our test wiki(s?) goes there by adding a pattern match to https://github.com/wbstack/deploy/blob/main/k8s/helm/platform-nginx/nginx.conf#L10

Optional ElasticSearch setup
We need to add a new service for ElasticSearch. In this case we can have it configured with a single tenant and not worry about routing so that it can be used by multiple wikis.
Therefore we should be able to use an off-the-shelf helmchart. We just need to add a helmfile to main/k8s/helm in the deploy repo.
We could add ElasticSearch env variables to the Mediawiki Service or we could even hard-code the values in the LocalSettings.php of the alpha/federated properties image.

If the above steps don't seem to be cutting too many corners this is actually less work than I expected.
The main work would be around regularly building an updated image
It would probably also be worth putting in the effort to at least partially automate the provisioning/setting up alpha 1.36 (or 1.37?!) wikis

Addshore added a project: wbstack.

Accidently closed while cleaning up boards.

Right now it still seems like there would be quite some things in the way of being able to use wbstack production cluster deployment for frequent testing of fed props while in development.
It could be that all of this becomes easier with the creation of a staging cluster (not ticket for that yet).
It could also be easier if this were the sort of thing that happened weekly, but not after every commit? (though even then there are a couple of pain points below)

Making the new images
Manually creating a new image looks like it could be done but altering the wbstack/mediawiki repo and adding a tag.
We could also adjust the github action in order to regularly build another branch.
Automatically pulling the latest code to the "another branch" could also be done with another manual, or cron triggered action.

Indeed making the image should be pretty easy, and an additional branch could be used and image auto built.

Setting up/provisioning 1.36 DBs
We'd need to make a 1.36 .sql file
We'd need to refactorin the api app/Jobs/ProvisionWikiDbJob.php to allow multiple "pools" of Wiki dbs, luckily there are already marked by version so this shouldn't be too hard
We'd also need to alter api app/Http/Controllers/WikiController.php to allow creation of "alpha" Wikis

Alternatively these above steps could be manual and we could just run the provision and controller logic once.

The DBs potentially end up being one of the tough points, as this would be bleeding edge and DB changes could happen, and also get undone etc.
Every new image version that is built could result in DB changes and updates needing to run.
If this were to be automated "in production" then this could result in an easy path to executing code that we don't want to be running.
We could decide that such DB changes would be infrequent and thus do them manually for now?
Probably some more discussion to have around this point.

Deploying an alpha service
We can reuse the existing "alpha" image tag in the mediawiki helm charts. We just need to point this at our newly tagged mediawiki image (see: https://github.com/wbstack/deploy/blob/main/charts/mediawiki/values.yaml#L13)
We and make sure all traffic to our test wiki(s?) goes there by adding a pattern match to https://github.com/wbstack/deploy/blob/main/k8s/helm/platform-nginx/nginx.conf#L10

Yes this sounds about right

Optional ElasticSearch setup
We need to add a new service for ElasticSearch. In this case we can have it configured with a single tenant and not worry about routing so that it can be used by multiple wikis.
Therefore we should be able to use an off-the-shelf helmchart. We just need to add a helmfile to main/k8s/helm in the deploy repo.
We could add ElasticSearch env variables to the Mediawiki Service or we could even hard-code the values in the LocalSettings.php of the alpha/federated properties image.

Underway, but still a couple of things to go in T285085: [EPIC] Deploy Elasticsearch to WBstack

other services

Right now other services also talk to mediawiki, for oauth connections (such as quickstatements and cradle / widar)
Even with all of the steps outlined above these would continue to talk to the non alpha versions of mediawiki.
This is currently at https://github.com/wbstack/deploy/blob/main/k8s/helm/tool-quickstatements/values.yaml.gotmpl#L4
This again probably doesn't matter too much, as this oauth cred retrieval is not a key part of what we want to test?