Get more clarity on intent, function, and use of Upgrade tests
Closed, ResolvedPublicSpike
Actions

Assigned To

Authored By

	lojo_wmde
	Aug 9 2023, 9:04 AM

Description

I feel unclear about the Upgrade test suite. There are a lot of variables here and I'm not sure which user story it represents.

My understanding so far is that it replicates moving from one version of WMDE Docker Images to another, and confirms whether the new distribution will still run without user intervention with an existing configuration. Is that true?

How many versions past do we intend to support / test upgrading from?

What about breaking changes (i.e to ENV VARS, or database migrations, etc)? Do these get implicitly documented in the upgrade code?

Related Objects
Search...

Status	Subtype	Assigned	Task
Invalid	Spike	None	T343358 Improve tests
Resolved	Spike	lojo_wmde	T344970 Cover as many paths as possible with automated tests. Define which paths need to be done manually (if any)
Resolved		roti_WMDE	T349621 Create Branch+Version structure in our github repository
Resolved		lojo_wmde	T349630 Rewrite Upgrade Tests after new branch versioning scheme
Resolved	Spike	darthmon_wmde	T343870 Get more clarity on intent, function, and use of Upgrade tests

Event Timeline

lojo_wmde created this task.Aug 9 2023, 9:04 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 9 2023, 9:04 AM

I also wonder why the set of tests run during update differs from the normal test suits? Shouldn't we run all tests after upgrade?

Yes, I think this is maybe what happens, and what I'd expect as well. It's just a confusing and potentially over-complicated setup. For it to be useful I'd like to understand how it currently does what it does a bit better, and then likely refactor the setup to re-use the current test suite in a more transparent way? ... Something like that.

https://github.com/wmde/wikibase-release-pipeline/pull/431/files#diff-dcd407a72352698a50b3306f982778e5cbc680b135de0dd529eeadf261e29de8R22

roti_WMDE added a project: Spike.Aug 9 2023, 12:35 PM

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptAug 9 2023, 12:35 PM

roti_WMDE moved this task from Inbox (no prioritisation) to Refined (ready to prioritise) on the wbSuite Release Pipeline Analysis board.Aug 9 2023, 12:36 PM

darthmon_wmde moved this task from Refined (ready to prioritise) to Backlog (prioritised top to bottom) on the wbSuite Release Pipeline Analysis board.Aug 9 2023, 1:19 PM

Since this ticket was created I've been working in this area with the "quiet tests" work and have come to this understanding of what currently happens at a high level:

The Wikibase Suite Series are is booted up at VERSION which must be one of the versions in test/upgrade/old-versions/*.env
The pre_upgrade test suite is ran against that set of services
The old version is then taken down but its Docker volumes are retained so as to not delete the previous data (including the database, MediaWiki LocalSettings.php config, etc)
The TO_VERSION environment is booted up and then php /var/www/html/maintenance/update.php is ran on that MediaWiki/Wikibase instance
The post_upgrade test suite is ran against that

There is also better code organisation and commenting in the test/test_upgrade.sh file in a coming branch.

To complete this task I would propose walking the entire dev team through this overview, and that file, then discuss upgrading testing most generally and the efficacy of this test to those ends.

darthmon_wmde edited projects, added Wikibase Suite Team; removed wbSuite Release Pipeline Analysis.Aug 14 2023, 1:13 PM

darthmon_wmde moved this task from wbSuite Release Pipeline Analysis to Prioritised + Refined (ready to be moved into the sprint) on the Wikibase Suite Team board.

darthmon_wmde mentioned this in T343358: Improve tests.Sep 12 2023, 3:29 PM

darthmon_wmde added a parent task: T344970: Cover as many paths as possible with automated tests. Define which paths need to be done manually (if any).

Upgrade from versions are currently hardcoded, this is probably not what we want to have https://github.com/wmde/wikibase-release-pipeline/blob/867491d486017ce3fd38ac191ef52c278d10087d/.github/workflows/build_and_test.yml#L563

With T345689 we gain channels and semver. How can we leverage that to automatically choose the versions to test upgrade from?

lojo_wmde moved this task from Prioritised + Refined (ready to be moved into the sprint) to Focus: User Metrics on the Wikibase Suite Team board.Oct 19 2023, 1:30 PM

darthmon_wmde added a project: Speedier Release.Oct 23 2023, 10:44 PM

darthmon_wmde added a project: Test-Coverage.Oct 24 2023, 2:42 PM

darthmon_wmde moved this task from Focus: User Metrics to Prioritised + Unrefined on the Wikibase Suite Team board.

I think this ticket can be closed as complete and the upcoming Upgrade test system re-factor / re-write effort will take it from here: https://phabricator.wikimedia.org/T349630

The above analysis which was the output of this Spike can be used as reference in that effort so will add this as a related ticket to that one.

lojo_wmde added a parent task: T349630: Rewrite Upgrade Tests after new branch versioning scheme.Oct 26 2023, 2:26 PM

darthmon_wmde removed a project: Speedier Release.Oct 26 2023, 2:48 PM

darthmon_wmde closed this task as Resolved.Nov 14 2023, 11:56 AM

darthmon_wmde claimed this task.

darthmon_wmde moved this task from Prioritised + Unrefined to Archive (closed) on the Wikibase Suite Team board.