jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/43

mount the MariaDB data volume at the right path

Cause

The chart we use to create the envs uses bitnami's MariaDB image. The image specifies where to mount a data volume if you want the DB data to be persistent (https://hub.docker.com/r/bitnami/mariadb#:~:text=Persisting,database). We were mounting the volume at the wrong location.

Fix at https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/43

Tested the fix locally on my machine.

Fixing production envs

Once the fix to the chart has been merged, log onto k3s.catalyst.eqiad1.wikimedia.cloud and run the following to fix existing environments. Note I tested this locally, but to be cautious we should try it on one env first before running the full loop. Also, existing DB data won't be preserved, but this shouldn't be a problem given that during last week's incident all that data was already lost anyway:

git clone https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts.git /tmp/ci-charts
hc='helm -n cat-env'
kc='kubectl -n cat-env'
for wrel in $($hc ls -f '^wiki-' -q); do
	# This recreates the DB pod with the data volume mounted correctly
	$hc upgrade --reuse-values $wrel /tmp/ci-charts/mediawiki
	# This restarts the Mediawiki pod (helm command above won't), keeps the init container from failing, and connects the MW instance to the new DB
	pod=$($kc get pod -o custom-columns=N:.metadata.name | grep $wrel | grep mediawiki)
	$kc exec -it $pod -c mw-fpm -- rm -rf /var/www/html/w &>/dev/null
	dep=$($kc get deploy -o custom-columns=N:.metadata.name | grep $wrel | grep mediawiki)
	$kc rollout restart deploy $dep
	sleep 1
	# Wait for the wiki to restart before moving on to the next one
	while $kc get po -o custom-columns=N:.metadata.name,S:.status.phase | grep $wrel | grep mediawiki | grep -qi pending; do
		sleep 5
	done
done

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/43

mount the MariaDB data volume at the right path

Fix for new envs is merged, @jnuche planning to apply fix to production envs on Tue Feb 11 in UTC morning hours.

Production environments have now been restarted. Most of them worked, but three failed due to merge conflicts with the latest HEADs, e.g.:

############### Applying requested patches ###############
+ cd /var/www/html/w
+ git fetch origin refs/changes/27/1113827/1 master
From https://gerrit.wikimedia.org/r/mediawiki/core
 * branch                    refs/changes/27/1113827/1 -> FETCH_HEAD
 * branch                    master     -> FETCH_HEAD
+ :
+ git rev-list --cherry-pick --right-only HEAD...d971168608ecdfa3c0112206463ee18380018df4 '^master' --count
+ grep '^0$'
+ git -c user.email=catalyst@example.com -c user.name=Catalyst cherry-pick --cherry-pick --right-only HEAD...d971168608ecdfa3c0112206463ee18380018df4 '^master'
error: could not apply d971168608e... [POC]: Skins support responsive footer icons
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

The three envs are:

• https://c66647917e.catalyst.wmcloud.org/w/
• https://afbbeb3389.catalyst.wmcloud.org/w/
• https://5648f3da62.catalyst.wmcloud.org/w/

Envs c66647917e and afbbeb3389 are actually very similar and were created by the same user one shortly after the other. If there's need to recover these envs, the original patches will need to be rebased first to fix the conflicts.

This actually makes me think this is something we want to think more about in the future, as it affects:

1. Env refresh/update: We have plans to offer these to users at some point. In order to refresh an env, the owner needs to ensure their patches are rebased
2. Env recovery: Any situation similar to what motivated this Phab task. We can rely on a policy similar to env update (paches need to be up-to-date so they can be applied) or we can create some kind of env "snapshot" and use it to recover the env transparently. In the latter case, we would store the commits of any HEADs used on env creation so they could be replayed to recreate an identical environment