some (hopefuly) helpful notes can be found at the end of this comment here https://phabricator.wikimedia.org/T411010#11413828

https://github.com/wmde/wbaas-deploy/pull/2057

First attempt at deploying this to staging failed:

FailedCreate      39s (x15 over 2m1s)  statefulset-controller  create Pod elasticsearch-3-data-0 in StatefulSet elasticsearch-3-data failed error: Pod "elasticsearch-3-data-0" is invalid: spec.containers[0]
.resources.requests: Invalid value: "250m": must be less than or equal to cpu limit of 0

staging: https://github.com/wmde/wbaas-deploy/pull/2566
production: https://github.com/wmde/wbaas-deploy/pull/2567

https://github.com/wbstack/charts/pull/212

In T416155#11586234, @Andrew-WMDE wrote:

If you change the target branch to reference the branch of the PR before it, GitHub should hide duplicate commits, making the review easier.

Node increase for production cluster: https://github.com/wmde/wbaas-deploy/pull/2563

Increasing staging GKE node count: https://github.com/wmde/wbaas-deploy/pull/2560

https://github.com/wmde/wbaas-deploy/pull/2557

Steps I took:

In T415670#11572529, @Tarrow wrote:

Did this include all the three ways of using mul

@Tarrow Good hint!

This morning I had the idea of diffing a SQL dump of a wiki with one that uses mul and one that does not. Seems like via SQL we could query something like SELECT COUNT(*) FROM mwt_978cbf2ed3_wbt_text_in_lang WHERE wbxl_language = 'mul'; and would even get numerical data on usage

queried wikis: 1863
usage of mul (according to SPARQL query): 29

In T389335#11557945, @Anton.Kokh wrote:

@dena Reading now your comment from Dec 15, do I understand correctly that you consider T356389: Permit upload of SVG icon for use with modern Vector skin that requires it (and perhaps others) a blocker for transitioning users to the new skin? Because they might lose visibility of SVG icons?

I ran this version of the CheckUserEmailExist Command today
https://github.com/wbstack/api/blob/0b8596f19562abcc92e790ce2d0b7d8f112c2f76/app/Console/Commands/User/CheckUserEmailExist.php

I looked a little bit at T401471: 📈Add metric: total # of users and the related T383421: 🤔Take daily measurements of metrics for each Wikibase

looking at last nights backup job it now runs how we intended:

created a draft PR to use the botstopper image: https://github.com/wmde/wbaas-deploy/pull/2419

There still hasn't been any new error reporting (yay?) so I provoked one on staging by changing the sql secondary service port for a few seconds and indeed now we see :

Querying MariaDB itself (secondary, production) gives a much different picture:

SELECT COUNT(*) from wiki_dbs where  wiki_id is null
+----------+
| COUNT(*) |
+----------+
| 56       |
+----------+

deployed - just waiting for an error to pop up to confirm it does what it was intended for.

we now uninstalled 1.39 workloads on staging

This seems to be the only error we still get for 1.39 workloads, and even this one died down quite much:

Some queries I ran on production via kubectl exec -ti deployments/api-app-backend -- php artisan tinker:

php
> WikiDb::with('wiki')->where('version', 'mw1.39-wbs1')->count()
= 699

PR: https://github.com/wbstack/mediawiki/pull/510