Followed the same steps as in T326077, size is now doubled without outage to 240GB

@Rosalie_WMDE and I had a chat and looked at how we are using icons. It's all fairly obscure and unclear but this is what we think is currently happening and what we should do.

It appears that we saw substantial growth just after restoring replication and this continued for around 3 days before levelling off again.

As per the comment on the PR this looks good to me; local manual testing failed but almost certainly not due to this change but rather due to the changes this is based upon

Did we discuss the relative risk / reward trade off of having no limit? To me it seems that merely alerting after this limit is reached leaves a long time for bad things to happen if the alerts fires at an inconvenient time like the start of a long weekend.

After inspecting this we believe that there is a necessity for a common api that returns wiki metrics for use here as well as in T324099 and whatever "discovery platform we build"

We looked at this and noticed that the details of the implementation are likely to overlap quite a lot with "Important note for implementation: We want to, in the future, make this insightful for our users as well, so they can find each other."

Merged (and added a slightly more descriptive commit message). Since there is nothing to deploy and no more work to be done I move this straight to done

We talked a little in person about the awkwardness of the k8s client we currently use. This is another place where the importance of T323211 has been surfaced

Looks like it does closely what we want it to do! To me using the auth_request module for this feels slightly counter intuitive but seems to work. To me this is because the use case for auth requires an upstream check on every request; for routing we would actually prefer something more like a short term cache perhaps with some sensible default behaviour.

a wiki will continue pointing to the old app server until it's maintenance update script is run. This should prevent us needing to manually add patches to change the routing.

Should this now move into review ?

We've subsequently been rolling fowards by batches for 5/10/15 at a time

We decided that trying to implement this view on long running tasks was taking too long and therefore thought we ought to try and move forwards.

Sounds like a good start.

I believe that the snipped I posted above did not successfully filter out deleted wikis. Manually inspecting the output of WikiSetting::where(['name' => "wwExtEnableElasticSearch", 'value' => 0])->with('wiki')->get()->pluck('wiki')->where('deleted_at', null) I currently only see 7.

Have tested this locally by:

• checking out main branch of the deploy repo and applying it to my local cluster
• tweaking the api db provision and use versions ( wikiDbProvisionVersion: mw1.38-wbs1 + wikiDbUseVersion: mw1.38-wbs1) to be 1.38
• checking out the patch (https://github.com/wbstack/mediawiki/pull/348) locally
• building and deploying this new image using skaffold (skaffold run -m mediawiki-139)
• running the update process as follows:
  • marking wiki read only (kubectl exec -it deployment/api-app-backend -- bash -c "php artisan wbs-wiki:setSetting domain peanutwiki.wbaas.localhost wgReadOnly migrate-to-139")
  • in a second terminal tailing the mediawiki backend logs k logs --follow mediawiki-139-app-backend-c84fd9ddb-fhnzp
  • dispatching the update job form the scheduling pod kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow MediawikiUpdate wikis.domain peanutwiki.wbaas.localhost mw1.38-wbs1 mw1.39-wbs1 mediawiki-139"

This seems to work; we're happy-ish that we are not totally blocked by T334912. However we still agree that it would be nicer to understand what is happening during the long running update.php. Therefore we block this until we have reviewed (and merged and deployed to staging + production) T334920

The issue also appears to be resolved after rolling the workload forward on Q160. We still saw issues on Q161/Q162 etc.

Marked beyond-notability.wikibase.cloud as readonly with kubectl exec -it deployment/api-app-backend -- bash -c "php artisan wbs-wiki:setSetting domain beyond-notability.wikibase.cloud wgReadOnly migrate-to-139" and run update.php with kubectl exec -it deployments/api-scheduler -- bash -c "php artisan job:dispatchNow MediawikiUpdate wikis.domain beyond-notability.wikibase.cloud mw1.38-wbs1 mw1.39-wbs1 mediawiki-139" as part of investigating T334912

This does appear to have resolved the issue in viewing the page; lets try rolling back to 1.39 workloads and then doing update.php to try and determine if this error is the result of a database mismatch

To investigate if this is an issue with 1.39 we roll this back to being routed to 1.39 workloads (https://github.com/wmde/wbaas-deploy/pull/851)

Yesterday we did the following.
Generate one small batch consisting of coffeebase.wikibase.cloud, mywayordatwayy.wikibase.cloud and andrew.wikibase.cloud. We marked these read only; switched the routing (https://github.com/wmde/wbaas-deploy/pull/849); ran the update script and marked them as read write again before manually confirming that they appear to be OK.

@ 2023-04-18T09:38:53+01:00
Updating Q556295: revision: 862243 updates: 14 on WikiFCD

Previously we had the assumption that this was due to some unusual level of load on the DBs from updating a bit wiki; however this morning while no updates are running it is still apparent.

I'm wondering if this as actually more than just a deprecated method. It appears to me that this might be causing some issue. While investigating TBC I noticed that when trying to load a problematic page I saw these errors in the logs:

{
  "textPayload": "[warning] [error-json] {\"id\":\"7c7dad14242f9dca7946d0ff\",\"type\":\"ErrorException\",\"file\":\"/var/www/html/w/includes/HookContainer/HookContainer.php\",\"line\":137,\"message\":\"PHP Deprecated: Use of PersonalUrls hook (used in onPersonalUrlsConfirmAccount) was deprecated in MediaWiki 1.39. [Called from MediaWiki\\\\HookContainer\\\\HookContainer::run]\",\"code\":0,\"url\":\"/wiki/Item:Q160\",\"caught_by\":\"mwe_handler\",\"backtrace\":[{\"function\":\"handleError\",\"class\":\"MWExceptionHandler\",\"type\":\"::\",\"args\":[\"integer\",\"string\",\"string\",\"string\",\"array\"]},{\"file\":\"/var/www/html/w/includes/debug/MWDebug.php\",\"line\":381,\"function\":\"trigger_error\",\"args\":[\"string\",\"integer\"]},{\"file\":\"/var/www/html/w/includes/debug/MWDebug.php\",\"line\":352,\"function\":\"sendRawDeprecated\",\"class\":\"MWDebug\",\"type\":\"::\",\"args\":[\"string\",\"boolean\",\"string\"]},{\"file\":\"/var/www/html/w/includes/debug/MWDebug.php\",\"line\":233,\"function\":\"deprecatedMsg\",\"class\":\"MWDebug\",\"type\":\"::\",\"args\":[\"string\",\"string\",\"string\",\"integer\"]},{\"file\":\"/var/www/html/w/includes/GlobalFunctions.php\",\"line\":765,\"function\":\"deprecated\",\"class\":\"MWDebug\",\"type\":\"::\",\"args\":[\"string\",\"string\",\"string\",\"integer\"]},{\"file\":\"/var/www/html/w/includes/HookContainer/HookContainer.php\",\"line\":334,\"function\":\"wfDeprecated\",\"args\":[\"string\",\"string\",\"boolean\"]},{\"file\":\"/var/www/html/w/includes/HookContainer/HookContainer.php\",\"line\":137,\"function\":\"callLegacyHook\",\"class\":\"MediaWiki\\\\HookContainer\\\\HookContainer\",\"type\":\"->\",\"args\":[\"string\",\"array\",\"array\",\"array\"]},{\"file\":\"/var/www/html/w/includes/HookContainer/HookRunner.php\",\"line\":3048,\"function\":\"run\",\"class\":\"MediaWiki\\\\HookContainer\\\\HookContainer\",\"type\":\"->\",\"args\":[\"string\",\"array\",\"array\"]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":582,\"function\":\"onPersonalUrls\",\"class\":\"MediaWiki\\\\HookContainer\\\\HookRunner\",\"type\":\"->\",\"args\":[\"array\",\"Title\",\"MediaWiki\\\\Skin\\\\Timeless\\\\SkinTimeless\"]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":1262,\"function\":\"buildPersonalUrls\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[\"boolean\"]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":790,\"function\":\"buildContentNavigationUrlsInternal\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":762,\"function\":\"getPortletsTemplateData\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/skins/Skin.php\",\"line\":667,\"function\":\"getCategoryLinks\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":259,\"function\":\"getCategories\",\"class\":\"Skin\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":146,\"function\":\"prepareQuickTemplate\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/skins/SkinTemplate.php\",\"line\":177,\"function\":\"generateHTML\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/OutputPage.php\",\"line\":2876,\"function\":\"outputPage\",\"class\":\"SkinTemplate\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/includes/MediaWiki.php\",\"line\":922,\"function\":\"output\",\"class\":\"OutputPage\",\"type\":\"->\",\"args\":[\"boolean\"]},{\"file\":\"/var/www/html/w/includes/MediaWiki.php\",\"line\":562,\"function\":\"main\",\"class\":\"MediaWiki\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/index.php\",\"line\":54,\"function\":\"run\",\"class\":\"MediaWiki\",\"type\":\"->\",\"args\":[]},{\"file\":\"/var/www/html/w/index.php\",\"line\":50,\"function\":\"wfIndexMain\",\"args\":[]}]}",
  "insertId": "31krgev0v8lpmfbq",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "namespace_name": "default",
      "project_id": "wikibase-cloud",
      "container_name": "mediawiki",
      "pod_name": "mediawiki-139-app-web-5b64d94c46-zpjm4",
      "location": "europe-west3-a",
      "cluster_name": "wbaas-3"
    }
  },
  "timestamp": "2023-04-18T08:19:51.499605218Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/app_kubernetes_io/component": "app-web",
    "k8s-pod/pod-template-hash": "5b64d94c46",
    "k8s-pod/app_kubernetes_io/name": "mediawiki",
    "compute.googleapis.com/resource_name": "gke-wbaas-3-n2-highmem-16-pool-85ebecb9-8r1o",
    "k8s-pod/app_kubernetes_io/instance": "mediawiki-139"
  },
  "logName": "projects/wikibase-cloud/logs/stderr",
  "receiveTimestamp": "2023-04-18T08:19:56.401987819Z"
}

PR to deploy new image to production https://github.com/wmde/wbaas-deploy/pull/847

https://github.com/wmde/wbaas-deploy/pull/845
https://github.com/wmde/wbaas-deploy/pull/846

We met and saw that there may be issues with the generic wikipage SQL backed search index not being updated when CirrusSearch is enabled.