┌─────────────────────────┬───────┐
│         Project         │ Wikis │
├─────────────────────────┼───────┤
│ Wikipedia               │ 364   │
├─────────────────────────┼───────┤
│ Wiktionary              │ 198   │
├─────────────────────────┼───────┤
│ Special / chapter wikis │ 122   │
├─────────────────────────┼───────┤
│ Wikibooks               │ 122   │
├─────────────────────────┼───────┤
│ Wikiquote               │ 100   │
├─────────────────────────┼───────┤
│ Wikisource              │ 84    │
├─────────────────────────┼───────┤
│ Wikinews                │ 36    │
├─────────────────────────┼───────┤
│ Wikivoyage              │ 27    │
├─────────────────────────┼───────┤
│ Wikiversity             │ 17    │
└─────────────────────────┴───────┘

'wmgUseCommunityConfiguration' => [
      'default' => true,
      'lockeddown' => false,
],

08:19:21 Finished production-check-commands (duration: 00m 10s)
08:19:21 Waiting 20 seconds for production traffic...
08:19:41 Logstash checker counted 368 error(s) in the last 20 seconds. The threshold is 150.
08:19:41 Top 5 errors:
[182 hits] PHP Warning: Undefined array key 0
[181 hits] Error: Call to a member function getEntityId() on null
[2 hits] PHP Deprecated: Caller from MediaWiki\Exception\MWExceptionHandler::rollbackPrimaryChanges ignored an error originally raised from MediaWiki\Pager\IndexPager::buildQueryInfo (contributions page unfiltered): [1969] Query executi
[2 hits] Wikimedia\Rdbms\DBQueryTimeoutError: A database query timeout has occurred. 
Query: SET STATEMENT max_statement_time=30 FOR SELECT  rev_id,rev_page,rev_actor,actor_rev_user.actor_user AS `rev_user`,actor_rev_user.actor_name AS 
[1 hits] Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded
08:19:41 Logstash checker counted 0 error(s) in the last 20 seconds. OK.
Production checks failed.

[r] Retry production checks
[i] Ignore failure and continue deployment
[b] Roll back all stages and terminate
[x] Exit without rollback
What do you want to do? (default: [r]): b
08:20:34 Rolling back deployment
08:20:34 Rolling back k8s deployment for stage production
08:21:01 mw-videoscaler-main-eqiad failed (exit status 1): (cd /srv/deployment-charts/helmfile.d/services/mw-videoscaler && helm --kubeconfig /etc/kubernetes/mw-videoscaler-deploy-eqiad.config rollback main 2691 --wait --timeout 600s)
stdout: 
stderr: Error: no Job with the name "mercurius-11e741d7f2-webvideotranscodeprioritized" found

08:21:04 mw-videoscaler-main-codfw failed (exit status 1): (cd /srv/deployment-charts/helmfile.d/services/mw-videoscaler && helm --kubeconfig /etc/kubernetes/mw-videoscaler-deploy-codfw.config rollback main 3358 --wait --timeout 600s)
stdout: 
stderr: Error: no Job with the name "mercurius-11e741d7f2-webvideotranscodeprioritized" found

08:23:55 K8s deployment progress:  98% (ok: 2737; fail: 0; left: 55)            
08:23:55 Rollback deployment for stage production failed: K8s Rollback had the following errors:
 Rollback of mw-videoscaler-main-eqiad failed: The rollback of mw-videoscaler-main-eqiad failed
Rollback of mw-videoscaler-main-codfw failed: The rollback of mw-videoscaler-main-codfw failed
Rollback deployment for stage production failed.

[r] Retry rollback deployment
[c] Cancel and seek help
What do you want to do? (default: [r]): r
08:25:11 Rolling back k8s deployment for stage production
08:25:50 K8s deployment progress: 100% (ok: 2792; fail: 0; left: 0)             
08:25:50 Rolling back k8s deployment for stage canaries
08:26:44 K8s deployment progress: 100% (ok: 64; fail: 0; left: 0)               
08:26:44 Rolling back k8s deployment for stage testservers
08:29:12 K8s deployment progress: 100% (ok: 16; fail: 0; left: 0)               

Kubernetes deployment summary:
- testservers: rolled-back
- canaries: rolled-back
- production: rolled-back
Result: rolled back
See the job log for detailed error information.
08:29:12 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23  refs T438219
08:29:12 deploy-promote failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-wikiversions', 'group1 to 1.47.0-wmf.23  refs T438219']' returned non-zero exit status 1. (scap version: 4.294.0) (duration: 25m 04s)
08:29:12 train failed: <CalledProcessError> Command '['/usr/bin/scap', 'deploy-promote', '--train', '--old-version', '1.47.0-wmf.22', 'group1', '1.47.0-wmf.23']' returned non-zero exit status 1. (scap version: 4.294.0) (duration: 25m 11s)
aklapper@deploy1003:/srv/mediawiki-staging$

<duesen> andre: My patch is hitting the getEntityId on null error. I will ignore that and push on with the  backport.
<logmsgbot> !log daniel@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352455|REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s)
<stashbot> T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534
<stashbot> T440377: InvalidArgumentException: Scalar parameter must be a string, number, Stringable, or MessageSpecifier; got array - https://phabricator.wikimedia.org/T440377
<andre> ack
<duesen> andre: confirmed working, backport done

<James_F> andre: If Wikidata injected a bunch of bad content into the job queue they’ll take a while to burn out.