Page MenuHomePhabricator
Paste P45285

mwpresync scap stage-train error
ActivePublic
Actions

Authored by jeena on Mar 7 2023, 8:39 PM.
Tags
None
Referenced Files
F36897062: mwpresync scap stage-train error
Mar 7 2023, 8:39 PM
Subscribers
None
jhuneidi@deploy2002:~$ sudo -u mwpresync /usr/bin/scap stage-train -Dfull_image_build:True --yes auto
18:46:38 Initializing stage-train auto mode
18:46:38 Retrieving train information...
18:46:40 Using version 1.40.0-wmf.26
18:46:40 ----------------------------
1. Starting: prep
18:46:40 Started scap prep 1.40.0-wmf.26
18:46:40 Update https://gerrit.wikimedia.org/r/mediawiki/core (wmf/1.40.0-wmf.26 branch) in /srv/mediawiki-staging/php-1.40.0-wmf.26
18:46:41 https://gerrit.wikimedia.org/r/mediawiki/core checked out at commit 4833c71bfb0c9579a488148db7793dd07e7acfb2
18:46:48 MediaWiki wmf/1.40.0-wmf.26 successfully checked out.
18:46:48 Finished scap prep 1.40.0-wmf.26 (duration: 00m 07s)
18:46:48 DONE!
18:46:48 ----------------------------
2. Starting: patch
...
18:46:48 DONE!
18:46:48 ----------------------------
3. Starting: testwikis
18:46:49 Updated /srv/mediawiki-staging/wikiversions.json: 3 inserted, 0 migrated.
18:46:49 Running scap prep auto
18:46:49 Started scap prep auto
18:46:49 Update https://gerrit.wikimedia.org/r/operations/mediawiki-config (master branch) in /srv/mediawiki-staging
18:46:50 Changes pulled down since last fetch:
commit ca0c0557abb0b4ed09025c4dac93fd1223675100
Author: Kosta Harlan <kharlan@wikimedia.org>
Date: Mon Mar 6 12:00:50 2023 +0100
GrowthExperiments: Make new impact module default on betalabs
Some LevelingUp features require the user to also have the new impact
module enabled.
Bug: T328757
Change-Id: Id19514fcb407ea972f5c73ad88d784a8dfb987c7
18:46:50 https://gerrit.wikimedia.org/r/operations/mediawiki-config checked out at commit ca0c0557abb0b4ed09025c4dac93fd1223675100
18:46:50 Update https://gerrit.wikimedia.org/r/mediawiki/core (wmf/1.40.0-wmf.25 branch) in /srv/mediawiki-staging/php-1.40.0-wmf.25
18:46:50 https://gerrit.wikimedia.org/r/mediawiki/core checked out at commit 503ad405e0abecafd6c59cef52cb45d6e1d2477b
Applying patch /srv/patches/1.40.0-wmf.25/core/01-T285159.patch in /srv/mediawiki-staging/php-1.40.0-wmf.25
Applying patch /srv/patches/1.40.0-wmf.25/extensions/CheckUser/01-T278365.patch in /srv/mediawiki-staging/php-1.40.0-wmf.25
[APPLIED] /srv/patches/1.40.0-wmf.25/core/01-T285159.patch
[APPLIED] /srv/patches/1.40.0-wmf.25/extensions/CheckUser/01-T278365.patch
18:46:58 MediaWiki wmf/1.40.0-wmf.25 successfully checked out.
18:46:58 Finished scap prep auto (duration: 00m 09s)
18:46:58 Running scap sync-world
___ ____
⎛ ⎛ ,----
\ //==--'
_//|,.·//==--' ____________________________
_OO≣=- ︶ ᴹw ⎞_§ ______ ___\ ___\ ,\__ \/ __ \
(∞)_, ) ( | ______/__ \/ /__ / /_/ / /_/ /
¨--¨|| |- ( / ______\____/ \___/ \__^_/ .__/
««_/ «_/ jgs/bd808 /_/
18:46:59 Started scap: testwikis wikis to 1.40.0-wmf.26 refs T330204
18:46:59 Started cache_git_info
18:46:59 Finished cache_git_info (duration: 00m 00s)
18:46:59 Started l10n-update
18:46:59 Updating ExtensionMessages-1.40.0-wmf.25.php
18:46:59 Updating LocalisationCache for 1.40.0-wmf.25 using 30 thread(s)
18:46:59 Running rebuildLocalisationCache.php as www-data
18:47:00 0 languages rebuilt out of 477
18:47:00 Use --force to rebuild the caches which are still fresh.
18:47:00 Generating JSON versions and md5 files (as www-data)
18:47:01 Finished l10n-update (duration: 00m 02s)
18:47:01 Checking for new runtime errors locally
18:47:01 Running sudo -u mwbuilder /usr/local/bin/update-mediawiki-tools-release
Already up to date.
18:47:02 Started build-and-push-container-images
18:47:02 K8s images build/push output redirected to /srv/mwpresync/scap-image-build-and-push-log
18:53:05 Finished build-and-push-container-images (duration: 06m 02s)
18:53:05 Started sync-masters
18:53:13 sync-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
18:53:13 Finished sync-masters (duration: 00m 08s)
18:53:13 Started sync-pull-masters
18:53:19 sync-pull-masters: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
18:53:19 Finished sync-pull-masters (duration: 00m 05s)
18:53:19 Started docker pull on k8s nodes
18:55:06 docker_pull_k8s: 100% (in-flight: 0; ok: 30; fail: 0; left: 0)
18:55:06 Finished docker pull on k8s nodes (duration: 01m 47s)
18:55:06 Started sync-testservers-k8s
18:55:07 Started Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
18:55:07 Command '['helmfile', '-e', 'codfw', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Stdout/stderr follows:
18:55:07 helmfile.yaml: basePath=.
skipping missing values file matching "values-pinkunicorn.yaml"
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: pinkunicorn (11 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-debug (8 bytes)
9: --values (8 bytes)
10: /tmp/values862301631 (20 bytes)
11: --values (8 bytes)
12: /tmp/values609030674 (20 bytes)
13: --values (8 bytes)
14: /tmp/values744735817 (20 bytes)
15: --values (8 bytes)
16: /tmp/values593356052 (20 bytes)
17: --values (8 bytes)
18: /tmp/values410407779 (20 bytes)
19: --values (8 bytes)
20: /tmp/values293946214 (20 bytes)
21: --values (8 bytes)
22: /tmp/values369936781 (20 bytes)
23: --values (8 bytes)
24: /tmp/values573070728 (20 bytes)
25: --values (8 bytes)
26: /tmp/values805823303 (20 bytes)
27: --detailed-exitcode (19 bytes)
28: --kubeconfig (12 bytes)
29: /etc/kubernetes/mw-debug-deploy-codfw.config (44 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:07 Finished Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 00s)
18:55:07 Started Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
18:55:07 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Stdout/stderr follows:
18:55:07 helmfile.yaml: basePath=.
skipping missing values file matching "values-pinkunicorn.yaml"
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: pinkunicorn (11 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-debug (8 bytes)
9: --values (8 bytes)
10: /tmp/values574079825 (20 bytes)
11: --values (8 bytes)
12: /tmp/values975295100 (20 bytes)
13: --values (8 bytes)
14: /tmp/values315658155 (20 bytes)
15: --values (8 bytes)
16: /tmp/values375166734 (20 bytes)
17: --values (8 bytes)
18: /tmp/values896379669 (20 bytes)
19: --values (8 bytes)
20: /tmp/values348120432 (20 bytes)
21: --values (8 bytes)
22: /tmp/values871773711 (20 bytes)
23: --values (8 bytes)
24: /tmp/values924015138 (20 bytes)
25: --values (8 bytes)
26: /tmp/values716950809 (20 bytes)
27: --detailed-exitcode (19 bytes)
28: --kubeconfig (12 bytes)
29: /etc/kubernetes/mw-debug-deploy-eqiad.config (44 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:07 Finished Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 00s)
18:55:07 K8s deployment to stage testservers failed: K8s deployment had the following errors:
codfw: Deployment of mw-debug-pinkunicorn failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
eqiad: Deployment of mw-debug-pinkunicorn failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Rolling back to prior state...
18:55:07 Started Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
18:55:07 Command '['helmfile', '-e', 'codfw', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Stdout/stderr follows:
18:55:07 helmfile.yaml: basePath=.
skipping missing values file matching "values-pinkunicorn.yaml"
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: pinkunicorn (11 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-debug (8 bytes)
9: --values (8 bytes)
10: /tmp/values223110619 (20 bytes)
11: --values (8 bytes)
12: /tmp/values488346494 (20 bytes)
13: --values (8 bytes)
14: /tmp/values120389829 (20 bytes)
15: --values (8 bytes)
16: /tmp/values352958816 (20 bytes)
17: --values (8 bytes)
18: /tmp/values045187391 (20 bytes)
19: --values (8 bytes)
20: /tmp/values002188178 (20 bytes)
21: --values (8 bytes)
22: /tmp/values158622665 (20 bytes)
23: --values (8 bytes)
24: /tmp/values125342356 (20 bytes)
25: --values (8 bytes)
26: /tmp/values707187939 (20 bytes)
27: --detailed-exitcode (19 bytes)
28: --kubeconfig (12 bytes)
29: /etc/kubernetes/mw-debug-deploy-codfw.config (44 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:07 Finished Running helmfile -e codfw --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 00s)
18:55:07 Started Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug
18:55:07 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Stdout/stderr follows:
18:55:07 helmfile.yaml: basePath=.
skipping missing values file matching "values-pinkunicorn.yaml"
Comparing release=pinkunicorn, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: pinkunicorn (11 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-debug (8 bytes)
9: --values (8 bytes)
10: /tmp/values003979507 (20 bytes)
11: --values (8 bytes)
12: /tmp/values537256886 (20 bytes)
13: --values (8 bytes)
14: /tmp/values751037341 (20 bytes)
15: --values (8 bytes)
16: /tmp/values701489240 (20 bytes)
17: --values (8 bytes)
18: /tmp/values610690007 (20 bytes)
19: --values (8 bytes)
20: /tmp/values247525706 (20 bytes)
21: --values (8 bytes)
22: /tmp/values821945889 (20 bytes)
23: --values (8 bytes)
24: /tmp/values382053644 (20 bytes)
25: --values (8 bytes)
26: /tmp/values436568315 (20 bytes)
27: --detailed-exitcode (19 bytes)
28: --kubeconfig (12 bytes)
29: /etc/kubernetes/mw-debug-deploy-eqiad.config (44 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-debug-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:07 Finished Running helmfile -e eqiad --selector name=pinkunicorn apply in /srv/deployment-charts/helmfile.d/services/mw-debug (duration: 00m 00s)
18:55:07 Caught another exception while trying to roll back. Giving up: K8s deployment had the following errors:
codfw: Deployment of mw-debug-pinkunicorn failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
eqiad: Deployment of mw-debug-pinkunicorn failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=pinkunicorn', 'apply']' returned non-zero exit status 1.
18:55:07 Finished sync-testservers-k8s (duration: 00m 01s)
18:55:07 Started sync-testservers
18:55:16 sync-testservers: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)
18:55:16 Per-host sync duration: average 8.0s, median 8.2s
18:55:16 rsync transfer: average 574,615 bytes/host, total 2,298,460 bytes
18:55:16 Finished sync-testservers (duration: 00m 08s)
18:55:16 Started sync-canaries-k8s
18:55:17 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:55:17 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:55:17 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:55:17 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values691448520 (20 bytes)
11: --values (8 bytes)
12: /tmp/values759447431 (20 bytes)
13: --values (8 bytes)
14: /tmp/values373628218 (20 bytes)
15: --values (8 bytes)
16: /tmp/values172577105 (20 bytes)
17: --values (8 bytes)
18: /tmp/values916913276 (20 bytes)
19: --values (8 bytes)
20: /tmp/values074604459 (20 bytes)
21: --values (8 bytes)
22: /tmp/values985625870 (20 bytes)
23: --values (8 bytes)
24: /tmp/values012472597 (20 bytes)
25: --values (8 bytes)
26: /tmp/values134551920 (20 bytes)
27: --values (8 bytes)
28: /tmp/values892459535 (20 bytes)
29: --values (8 bytes)
30: /tmp/values242396706 (20 bytes)
31: --values (8 bytes)
32: /tmp/values242239257 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-web (27 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-web-deploy-eqiad.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:55:17 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:55:17 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:55:17 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:55:17 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values834134902 (20 bytes)
11: --values (8 bytes)
12: /tmp/values412944477 (20 bytes)
13: --values (8 bytes)
14: /tmp/values738332696 (20 bytes)
15: --values (8 bytes)
16: /tmp/values626268823 (20 bytes)
17: --values (8 bytes)
18: /tmp/values024161546 (20 bytes)
19: --values (8 bytes)
20: /tmp/values272000737 (20 bytes)
21: --values (8 bytes)
22: /tmp/values732152524 (20 bytes)
23: --values (8 bytes)
24: /tmp/values482611131 (20 bytes)
25: --values (8 bytes)
26: /tmp/values882623966 (20 bytes)
27: --values (8 bytes)
28: /tmp/values272738725 (20 bytes)
29: --values (8 bytes)
30: /tmp/values936480960 (20 bytes)
31: --values (8 bytes)
32: /tmp/values028274463 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-web (27 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-web-deploy-codfw.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:55:17 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:55:17 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values651762418 (20 bytes)
11: --values (8 bytes)
12: /tmp/values682028969 (20 bytes)
13: --values (8 bytes)
14: /tmp/values311949556 (20 bytes)
15: --values (8 bytes)
16: /tmp/values187420611 (20 bytes)
17: --values (8 bytes)
18: /tmp/values356884038 (20 bytes)
19: --values (8 bytes)
20: /tmp/values569613037 (20 bytes)
21: --values (8 bytes)
22: /tmp/values483388776 (20 bytes)
23: --values (8 bytes)
24: /tmp/values507011495 (20 bytes)
25: --values (8 bytes)
26: /tmp/values687760090 (20 bytes)
27: --values (8 bytes)
28: /tmp/values783767665 (20 bytes)
29: --values (8 bytes)
30: /tmp/values275361052 (20 bytes)
31: --values (8 bytes)
32: /tmp/values178557899 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-int (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-int-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:55:17 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 Stdout/stderr follows:
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values570625026 (20 bytes)
11: --values (8 bytes)
12: /tmp/values502698361 (20 bytes)
13: --values (8 bytes)
14: /tmp/values719207044 (20 bytes)
15: --values (8 bytes)
16: /tmp/values258438163 (20 bytes)
17: --values (8 bytes)
18: /tmp/values836772950 (20 bytes)
19: --values (8 bytes)
20: /tmp/values218714557 (20 bytes)
21: --values (8 bytes)
22: /tmp/values548983800 (20 bytes)
23: --values (8 bytes)
24: /tmp/values405736183 (20 bytes)
25: --values (8 bytes)
26: /tmp/values196420074 (20 bytes)
27: --values (8 bytes)
28: /tmp/values176606273 (20 bytes)
29: --values (8 bytes)
30: /tmp/values037396652 (20 bytes)
31: --values (8 bytes)
32: /tmp/values923588123 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-ext (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-ext-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Comparing release=canary, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values660640914 (20 bytes)
11: --values (8 bytes)
12: /tmp/values969583305 (20 bytes)
13: --values (8 bytes)
14: /tmp/values573645204 (20 bytes)
15: --values (8 bytes)
16: /tmp/values926781411 (20 bytes)
17: --values (8 bytes)
18: /tmp/values042574822 (20 bytes)
19: --values (8 bytes)
20: /tmp/values283143693 (20 bytes)
21: --values (8 bytes)
22: /tmp/values123203080 (20 bytes)
23: --values (8 bytes)
24: /tmp/values356692423 (20 bytes)
25: --values (8 bytes)
26: /tmp/values909701754 (20 bytes)
27: --values (8 bytes)
28: /tmp/values915039121 (20 bytes)
29: --values (8 bytes)
30: /tmp/values729442748 (20 bytes)
31: --values (8 bytes)
32: /tmp/values542876139 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-jobrunner (33 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:55:17 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:55:17 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values040657674 (20 bytes)
11: --values (8 bytes)
12: /tmp/values972071137 (20 bytes)
13: --values (8 bytes)
14: /tmp/values197298892 (20 bytes)
15: --values (8 bytes)
16: /tmp/values867756475 (20 bytes)
17: --values (8 bytes)
18: /tmp/values772376030 (20 bytes)
19: --values (8 bytes)
20: /tmp/values268758437 (20 bytes)
21: --values (8 bytes)
22: /tmp/values782135488 (20 bytes)
23: --values (8 bytes)
24: /tmp/values564272415 (20 bytes)
25: --values (8 bytes)
26: /tmp/values006390258 (20 bytes)
27: --values (8 bytes)
28: /tmp/values958610089 (20 bytes)
29: --values (8 bytes)
30: /tmp/values877192692 (20 bytes)
31: --values (8 bytes)
32: /tmp/values896076483 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-int (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-int-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:55:17 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values856679463 (20 bytes)
11: --values (8 bytes)
12: /tmp/values269077850 (20 bytes)
13: --values (8 bytes)
14: /tmp/values877114609 (20 bytes)
15: --values (8 bytes)
16: /tmp/values399955868 (20 bytes)
17: --values (8 bytes)
18: /tmp/values113338955 (20 bytes)
19: --values (8 bytes)
20: /tmp/values142533934 (20 bytes)
21: --values (8 bytes)
22: /tmp/values820466357 (20 bytes)
23: --values (8 bytes)
24: /tmp/values054345872 (20 bytes)
25: --values (8 bytes)
26: /tmp/values309807279 (20 bytes)
27: --values (8 bytes)
28: /tmp/values805654082 (20 bytes)
29: --values (8 bytes)
30: /tmp/values225047225 (20 bytes)
31: --values (8 bytes)
32: /tmp/values122975428 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-ext (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-ext-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:55:17 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Stdout/stderr follows:
18:55:17 Comparing release=canary, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values790480040 (20 bytes)
11: --values (8 bytes)
12: /tmp/values520954343 (20 bytes)
13: --values (8 bytes)
14: /tmp/values529525786 (20 bytes)
15: --values (8 bytes)
16: /tmp/values863828657 (20 bytes)
17: --values (8 bytes)
18: /tmp/values221810780 (20 bytes)
19: --values (8 bytes)
20: /tmp/values076885515 (20 bytes)
21: --values (8 bytes)
22: /tmp/values569741294 (20 bytes)
23: --values (8 bytes)
24: /tmp/values596040821 (20 bytes)
25: --values (8 bytes)
26: /tmp/values301005136 (20 bytes)
27: --values (8 bytes)
28: /tmp/values012490351 (20 bytes)
29: --values (8 bytes)
30: /tmp/values621757698 (20 bytes)
31: --values (8 bytes)
32: /tmp/values415655545 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-jobrunner (33 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-jobrunner-deploy-codfw.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:17 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:55:17 K8s deployment to stage canaries failed: K8s deployment had the following errors:
eqiad: Deployment of mw-web-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
codfw: Deployment of mw-web-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:17 Rolling back to prior state...
18:55:18 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:55:18 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:55:18 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:55:18 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:55:18 Started Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:55:18 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:55:18 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 Comparing release=canary, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values438922789 (20 bytes)
11: --values (8 bytes)
12: /tmp/values096911680 (20 bytes)
13: --values (8 bytes)
14: /tmp/values085886367 (20 bytes)
15: --values (8 bytes)
16: /tmp/values032487538 (20 bytes)
17: --values (8 bytes)
18: /tmp/values608404265 (20 bytes)
19: --values (8 bytes)
20: /tmp/values374314100 (20 bytes)
21: --values (8 bytes)
22: /tmp/values673979715 (20 bytes)
23: --values (8 bytes)
24: /tmp/values259899334 (20 bytes)
25: --values (8 bytes)
26: /tmp/values414976109 (20 bytes)
27: --values (8 bytes)
28: /tmp/values380392168 (20 bytes)
29: --values (8 bytes)
30: /tmp/values618633511 (20 bytes)
31: --values (8 bytes)
32: /tmp/values664683610 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-jobrunner (33 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values816121479 (20 bytes)
11: --values (8 bytes)
12: /tmp/values621219386 (20 bytes)
13: --values (8 bytes)
14: /tmp/values195538001 (20 bytes)
15: --values (8 bytes)
16: /tmp/values196895100 (20 bytes)
17: --values (8 bytes)
18: /tmp/values969310379 (20 bytes)
19: --values (8 bytes)
20: /tmp/values922062350 (20 bytes)
21: --values (8 bytes)
22: /tmp/values542748181 (20 bytes)
23: --values (8 bytes)
24: /tmp/values997974640 (20 bytes)
25: --values (8 bytes)
26: /tmp/values674911503 (20 bytes)
27: --values (8 bytes)
28: /tmp/values552591650 (20 bytes)
29: --values (8 bytes)
30: /tmp/values232478233 (20 bytes)
31: --values (8 bytes)
32: /tmp/values089788068 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-web (27 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-web-deploy-eqiad.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values903264634 (20 bytes)
11: --values (8 bytes)
12: /tmp/values737240721 (20 bytes)
13: --values (8 bytes)
14: /tmp/values238214844 (20 bytes)
15: --values (8 bytes)
16: /tmp/values570380523 (20 bytes)
17: --values (8 bytes)
18: /tmp/values751769934 (20 bytes)
19: --values (8 bytes)
20: /tmp/values350140501 (20 bytes)
21: --values (8 bytes)
22: /tmp/values367733168 (20 bytes)
23: --values (8 bytes)
24: /tmp/values340713295 (20 bytes)
25: --values (8 bytes)
26: /tmp/values523664994 (20 bytes)
27: --values (8 bytes)
28: /tmp/values299754073 (20 bytes)
29: --values (8 bytes)
30: /tmp/values351021540 (20 bytes)
31: --values (8 bytes)
32: /tmp/values086792179 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-web (27 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-web-deploy-codfw.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values299944402 (20 bytes)
11: --values (8 bytes)
12: /tmp/values999821577 (20 bytes)
13: --values (8 bytes)
14: /tmp/values585638100 (20 bytes)
15: --values (8 bytes)
16: /tmp/values559503395 (20 bytes)
17: --values (8 bytes)
18: /tmp/values578903846 (20 bytes)
19: --values (8 bytes)
20: /tmp/values550473293 (20 bytes)
21: --values (8 bytes)
22: /tmp/values342442824 (20 bytes)
23: --values (8 bytes)
24: /tmp/values279368711 (20 bytes)
25: --values (8 bytes)
26: /tmp/values108443578 (20 bytes)
27: --values (8 bytes)
28: /tmp/values009558993 (20 bytes)
29: --values (8 bytes)
30: /tmp/values177396476 (20 bytes)
31: --values (8 bytes)
32: /tmp/values421051435 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-int (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-int-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values292556210 (20 bytes)
11: --values (8 bytes)
12: /tmp/values286867305 (20 bytes)
13: --values (8 bytes)
14: /tmp/values904693172 (20 bytes)
15: --values (8 bytes)
16: /tmp/values162573187 (20 bytes)
17: --values (8 bytes)
18: /tmp/values273044230 (20 bytes)
19: --values (8 bytes)
20: /tmp/values651050157 (20 bytes)
21: --values (8 bytes)
22: /tmp/values682261032 (20 bytes)
23: --values (8 bytes)
24: /tmp/values665524071 (20 bytes)
25: --values (8 bytes)
26: /tmp/values028364698 (20 bytes)
27: --values (8 bytes)
28: /tmp/values395892273 (20 bytes)
29: --values (8 bytes)
30: /tmp/values896862172 (20 bytes)
31: --values (8 bytes)
32: /tmp/values867075467 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-ext (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-ext-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e eqiad --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:55:18 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:55:18 Started Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:55:18 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values495726555 (20 bytes)
11: --values (8 bytes)
12: /tmp/values975458686 (20 bytes)
13: --values (8 bytes)
14: /tmp/values025705157 (20 bytes)
15: --values (8 bytes)
16: /tmp/values800190816 (20 bytes)
17: --values (8 bytes)
18: /tmp/values553940287 (20 bytes)
19: --values (8 bytes)
20: /tmp/values596050322 (20 bytes)
21: --values (8 bytes)
22: /tmp/values230755273 (20 bytes)
23: --values (8 bytes)
24: /tmp/values052448404 (20 bytes)
25: --values (8 bytes)
26: /tmp/values101881059 (20 bytes)
27: --values (8 bytes)
28: /tmp/values094920934 (20 bytes)
29: --values (8 bytes)
30: /tmp/values988653837 (20 bytes)
31: --values (8 bytes)
32: /tmp/values056172296 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-ext (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-ext-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 Comparing release=canary, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values009601633 (20 bytes)
11: --values (8 bytes)
12: /tmp/values346832972 (20 bytes)
13: --values (8 bytes)
14: /tmp/values842563387 (20 bytes)
15: --values (8 bytes)
16: /tmp/values099716446 (20 bytes)
17: --values (8 bytes)
18: /tmp/values728562981 (20 bytes)
19: --values (8 bytes)
20: /tmp/values732558400 (20 bytes)
21: --values (8 bytes)
22: /tmp/values303027871 (20 bytes)
23: --values (8 bytes)
24: /tmp/values651106162 (20 bytes)
25: --values (8 bytes)
26: /tmp/values249203753 (20 bytes)
27: --values (8 bytes)
28: /tmp/values783919476 (20 bytes)
29: --values (8 bytes)
30: /tmp/values565495363 (20 bytes)
31: --values (8 bytes)
32: /tmp/values445377734 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-jobrunner (33 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-jobrunner-deploy-codfw.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:55:18 Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Stdout/stderr follows:
18:55:18 helmfile.yaml: basePath=.
Comparing release=canary, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: canary (6 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values836954993 (20 bytes)
11: --values (8 bytes)
12: /tmp/values751627804 (20 bytes)
13: --values (8 bytes)
14: /tmp/values785869003 (20 bytes)
15: --values (8 bytes)
16: /tmp/values441149870 (20 bytes)
17: --values (8 bytes)
18: /tmp/values032603445 (20 bytes)
19: --values (8 bytes)
20: /tmp/values099475728 (20 bytes)
21: --values (8 bytes)
22: /tmp/values644891951 (20 bytes)
23: --values (8 bytes)
24: /tmp/values907320514 (20 bytes)
25: --values (8 bytes)
26: /tmp/values536342329 (20 bytes)
27: --values (8 bytes)
28: /tmp/values866469700 (20 bytes)
29: --values (8 bytes)
30: /tmp/values422898643 (20 bytes)
31: --values (8 bytes)
32: /tmp/values712665878 (20 bytes)
33: --set (5 bytes)
34: php.servergroup=kube-mw-api-int (31 bytes)
35: --detailed-exitcode (19 bytes)
36: --kubeconfig (12 bytes)
37: /etc/kubernetes/mw-api-int-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:55:18 Finished Running helmfile -e codfw --selector name=canary apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:55:18 Caught another exception while trying to roll back. Giving up: K8s deployment had the following errors:
eqiad: Deployment of mw-jobrunner-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-web-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-canary failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
codfw: Deployment of mw-web-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-canary failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=canary', 'apply']' returned non-zero exit status 1.
18:55:18 Finished sync-canaries-k8s (duration: 00m 01s)
18:55:18 Started sync-check-canaries
18:55:49 sync-canaries: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)
18:55:49 Per-host sync duration: average 25.0s, median 25.1s
18:55:49 rsync transfer: average 574,615 bytes/host, total 2,298,460 bytes
18:55:49 Finished Canaries Synced (duration: 00m 30s)
18:55:49 Executing check 'Check endpoints for mw2376.codfw.wmnet'
18:55:49 Executing check 'Check endpoints for mw2374.codfw.wmnet'
18:55:49 Executing check 'Check endpoints for mw2271.codfw.wmnet'
18:55:49 Executing check 'Check endpoints for mw2272.codfw.wmnet'
18:55:50 Finished Canary Endpoint Check Complete (duration: 00m 01s)
18:55:50 Waiting for canary traffic...
18:56:09 Executing check 'Logstash Error rate for mw2376.codfw.wmnet'
18:56:09 Executing check 'Logstash Error rate for mw2374.codfw.wmnet'
18:56:09 Executing check 'Logstash Error rate for mw2271.codfw.wmnet'
18:56:09 Executing check 'Logstash Error rate for mw2272.codfw.wmnet'
18:56:09 Finished sync-check-canaries (duration: 00m 51s)
18:56:09 Started sync-prod-k8s
18:56:10 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:56:10 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:56:10 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:56:10 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:56:10 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:56:10 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values725949000 (20 bytes)
11: --values (8 bytes)
12: /tmp/values663204615 (20 bytes)
13: --values (8 bytes)
14: /tmp/values950998714 (20 bytes)
15: --values (8 bytes)
16: /tmp/values411776721 (20 bytes)
17: --values (8 bytes)
18: /tmp/values304494588 (20 bytes)
19: --values (8 bytes)
20: /tmp/values201829163 (20 bytes)
21: --values (8 bytes)
22: /tmp/values194054286 (20 bytes)
23: --values (8 bytes)
24: /tmp/values913845397 (20 bytes)
25: --values (8 bytes)
26: /tmp/values855393008 (20 bytes)
27: --values (8 bytes)
28: /tmp/values121249679 (20 bytes)
29: --values (8 bytes)
30: /tmp/values536169890 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-int (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-int-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values050282991 (20 bytes)
11: --values (8 bytes)
12: /tmp/values029869186 (20 bytes)
13: --values (8 bytes)
14: /tmp/values313812985 (20 bytes)
15: --values (8 bytes)
16: /tmp/values319223044 (20 bytes)
17: --values (8 bytes)
18: /tmp/values872193683 (20 bytes)
19: --values (8 bytes)
20: /tmp/values272273110 (20 bytes)
21: --values (8 bytes)
22: /tmp/values297753661 (20 bytes)
23: --values (8 bytes)
24: /tmp/values883892856 (20 bytes)
25: --values (8 bytes)
26: /tmp/values916957559 (20 bytes)
27: --values (8 bytes)
28: /tmp/values770321002 (20 bytes)
29: --values (8 bytes)
30: /tmp/values712211649 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-web (27 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-web-deploy-codfw.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values375291335 (20 bytes)
11: --values (8 bytes)
12: /tmp/values139428986 (20 bytes)
13: --values (8 bytes)
14: /tmp/values402553745 (20 bytes)
15: --values (8 bytes)
16: /tmp/values312485308 (20 bytes)
17: --values (8 bytes)
18: /tmp/values110322667 (20 bytes)
19: --values (8 bytes)
20: /tmp/values272519758 (20 bytes)
21: --values (8 bytes)
22: /tmp/values906689365 (20 bytes)
23: --values (8 bytes)
24: /tmp/values866006704 (20 bytes)
25: --values (8 bytes)
26: /tmp/values222628943 (20 bytes)
27: --values (8 bytes)
28: /tmp/values050986338 (20 bytes)
29: --values (8 bytes)
30: /tmp/values163395929 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-ext (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-ext-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values055939004 (20 bytes)
11: --values (8 bytes)
12: /tmp/values497061867 (20 bytes)
13: --values (8 bytes)
14: /tmp/values216640078 (20 bytes)
15: --values (8 bytes)
16: /tmp/values680532309 (20 bytes)
17: --values (8 bytes)
18: /tmp/values164994736 (20 bytes)
19: --values (8 bytes)
20: /tmp/values488069199 (20 bytes)
21: --values (8 bytes)
22: /tmp/values282930018 (20 bytes)
23: --values (8 bytes)
24: /tmp/values938109273 (20 bytes)
25: --values (8 bytes)
26: /tmp/values811140324 (20 bytes)
27: --values (8 bytes)
28: /tmp/values737507059 (20 bytes)
29: --values (8 bytes)
30: /tmp/values182842806 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-web (27 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-web-deploy-eqiad.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values414473856 (20 bytes)
11: --values (8 bytes)
12: /tmp/values720840671 (20 bytes)
13: --values (8 bytes)
14: /tmp/values296655282 (20 bytes)
15: --values (8 bytes)
16: /tmp/values052121449 (20 bytes)
17: --values (8 bytes)
18: /tmp/values337956276 (20 bytes)
19: --values (8 bytes)
20: /tmp/values882152835 (20 bytes)
21: --values (8 bytes)
22: /tmp/values581959430 (20 bytes)
23: --values (8 bytes)
24: /tmp/values110185133 (20 bytes)
25: --values (8 bytes)
26: /tmp/values228937256 (20 bytes)
27: --values (8 bytes)
28: /tmp/values035144551 (20 bytes)
29: --values (8 bytes)
30: /tmp/values751620506 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-jobrunner (33 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:56:10 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:56:10 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:56:10 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:56:10 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values862157108 (20 bytes)
11: --values (8 bytes)
12: /tmp/values356518147 (20 bytes)
13: --values (8 bytes)
14: /tmp/values585751686 (20 bytes)
15: --values (8 bytes)
16: /tmp/values108145709 (20 bytes)
17: --values (8 bytes)
18: /tmp/values789402024 (20 bytes)
19: --values (8 bytes)
20: /tmp/values267202279 (20 bytes)
21: --values (8 bytes)
22: /tmp/values757553946 (20 bytes)
23: --values (8 bytes)
24: /tmp/values292003249 (20 bytes)
25: --values (8 bytes)
26: /tmp/values826960732 (20 bytes)
27: --values (8 bytes)
28: /tmp/values131675403 (20 bytes)
29: --values (8 bytes)
30: /tmp/values746535150 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-jobrunner (33 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-jobrunner-deploy-codfw.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values998013506 (20 bytes)
11: --values (8 bytes)
12: /tmp/values317864633 (20 bytes)
13: --values (8 bytes)
14: /tmp/values915921604 (20 bytes)
15: --values (8 bytes)
16: /tmp/values305176403 (20 bytes)
17: --values (8 bytes)
18: /tmp/values228575894 (20 bytes)
19: --values (8 bytes)
20: /tmp/values999982333 (20 bytes)
21: --values (8 bytes)
22: /tmp/values506012728 (20 bytes)
23: --values (8 bytes)
24: /tmp/values201963575 (20 bytes)
25: --values (8 bytes)
26: /tmp/values796423722 (20 bytes)
27: --values (8 bytes)
28: /tmp/values618552705 (20 bytes)
29: --values (8 bytes)
30: /tmp/values974063852 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-int (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-int-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:56:10 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Stdout/stderr follows:
18:56:10 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values305516317 (20 bytes)
11: --values (8 bytes)
12: /tmp/values922744280 (20 bytes)
13: --values (8 bytes)
14: /tmp/values760915287 (20 bytes)
15: --values (8 bytes)
16: /tmp/values359778506 (20 bytes)
17: --values (8 bytes)
18: /tmp/values156620705 (20 bytes)
19: --values (8 bytes)
20: /tmp/values707495052 (20 bytes)
21: --values (8 bytes)
22: /tmp/values377010299 (20 bytes)
23: --values (8 bytes)
24: /tmp/values115261342 (20 bytes)
25: --values (8 bytes)
26: /tmp/values879085669 (20 bytes)
27: --values (8 bytes)
28: /tmp/values297622144 (20 bytes)
29: --values (8 bytes)
30: /tmp/values639856607 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-ext (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-ext-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:10 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:56:10 K8s deployment to stage production failed: K8s deployment had the following errors:
eqiad: Deployment of mw-api-int-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-web-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
codfw: Deployment of mw-web-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:10 Rolling back to prior state...
18:56:11 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:56:11 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:56:11 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:56:11 Started Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:56:11 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values870381424 (20 bytes)
11: --values (8 bytes)
12: /tmp/values414869007 (20 bytes)
13: --values (8 bytes)
14: /tmp/values121028130 (20 bytes)
15: --values (8 bytes)
16: /tmp/values129508121 (20 bytes)
17: --values (8 bytes)
18: /tmp/values649958820 (20 bytes)
19: --values (8 bytes)
20: /tmp/values730207923 (20 bytes)
21: --values (8 bytes)
22: /tmp/values050972790 (20 bytes)
23: --values (8 bytes)
24: /tmp/values461281629 (20 bytes)
25: --values (8 bytes)
26: /tmp/values887943448 (20 bytes)
27: --values (8 bytes)
28: /tmp/values669484951 (20 bytes)
29: --values (8 bytes)
30: /tmp/values237347338 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-web (27 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-web-deploy-eqiad.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:56:11 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int
18:56:11 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values998273331 (20 bytes)
11: --values (8 bytes)
12: /tmp/values932309238 (20 bytes)
13: --values (8 bytes)
14: /tmp/values968397277 (20 bytes)
15: --values (8 bytes)
16: /tmp/values085344664 (20 bytes)
17: --values (8 bytes)
18: /tmp/values045501463 (20 bytes)
19: --values (8 bytes)
20: /tmp/values257340554 (20 bytes)
21: --values (8 bytes)
22: /tmp/values660719713 (20 bytes)
23: --values (8 bytes)
24: /tmp/values657503820 (20 bytes)
25: --values (8 bytes)
26: /tmp/values252458811 (20 bytes)
27: --values (8 bytes)
28: /tmp/values955576670 (20 bytes)
29: --values (8 bytes)
30: /tmp/values636038437 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-int (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-int-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:56:11 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values309376172 (20 bytes)
11: --values (8 bytes)
12: /tmp/values016346651 (20 bytes)
13: --values (8 bytes)
14: /tmp/values805339838 (20 bytes)
15: --values (8 bytes)
16: /tmp/values309562629 (20 bytes)
17: --values (8 bytes)
18: /tmp/values227472032 (20 bytes)
19: --values (8 bytes)
20: /tmp/values214123391 (20 bytes)
21: --values (8 bytes)
22: /tmp/values019717842 (20 bytes)
23: --values (8 bytes)
24: /tmp/values923317769 (20 bytes)
25: --values (8 bytes)
26: /tmp/values195952084 (20 bytes)
27: --values (8 bytes)
28: /tmp/values584632099 (20 bytes)
29: --values (8 bytes)
30: /tmp/values082934310 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-ext (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-ext-deploy-eqiad.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:56:11 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner
18:56:11 Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values763289001 (20 bytes)
11: --values (8 bytes)
12: /tmp/values556628724 (20 bytes)
13: --values (8 bytes)
14: /tmp/values879726531 (20 bytes)
15: --values (8 bytes)
16: /tmp/values876553286 (20 bytes)
17: --values (8 bytes)
18: /tmp/values906193133 (20 bytes)
19: --values (8 bytes)
20: /tmp/values048691048 (20 bytes)
21: --values (8 bytes)
22: /tmp/values447700903 (20 bytes)
23: --values (8 bytes)
24: /tmp/values842877146 (20 bytes)
25: --values (8 bytes)
26: /tmp/values747439729 (20 bytes)
27: --values (8 bytes)
28: /tmp/values362251036 (20 bytes)
29: --values (8 bytes)
30: /tmp/values000449995 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-jobrunner (33 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-eqiad.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e eqiad --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:56:11 Started Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext
18:56:11 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-int (10 bytes)
9: --values (8 bytes)
10: /tmp/values171117233 (20 bytes)
11: --values (8 bytes)
12: /tmp/values788529244 (20 bytes)
13: --values (8 bytes)
14: /tmp/values367518219 (20 bytes)
15: --values (8 bytes)
16: /tmp/values495481326 (20 bytes)
17: --values (8 bytes)
18: /tmp/values181584501 (20 bytes)
19: --values (8 bytes)
20: /tmp/values595943760 (20 bytes)
21: --values (8 bytes)
22: /tmp/values323335279 (20 bytes)
23: --values (8 bytes)
24: /tmp/values986321154 (20 bytes)
25: --values (8 bytes)
26: /tmp/values523403385 (20 bytes)
27: --values (8 bytes)
28: /tmp/values305731460 (20 bytes)
29: --values (8 bytes)
30: /tmp/values090894099 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-int (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-int-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-int-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-int (duration: 00m 00s)
18:56:11 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-web (6 bytes)
9: --values (8 bytes)
10: /tmp/values481587429 (20 bytes)
11: --values (8 bytes)
12: /tmp/values841569536 (20 bytes)
13: --values (8 bytes)
14: /tmp/values580824159 (20 bytes)
15: --values (8 bytes)
16: /tmp/values417368626 (20 bytes)
17: --values (8 bytes)
18: /tmp/values810534889 (20 bytes)
19: --values (8 bytes)
20: /tmp/values206695476 (20 bytes)
21: --values (8 bytes)
22: /tmp/values280347651 (20 bytes)
23: --values (8 bytes)
24: /tmp/values534009734 (20 bytes)
25: --values (8 bytes)
26: /tmp/values854212909 (20 bytes)
27: --values (8 bytes)
28: /tmp/values077482664 (20 bytes)
29: --values (8 bytes)
30: /tmp/values315302375 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-web (27 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-web-deploy-codfw.config (42 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-web-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-web (duration: 00m 00s)
18:56:11 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-jobrunner (12 bytes)
9: --values (8 bytes)
10: /tmp/values591348860 (20 bytes)
11: --values (8 bytes)
12: /tmp/values960133035 (20 bytes)
13: --values (8 bytes)
14: /tmp/values741745934 (20 bytes)
15: --values (8 bytes)
16: /tmp/values430484245 (20 bytes)
17: --values (8 bytes)
18: /tmp/values932984688 (20 bytes)
19: --values (8 bytes)
20: /tmp/values616343055 (20 bytes)
21: --values (8 bytes)
22: /tmp/values074369058 (20 bytes)
23: --values (8 bytes)
24: /tmp/values291767577 (20 bytes)
25: --values (8 bytes)
26: /tmp/values761082788 (20 bytes)
27: --values (8 bytes)
28: /tmp/values082939059 (20 bytes)
29: --values (8 bytes)
30: /tmp/values225825398 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-jobrunner (33 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-jobrunner-deploy-codfw.config (48 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-jobrunner-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-jobrunner (duration: 00m 00s)
18:56:11 Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Stdout/stderr follows:
18:56:11 helmfile.yaml: basePath=.
skipping missing values file matching "values-main.yaml"
Comparing release=main, chart=wmf-stable/mediawiki
helmfile.yaml: basePath=.
in ./helmfile.yaml: command "/usr/bin/helm3" exited with non-zero status:
PATH:
/usr/bin/helm3
ARGS:
0: helm3 (5 bytes)
1: diff (4 bytes)
2: upgrade (7 bytes)
3: --reset-values (14 bytes)
4: --allow-unreleased (18 bytes)
5: main (4 bytes)
6: wmf-stable/mediawiki (20 bytes)
7: --namespace (11 bytes)
8: mw-api-ext (10 bytes)
9: --values (8 bytes)
10: /tmp/values189570428 (20 bytes)
11: --values (8 bytes)
12: /tmp/values821336235 (20 bytes)
13: --values (8 bytes)
14: /tmp/values210011150 (20 bytes)
15: --values (8 bytes)
16: /tmp/values809170453 (20 bytes)
17: --values (8 bytes)
18: /tmp/values732528240 (20 bytes)
19: --values (8 bytes)
20: /tmp/values806060815 (20 bytes)
21: --values (8 bytes)
22: /tmp/values263427362 (20 bytes)
23: --values (8 bytes)
24: /tmp/values642454041 (20 bytes)
25: --values (8 bytes)
26: /tmp/values689272484 (20 bytes)
27: --values (8 bytes)
28: /tmp/values185055667 (20 bytes)
29: --values (8 bytes)
30: /tmp/values247050102 (20 bytes)
31: --set (5 bytes)
32: php.servergroup=kube-mw-api-ext (31 bytes)
33: --detailed-exitcode (19 bytes)
34: --kubeconfig (12 bytes)
35: /etc/kubernetes/mw-api-ext-deploy-codfw.config (46 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
COMBINED OUTPUT:
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mw-api-ext-deploy-codfw.config
Error: unknown command "diff" for "helm"
Run 'helm --help' for usage.
18:56:11 Finished Running helmfile -e codfw --selector name=main apply in /srv/deployment-charts/helmfile.d/services/mw-api-ext (duration: 00m 00s)
18:56:11 Caught another exception while trying to roll back. Giving up: K8s deployment had the following errors:
eqiad: Deployment of mw-web-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-int-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-main failed: Command '['helmfile', '-e', 'eqiad', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
codfw: Deployment of mw-api-int-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-web-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-jobrunner-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
Deployment of mw-api-ext-main failed: Command '['helmfile', '-e', 'codfw', '--selector', 'name=main', 'apply']' returned non-zero exit status 1.
18:56:11 Finished sync-prod-k8s (duration: 00m 01s)
18:56:11 Started sync-proxies
18:56:17 sync-proxies: 100% (in-flight: 0; ok: 8; fail: 0; left: 0)
18:56:17 Per-host sync duration: average 4.9s, median 4.8s
18:56:17 rsync transfer: average 574,615 bytes/host, total 4,596,920 bytes
18:56:17 Finished sync-proxies (duration: 00m 05s)
18:56:17 Started sync-apaches
18:56:56 sync-apaches: 100% (in-flight: 0; ok: 346; fail: 0; left: 0)
18:56:56 Per-host sync duration: average 5.3s, median 4.5s
18:56:56 rsync transfer: average 574,058 bytes/host, total 198,624,071 bytes
18:56:56 Finished sync-apaches (duration: 00m 38s)
18:56:56 Started scap-cdb-rebuild
18:57:01 scap-cdb-rebuild: 100% (in-flight: 0; ok: 362; fail: 0; left: 0)
18:57:01 Finished scap-cdb-rebuild (duration: 00m 05s)
18:57:01 Started sync_wikiversions
18:57:06 sync_wikiversions: 100% (in-flight: 0; ok: 362; fail: 0; left: 0)
18:57:06 Finished sync_wikiversions (duration: 00m 04s)
18:57:06 Started php-fpm-restarts
18:57:06 Running '/usr/local/sbin/restart-php-fpm-all php7.4-fpm 9223372036854775807' on 301 host(s)
18:59:37 php-fpm-restart: 100% (in-flight: 0; ok: 301; fail: 0; left: 0)
18:59:37 Finished php-fpm-restarts (duration: 02m 31s)
18:59:37 Running /usr/local/bin/mwscript purgeMessageBlobStore.php
18:59:37 Finished scap: testwikis wikis to 1.40.0-wmf.26 refs T330204 (duration: 12m 38s)
18:59:38 deploy-promote failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-world', 'testwikis wikis to 1.40.0-wmf.26 refs T330204', '-D', 'full_image_build:True']' returned non-zero exit status 1.
18:59:38 stage-train failed: <CalledProcessError> Command '['/usr/bin/scap', 'deploy-promote', '--yes', 'testwikis', '1.40.0-wmf.26', '-D', 'full_image_build:True']' returned non-zero exit status 70.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL