Page MenuHomePhabricator

Upgrade to nginx-light package on elastic* hosts
Closed, ResolvedPublic

Description

Procedure

A simple sudo apt-get install nginx-light is sufficient; nginx will automatically detect the new master process and switch the worker nodes over fairly cleanly.

Clusters

  • eqiad
  • codfw
  • relforge
  • cloudelastic

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:06:35Z] <ryankemper> T287223 Installed nginx-light on all of elastic2* (codfw)

ryankemper@cumin2001:~$ sudo -E cumin 'P{elastic2*}' 'apt list --installed | grep -i nginx'
36 hosts will be targeted:
elastic[2025-2060].codfw.wmnet
Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36
===== NODE GROUP =====
(1) elastic2043.codfw.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic]
nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed]
nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]
===== NODE GROUP =====
(27) elastic[2025-2042,2044,2046-2053].codfw.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic]
libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed,automatic]
nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]
===== NODE GROUP =====
(8) elastic[2045,2054-2060].codfw.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic]
libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed]
nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]
================
PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (36/36) [00:01<00:00,  5.49hosts/s]
FAIL |                                                                                                                                                                                                                                                                                                                       |   0% (0/36) [00:01<?, ?hosts/s]
100.0% (36/36) success ratio (>= 100.0% threshold) for command: 'apt list --insta... | grep -i nginx'.
100.0% (36/36) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Here's state after doing codfw

Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:09:25Z] <ryankemper> T287223 Installed nginx-light on all of elastic1* (eqiad)

eqiad state before:

ryankemper@cumin2001:~$ sudo -E cumin 'P{elastic1*}' 'apt list --installed | grep -i nginx'
36 hosts will be targeted:
elastic[1032-1067].eqiad.wmnet
Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36
===== NODE GROUP =====
(36) elastic[1032-1067].eqiad.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-auth-pam/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-dav-ext/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-echo/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-geoip/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-image-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-subs-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-upstream-fair/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-http-xslt-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-mail/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
libnginx-mod-stream/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
nginx-common/now 1.13.9-1+wmf2 all [installed,upgradable to: 1.13.9-1+wmf3]
nginx-light/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]

eqiad state after:

ryankemper@cumin2001:~$ sudo -E cumin -b 9 'P{elastic1*}' 'apt list --installed | grep -i nginx'
36 hosts will be targeted:
elastic[1032-1067].eqiad.wmnet
Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36
===== NODE GROUP =====
(17) elastic[1039,1046,1053-1067].eqiad.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic]
libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed]
nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]
===== NODE GROUP =====
(19) elastic[1032-1038,1040-1045,1047-1052].eqiad.wmnet
----- OUTPUT of 'apt list --insta... | grep -i nginx' -----

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic]
libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable]
nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed,automatic]
nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]

Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:11:36Z] <ryankemper> T287223 Installed nginx-light on all of cloudelastic*, and it looks like relforge didn't need the upgrade. This operation is done.