Procedure
A simple sudo apt-get install nginx-light is sufficient; nginx will automatically detect the new master process and switch the worker nodes over fairly cleanly.
Clusters
- eqiad
- codfw
- relforge
- cloudelastic
A simple sudo apt-get install nginx-light is sufficient; nginx will automatically detect the new master process and switch the worker nodes over fairly cleanly.
Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:06:35Z] <ryankemper> T287223 Installed nginx-light on all of elastic2* (codfw)
ryankemper@cumin2001:~$ sudo -E cumin 'P{elastic2*}' 'apt list --installed | grep -i nginx' 36 hosts will be targeted: elastic[2025-2060].codfw.wmnet Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36 ===== NODE GROUP ===== (1) elastic2043.codfw.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic] nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed] nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed] ===== NODE GROUP ===== (27) elastic[2025-2042,2044,2046-2053].codfw.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic] libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed,automatic] nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed] ===== NODE GROUP ===== (8) elastic[2045,2054-2060].codfw.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic] libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed] nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed] ================ PASS |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (36/36) [00:01<00:00, 5.49hosts/s] FAIL | | 0% (0/36) [00:01<?, ?hosts/s] 100.0% (36/36) success ratio (>= 100.0% threshold) for command: 'apt list --insta... | grep -i nginx'. 100.0% (36/36) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Here's state after doing codfw
Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:09:25Z] <ryankemper> T287223 Installed nginx-light on all of elastic1* (eqiad)
eqiad state before:
ryankemper@cumin2001:~$ sudo -E cumin 'P{elastic1*}' 'apt list --installed | grep -i nginx' 36 hosts will be targeted: elastic[1032-1067].eqiad.wmnet Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36 ===== NODE GROUP ===== (36) elastic[1032-1067].eqiad.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-auth-pam/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-dav-ext/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-echo/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-geoip/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-image-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-subs-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-upstream-fair/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-http-xslt-filter/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-mail/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] libnginx-mod-stream/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3] nginx-common/now 1.13.9-1+wmf2 all [installed,upgradable to: 1.13.9-1+wmf3] nginx-light/now 1.13.9-1+wmf2 amd64 [installed,upgradable to: 1.13.9-1+wmf3]
eqiad state after:
ryankemper@cumin2001:~$ sudo -E cumin -b 9 'P{elastic1*}' 'apt list --installed | grep -i nginx' 36 hosts will be targeted: elastic[1032-1067].eqiad.wmnet Ok to proceed on 36 hosts? Enter the number of affected hosts to confirm or "q" to quit 36 ===== NODE GROUP ===== (17) elastic[1039,1046,1053-1067].eqiad.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic] libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed] nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed] ===== NODE GROUP ===== (19) elastic[1032-1038,1040-1045,1047-1052].eqiad.wmnet ----- OUTPUT of 'apt list --insta... | grep -i nginx' ----- WARNING: apt does not have a stable CLI interface. Use with caution in scripts. libnginx-mod-http-auth-pam/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-dav-ext/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-echo/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,automatic] libnginx-mod-http-geoip/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-image-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-subs-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-upstream-fair/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-http-xslt-filter/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-mail/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] libnginx-mod-stream/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed,auto-removable] nginx-common/stretch-wikimedia,now 1.13.9-1+wmf3 all [installed,automatic] nginx-light/stretch-wikimedia,now 1.13.9-1+wmf3 amd64 [installed]
Mentioned in SAL (#wikimedia-operations) [2021-07-23T03:11:36Z] <ryankemper> T287223 Installed nginx-light on all of cloudelastic*, and it looks like relforge didn't need the upgrade. This operation is done.