Page MenuHomePhabricator

Remaining nginx packages on some mw servers
Closed, ResolvedPublic

Description

After the switch to Envoy a number of mw hosts have nginx-common installed (along with various libnginx-mod packages), but no nginx binary package providing /usr/sbin/nginx, which leads to a failing nginx.service on startup:

mw1261.eqiad.wmnet (host)
mw1262.eqiad.wmnet (host)
mw1263.eqiad.wmnet (host)
mw1264.eqiad.wmnet (host)
mw1265.eqiad.wmnet (host)
mw1276.eqiad.wmnet (host)
mw1277.eqiad.wmnet (host)
mw1278.eqiad.wmnet (host)
mw1279.eqiad.wmnet (host)
mw1281.eqiad.wmnet (host)
mw1282.eqiad.wmnet (host)
mw1283.eqiad.wmnet (host)
mw1284.eqiad.wmnet (host)
mw1285.eqiad.wmnet (host)
mw1286.eqiad.wmnet (host)
mw1287.eqiad.wmnet (host)
mw1288.eqiad.wmnet (host)
mw1289.eqiad.wmnet (host)
mw1290.eqiad.wmnet (host)
mw1297.eqiad.wmnet (host)
mw1312.eqiad.wmnet (host)
mw1313.eqiad.wmnet (host)
mw1314.eqiad.wmnet (host)
mw1315.eqiad.wmnet (host)
mw1316.eqiad.wmnet (host)
mw1317.eqiad.wmnet (host)
mw1339.eqiad.wmnet (host)
mw1340.eqiad.wmnet (host)
mw1341.eqiad.wmnet (host)
mw1342.eqiad.wmnet (host)
mw1343.eqiad.wmnet (host)
mw1344.eqiad.wmnet (host)
mw1345.eqiad.wmnet (host)
mw1346.eqiad.wmnet (host)
mw1347.eqiad.wmnet (host)
mw1348.eqiad.wmnet (host)
mw1356.eqiad.wmnet (host)
mw1357.eqiad.wmnet (host)
mw1358.eqiad.wmnet (host)
mw1359.eqiad.wmnet (host)
mw1360.eqiad.wmnet (host)
mw1361.eqiad.wmnet (host)
mw1362.eqiad.wmnet (host)
mw1363.eqiad.wmnet (host)
mw1374.eqiad.wmnet (host)
mw1375.eqiad.wmnet (host)
mw1376.eqiad.wmnet (host)
mw1377.eqiad.wmnet (host)
mw1378.eqiad.wmnet (host)
mw1379.eqiad.wmnet (host)
mw1380.eqiad.wmnet (host)
mw1381.eqiad.wmnet (host)
mw1382.eqiad.wmnet (host)
mw1383.eqiad.wmnet (host)
mw1386.eqiad.wmnet (host)
mw1388.eqiad.wmnet (host)
mw1390.eqiad.wmnet (host)
mw1392.eqiad.wmnet (host)
mw1394.eqiad.wmnet (host)
mw1396.eqiad.wmnet (host)
mw1398.eqiad.wmnet (host)
mw1400.eqiad.wmnet (host)
mw1402.eqiad.wmnet (host)
mw1404.eqiad.wmnet (host)
mw1406.eqiad.wmnet (host)
mw1408.eqiad.wmnet (host)
mw1410.eqiad.wmnet (host)
mw1412.eqiad.wmnet (host)
mw2215.codfw.wmnet (host)
mw2216.codfw.wmnet (host)
mw2217.codfw.wmnet (host)
mw2218.codfw.wmnet (host)
mw2219.codfw.wmnet (host)
mw2220.codfw.wmnet (host)
mw2221.codfw.wmnet (host)
mw2222.codfw.wmnet (host)
mw2223.codfw.wmnet (host)
mw2244.codfw.wmnet (host)
mw2245.codfw.wmnet (host)
mw2251.codfw.wmnet (host)
mw2252.codfw.wmnet (host)
mw2253.codfw.wmnet (host)
mw2261.codfw.wmnet (host)
mw2271.codfw.wmnet (host)
mw2272.codfw.wmnet (host)
mw2283.codfw.wmnet (host)
mw2284.codfw.wmnet (host)
mw2285.codfw.wmnet (host)
mw2286.codfw.wmnet (host)
mw2287.codfw.wmnet (host)
mw2288.codfw.wmnet (host)
mw2289.codfw.wmnet (host)
mw2290.codfw.wmnet (host)
mw2291.codfw.wmnet (host)
mw2292.codfw.wmnet (host)
mw2293.codfw.wmnet (host)
mw2294.codfw.wmnet (host)
mw2295.codfw.wmnet (host)
mw2296.codfw.wmnet (host)
mw2297.codfw.wmnet (host)
mw2298.codfw.wmnet (host)
mw2299.codfw.wmnet (host)
mw2300.codfw.wmnet (host)
mw2302.codfw.wmnet (host)
mw2304.codfw.wmnet (host)
mw2306.codfw.wmnet (host)
mw2308.codfw.wmnet (host)
mw2317.codfw.wmnet (host)
mw2318.codfw.wmnet (host)
mw2319.codfw.wmnet (host)
mw2320.codfw.wmnet (host)
mw2321.codfw.wmnet (host)
mw2322.codfw.wmnet (host)
mw2323.codfw.wmnet (host)
mw2324.codfw.wmnet (host)
mw2326.codfw.wmnet (host)
mw2328.codfw.wmnet (host)
mw2330.codfw.wmnet (host)
mw2332.codfw.wmnet (host)
mw2334.codfw.wmnet (host)
mw2350.codfw.wmnet (host)
mw2352.codfw.wmnet (host)
mw2354.codfw.wmnet (host)
mw2356.codfw.wmnet (host)
mw2358.codfw.wmnet (host)
mw2360.codfw.wmnet (host)
mw2362.codfw.wmnet (host)
mw2364.codfw.wmnet (host)
mw2366.codfw.wmnet (host)
mw2368.codfw.wmnet (host)
mw2370.codfw.wmnet (host)
mw2372.codfw.wmnet (host)
mw2374.codfw.wmnet (host)
mw2376.codfw.wmnet (host)

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2020-06-16T14:10:59Z] <moritzm> removing stray nginx packages from mw canaries (mw1261-mw1265 and mw1276-mw1283) T255565

Mentioned in SAL (#wikimedia-operations) [2020-06-17T08:29:49Z] <moritzm> prune nginx from remaining mw* servers in codfw T255565

Mentioned in SAL (#wikimedia-operations) [2020-06-23T15:37:42Z] <moritzm> prune nginx packages on mw1380-mw1412 T255565

Mentioned in SAL (#wikimedia-operations) [2020-06-23T15:47:37Z] <moritzm> prune nginx packages on mwdebug hosts T255565

A few gotchas on this one, noting on the task as it can happen to other roles currently using Nginx when moving to Envoy:

  • The nginx init scripts get marked as a conffile, which means it's still around on normal removal (unless purged) and acts as a systemd unit via the systemd-generator
  • nginx sets up a tmpfs for /var/lib/nginx which breaks the postinst unless it's unmounted via Cumin beforehand

Mentioned in SAL (#wikimedia-operations) [2020-06-24T08:40:30Z] <moritzm> prune remaining nginx packages on mw* servers T255565

MoritzMuehlenhoff claimed this task.

This is complete