Splitting from discussion on https://gerrit.wikimedia.org/r/c/operations/puppet/+/728568
All SVG rendering should happen on thumbor servers, and Timeline/Score have moved to Shellbox which manage their own fonts. I am not aware of any remaining usage of these fonts on appservers.
Change 734798 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki: add parameter to allow absenting fonts, remove them from appservers
Change 735042 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] wikitech::web: remove font packages from wikitech servers
Change 735075 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki: purge font packages from mwdebug1001
Change 734798 merged by Dzahn:
[operations/puppet@production] mediawiki: allow absenting font packages via hiera
Change 735075 merged by Dzahn:
[operations/puppet@production] mediawiki: purge font packages from mwdebug1001
Mentioned in SAL (#wikimedia-operations) [2021-10-28T22:16:27Z] <mutante> mwdebug1001 - letting puppet remove all mediawiki font packages using new Hiera key 'profile::mediawiki::webserver::install_fonts: false' to make sure we really don't need them (T294378)
I merged the mediawiki::webserver change to allow absenting fonts via Hiera: https://gerrit.wikimedia.org/r/c/operations/puppet/+/734798/10
Then used it on mwdebug1001 only: https://gerrit.wikimedia.org/r/c/operations/puppet/+/735075/2
You can now use profile::mediawiki::webserver::install_fonts: false to get this:
Info: Caching catalog for mwdebug1001.eqiad.wmnet Info: Applying configuration version '(53eb64ff96) Dzahn - mediawiki: purge font packages from mwdebug1001' Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arabeyes]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arphic-ukai]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-hinted]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-unhinted]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arphic-uming]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-farsiweb]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-kacst]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-khmeros]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lao]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-liberation]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-liberation2]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-linuxlibertine]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-manchufont]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-nafees]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-abyssinica]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-ezra]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-padauk]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-scheherazade]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-takao-gothic]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-takao-mincho]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-thai-tlwg]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-tibetan-machine]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-unfonts-core]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-unfonts-extra]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[texlive-fonts-recommended]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-alee]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-wqy-zenhei]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-100dpi]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-75dpi]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-base]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-mplus]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-scalable]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-nuosusil]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[culmus]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[culmus-fancy]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lklug-sinhala]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-vlgothic]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-dejavu-core]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-dejavu-extra]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lyx]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-crosextra-carlito]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-crosextra-caladea]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-smc]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-hosny-amiri]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-taml-tscu]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-beng]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-deva]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-gujr]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-knda]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-mlym]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-orya]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-guru]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-taml]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-telu]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-gujr-extra]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-cjk]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-lateef]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-ipafont-gothic]/ensure: removed Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-ipafont-mincho]/ensure: removed Notice: Applied catalog in 178.45 seconds
Change 735685 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki: remove font packages from all canary appservers
Mentioned in SAL (#wikimedia-operations) [2021-11-01T17:06:05Z] <mutante> removing font packages from canary appservers (T294378, gerrit:735685)
Change 735685 merged by Dzahn:
[operations/puppet@production] mediawiki: remove font packages from all canary appservers
Change 736017 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] canary_api: remove font packages from canary API servers
Change 736017 merged by Dzahn:
[operations/puppet@production] canary_api: remove font packages from canary API servers
Change 736020 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] canary_api: actually remove font packages from canary API servers
Change 736020 merged by Dzahn:
[operations/puppet@production] canary_api: actually remove font packages from canary API servers
Mentioned in SAL (#wikimedia-operations) [2021-11-01T17:46:19Z] <mutante> removing mediawiki font packages from the 8 canary API servers, in addition to 11 canary appservers T294378
Change 736818 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] parsoid::testing: (scandium) remove MediaWiki font packages
Change 736818 merged by Dzahn:
[operations/puppet@production] parsoid: remove mw font packages from test server (scandium)
Change 736825 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] parsoid: remove mw font packages from canary servers
Change 736825 merged by Dzahn:
[operations/puppet@production] parsoid: remove mw font packages from canary servers
Change 736925 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] parsoid: remove mw font packages from parsoid-canary, for real
Change 736925 merged by Dzahn:
[operations/puppet@production] parsoid: remove mw font packages from parsoid-canary, for real
Mentioned in SAL (#wikimedia-operations) [2021-11-04T23:19:54Z] <mutante> wtp1025, wtp1026, parse2001, parse2002 (parsoid-canary): purging mediawiki font packages (T294378)
Change 736927 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] cloudweb2002-dev (labtestwikitech): purge mediawiki font packages
Change 736928 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] profile::openstack::base::wikitech::web: add parameter to remove font packages
Change 736928 merged by Dzahn:
[operations/puppet@production] profile::openstack::base::wikitech::web: add parameter to remove font packages
Change 736927 merged by Dzahn:
[operations/puppet@production] cloudweb2002-dev (labtestwikitech): purge mediawiki font packages
Change 735042 merged by Dzahn:
[operations/puppet@production] wikitech::web: remove font packages from wikitech servers
Change 737800 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] parsoid: remove mediawiki font packages from all parsoid servers
Mentioned in SAL (#wikimedia-operations) [2021-11-10T18:57:44Z] <mutante> removing mediawiki font packages from parsoid hosts - T294378
Change 737800 merged by Dzahn:
[operations/puppet@production] parsoid: remove mediawiki font packages from all parsoid servers
Change 738031 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki: remove font packages from API appservers
Change 738031 merged by Dzahn:
[operations/puppet@production] mediawiki: remove font packages from API appservers
Mentioned in SAL (#wikimedia-operations) [2021-11-15T19:06:35Z] <mutante> removing font packages from MW API appservers T294378
Change 739002 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki: remove font packages from appservers
Change 739002 merged by Dzahn:
[operations/puppet@production] mediawiki: remove font packages from appservers
Mentioned in SAL (#wikimedia-operations) [2021-11-15T21:19:46Z] <mutante> removing mediawiki font packages from remaining regular appservers globally (T294378)
Change 739012 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] mediawiki/parsoid/wikitech: flip default for font install
Change 739012 merged by Dzahn:
[operations/puppet@production] mediawiki/parsoid/wikitech: flip default for font install
reopening
While we have purged all the font packages that were specifically in the config, these had also pulled in more font packages as dependencies that were not specifically in our list.
Since these unlisted fonts now had nothing depending on them anymore, an apt-get autoremove shows a bunch of font packages to be removed, as reported by @Joe today.
We should also purge those.
example:
a font that was in our list, fonts-alee is properly gone: https://debmonitor.wikimedia.org/packages/fonts-alee (except on thumbor, expected)
a font that was not in our list, fonts-kalapi is still there https://debmonitor.wikimedia.org/packages/fonts-kalapi
Mentioned in SAL (#wikimedia-operations) [2022-01-11T20:56:00Z] <mutante> mw1418 (lowest numbered canary appserver that we use for httpbb hourly tests on cumin1001) - apt-get autoremove - removed font* and python3* packages - reason: T294378
Doing the apt-get autoremove and accepting what it suggests also removes python packages in addition to font packages.
When running puppet afterwards the following python packages are getting reinstalled though:
Notice: /Stage[main]/Scap/Package[python-psutil]/ensure: created (corrective) Notice: /Stage[main]/Scap/Package[python-yaml]/ensure: created (corrective) Notice: /Stage[main]/Scap/Package[python-requests]/ensure: created (corrective) Notice: /Stage[main]/Scap/Package[python-jinja2]/ensure: created (corrective)
That is unrelated to the font package issue but means we should not just run apt-get autoremove via cumin but more specifically wipe only font packages.
Mentioned in SAL (#wikimedia-operations) [2022-01-11T21:29:50Z] <mutante> mw1418 - apt-get remove --purge fonts*; apt-get remove --purge xfonts*; running puppet - nothing gets reinstalled and with --purge it means 'dpkg -l | grep fonts' is actually empty, not full of "rc" still - T294378
Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:34:57Z] <mutante> mw2271, mw2272, mw2251, mw2252 (appserver and API canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)
Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:40:29Z] <mutante> mw2278, mw2279, mw2374, mw2376 (API and jobrunner canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)
Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:46:28Z] <mutante> parse2001, parse2002, wtp1025, wtp1026 (all parsoid canaries - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)
Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:54:29Z] <mutante> mw1414,mw1415,mw1416,mw1417,mw1418,mw1447,mw1448,mw1449,mw1450,mw1437,mw1438 (all canaries eqiad) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)
I checked the "status code 5xx" graph in the "RED dashboard" for appservers, looked at the last 30 days, especially if anything went up since Jan 17 when I removed the packages above.
And I can't see any obvious change there. This was for appserver, API_appserver and parsoid.
Mentioned in SAL (#wikimedia-operations) [2022-01-26T19:53:37Z] <mutante> labweb1001, labweb1002, cloudweb2001-dev (wikitech hosts) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* | purging font packages that had been installed as dependencies (T294378)
Mentioned in SAL (#wikimedia-operations) [2022-01-27T19:36:18Z] <mutante> purging font* / xfont* packages from further eqiad appservers (mw14*) for T294378
Mentioned in SAL (#wikimedia-operations) [2022-01-28T21:47:21Z] <mutante> purging font packages from remaining appservers in codfw mw23* ranges.. T294378
Mentioned in SAL (#wikimedia-operations) [2022-01-28T21:52:45Z] <mutante> purging font packages from mwdebug* and scandium* T294378
@Joe All the font* and xfont* packages are gone. from mw1, mw2, wtp, parse, labweb, scandium etc
For example https://debmonitor.wikimedia.org/packages/fonts-kalapi should be empty once debmonitor updates the next time.
Also:
[cumin1001:~] $ sudo cumin mw1* 'dpkg -l | grep fonts' 155 hosts will be targeted: mw[1302-1456].eqiad.wmnet Ok to proceed on 155 hosts? Enter the number of affected hosts to confirm or "q" to quit 155 ===== NODE GROUP ===== (22) mw[1302-1311,1318,1334-1338,1437-1440,1445-1446].eqiad.wmnet ----- OUTPUT of 'dpkg -l | grep fonts' ----- ii libwoff1:amd64 1.0.2-1 amd64 library for converting fonts to WOFF 2.0 ii ttf-bitstream-vera 1.10-8 all The Bitstream Vera family of free TrueType fonts ===== NODE GROUP ===== (133) mw[1312-1317,1319-1333,1339-1436,1441-1444,1447-1456].eqiad.wmnet ----- OUTPUT of 'dpkg -l | grep fonts' ----- ii libwoff1:amd64 1.0.2-1 amd64 library for converting fonts to WOFF 2.0 rc texlive-fonts-recommended 2018.20190227-2 all TeX Live: Recommended fonts ii ttf-bitstream-vera 1.10-8 all The Bitstream Vera family of free TrueType fonts ================ PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (155/155) [00:01<00:00, 87.71hosts/s] FAIL | | 0% (0/155) [00:01<?, ?hosts/s] 100.0% (155/155) success ratio (>= 100.0% threshold) for command: 'dpkg -l | grep fonts'. 100.0% (155/155) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. [cumin1001:~] $ sudo cumin mw2* 'dpkg -l | grep fonts' 157 hosts will be targeted: mw[2251-2255,2257-2279,2281-2339,2350-2419].codfw.wmnet Ok to proceed on 157 hosts? Enter the number of affected hosts to confirm or "q" to quit 157 ===== NODE GROUP ===== (21) mw[2259-2260,2263-2267,2278-2279,2281-2282,2351,2353,2355,2357,2381-2382,2394-2395,2410-2411].codfw.wmnet ----- OUTPUT of 'dpkg -l | grep fonts' ----- ii libwoff1:amd64 1.0.2-1 amd64 library for converting fonts to WOFF 2.0 ii ttf-bitstream-vera 1.10-8 all The Bitstream Vera family of free TrueType fonts ===== NODE GROUP ===== (128) mw[2251-2255,2257-2258,2261-2262,2268-2277,2283-2339,2350,2352,2354,2356,2358-2380,2383-2393,2396-2409].codfw.wmnet ----- OUTPUT of 'dpkg -l | grep fonts' ----- ii libwoff1:amd64 1.0.2-1 amd64 library for converting fonts to WOFF 2.0 rc texlive-fonts-recommended 2018.20190227-2 all TeX Live: Recommended fonts ii ttf-bitstream-vera 1.10-8 all The Bitstream Vera family of free TrueType fonts ================ PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 95% (149/157) [00:02<00:00, 51.48hosts/s] FAIL |███████▊ | 5% (8/157) [00:02<00:53, 2.76hosts/s] 5.1% (8/157) of nodes failed to execute command 'dpkg -l | grep fonts': mw[2412-2419].codfw.wmnet 94.9% (149/157) success ratio (< 100.0% threshold) for command: 'dpkg -l | grep fonts'. Aborting.: mw[2251-2255,2257-2279,2281-2339,2350-2411].codfw.wmnet 94.9% (149/157) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.: mw[2251-2255,2257-2279,2281-2339,2350-2411].codfw.wmnet
ttf-bitstream-vera is there because it gets installed if you remove fonts-dejavu-core.
The following packages will be REMOVED: fonts-dejavu-core* The following NEW packages will be installed: ttf-bitstream-vera
I think we can close this again now.
@Joe The one package I was talking about is "ttf-bitstream-vera" which gets installed when you remove "fonts-dejavu-core*" and since I did "--purge fonts*" I caused this new install. But besides that..nothing called fonts* or xfonts* or ttf-* on any mw* etc. resolved?