Page MenuHomePhabricator

Remove mediawiki::packages::fonts from non thumbor servers
Closed, ResolvedPublic

Description

Splitting from discussion on https://gerrit.wikimedia.org/r/c/operations/puppet/+/728568

All SVG rendering should happen on thumbor servers, and Timeline/Score have moved to Shellbox which manage their own fonts. I am not aware of any remaining usage of these fonts on appservers.

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptOct 26 2021, 5:57 PM

Change 734798 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: add parameter to allow absenting fonts, remove them from appservers

https://gerrit.wikimedia.org/r/734798

Change 735042 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] wikitech::web: remove font packages from wikitech servers

https://gerrit.wikimedia.org/r/735042

Change 735075 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: purge font packages from mwdebug1001

https://gerrit.wikimedia.org/r/735075

Change 734798 merged by Dzahn:

[operations/puppet@production] mediawiki: allow absenting font packages via hiera

https://gerrit.wikimedia.org/r/734798

Change 735075 merged by Dzahn:

[operations/puppet@production] mediawiki: purge font packages from mwdebug1001

https://gerrit.wikimedia.org/r/735075

Mentioned in SAL (#wikimedia-operations) [2021-10-28T22:16:27Z] <mutante> mwdebug1001 - letting puppet remove all mediawiki font packages using new Hiera key 'profile::mediawiki::webserver::install_fonts: false' to make sure we really don't need them (T294378)

I merged the mediawiki::webserver change to allow absenting fonts via Hiera: https://gerrit.wikimedia.org/r/c/operations/puppet/+/734798/10

Then used it on mwdebug1001 only: https://gerrit.wikimedia.org/r/c/operations/puppet/+/735075/2

You can now use profile::mediawiki::webserver::install_fonts: false to get this:

Info: Caching catalog for mwdebug1001.eqiad.wmnet
Info: Applying configuration version '(53eb64ff96) Dzahn - mediawiki: purge font packages from mwdebug1001'
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arabeyes]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arphic-ukai]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-hinted]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-unhinted]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-arphic-uming]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-farsiweb]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-kacst]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-khmeros]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lao]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-liberation]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-liberation2]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-linuxlibertine]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-manchufont]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-nafees]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-abyssinica]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-ezra]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-padauk]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-scheherazade]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-takao-gothic]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-takao-mincho]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-thai-tlwg]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-tibetan-machine]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-unfonts-core]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-unfonts-extra]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[texlive-fonts-recommended]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-alee]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-wqy-zenhei]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-100dpi]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-75dpi]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-base]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-mplus]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[xfonts-scalable]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-nuosusil]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[culmus]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[culmus-fancy]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lklug-sinhala]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-vlgothic]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-dejavu-core]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-dejavu-extra]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-lyx]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-crosextra-carlito]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-crosextra-caladea]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-smc]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-hosny-amiri]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-taml-tscu]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-beng]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-deva]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-gujr]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-knda]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-mlym]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-orya]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-guru]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-taml]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-telu]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-gujr-extra]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-noto-cjk]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-sil-lateef]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-ipafont-gothic]/ensure: removed
Notice: /Stage[main]/Mediawiki::Packages::Fonts/Package[fonts-ipafont-mincho]/ensure: removed
Notice: Applied catalog in 178.45 seconds

Change 735685 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: remove font packages from all canary appservers

https://gerrit.wikimedia.org/r/735685

Mentioned in SAL (#wikimedia-operations) [2021-11-01T17:06:05Z] <mutante> removing font packages from canary appservers (T294378, gerrit:735685)

Change 735685 merged by Dzahn:

[operations/puppet@production] mediawiki: remove font packages from all canary appservers

https://gerrit.wikimedia.org/r/735685

Change 736017 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] canary_api: remove font packages from canary API servers

https://gerrit.wikimedia.org/r/736017

Change 736017 merged by Dzahn:

[operations/puppet@production] canary_api: remove font packages from canary API servers

https://gerrit.wikimedia.org/r/736017

Change 736020 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] canary_api: actually remove font packages from canary API servers

https://gerrit.wikimedia.org/r/736020

Change 736020 merged by Dzahn:

[operations/puppet@production] canary_api: actually remove font packages from canary API servers

https://gerrit.wikimedia.org/r/736020

Mentioned in SAL (#wikimedia-operations) [2021-11-01T17:46:19Z] <mutante> removing mediawiki font packages from the 8 canary API servers, in addition to 11 canary appservers T294378

Change 736818 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] parsoid::testing: (scandium) remove MediaWiki font packages

https://gerrit.wikimedia.org/r/736818

Change 736818 merged by Dzahn:

[operations/puppet@production] parsoid: remove mw font packages from test server (scandium)

https://gerrit.wikimedia.org/r/736818

Change 736825 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] parsoid: remove mw font packages from canary servers

https://gerrit.wikimedia.org/r/736825

Change 736825 merged by Dzahn:

[operations/puppet@production] parsoid: remove mw font packages from canary servers

https://gerrit.wikimedia.org/r/736825

Change 736925 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] parsoid: remove mw font packages from parsoid-canary, for real

https://gerrit.wikimedia.org/r/736925

Change 736925 merged by Dzahn:

[operations/puppet@production] parsoid: remove mw font packages from parsoid-canary, for real

https://gerrit.wikimedia.org/r/736925

Mentioned in SAL (#wikimedia-operations) [2021-11-04T23:19:54Z] <mutante> wtp1025, wtp1026, parse2001, parse2002 (parsoid-canary): purging mediawiki font packages (T294378)

Change 736927 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] cloudweb2002-dev (labtestwikitech): purge mediawiki font packages

https://gerrit.wikimedia.org/r/736927

Change 736928 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] profile::openstack::base::wikitech::web: add parameter to remove font packages

https://gerrit.wikimedia.org/r/736928

Change 736928 merged by Dzahn:

[operations/puppet@production] profile::openstack::base::wikitech::web: add parameter to remove font packages

https://gerrit.wikimedia.org/r/736928

Change 736927 merged by Dzahn:

[operations/puppet@production] cloudweb2002-dev (labtestwikitech): purge mediawiki font packages

https://gerrit.wikimedia.org/r/736927

Change 735042 merged by Dzahn:

[operations/puppet@production] wikitech::web: remove font packages from wikitech servers

https://gerrit.wikimedia.org/r/735042

Change 737800 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] parsoid: remove mediawiki font packages from all parsoid servers

https://gerrit.wikimedia.org/r/737800

Mentioned in SAL (#wikimedia-operations) [2021-11-10T18:57:44Z] <mutante> removing mediawiki font packages from parsoid hosts - T294378

Change 737800 merged by Dzahn:

[operations/puppet@production] parsoid: remove mediawiki font packages from all parsoid servers

https://gerrit.wikimedia.org/r/737800

Change 738031 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: remove font packages from API appservers

https://gerrit.wikimedia.org/r/738031

Change 738031 merged by Dzahn:

[operations/puppet@production] mediawiki: remove font packages from API appservers

https://gerrit.wikimedia.org/r/738031

Mentioned in SAL (#wikimedia-operations) [2021-11-15T19:06:35Z] <mutante> removing font packages from MW API appservers T294378

Change 739002 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: remove font packages from appservers

https://gerrit.wikimedia.org/r/739002

Change 739002 merged by Dzahn:

[operations/puppet@production] mediawiki: remove font packages from appservers

https://gerrit.wikimedia.org/r/739002

Mentioned in SAL (#wikimedia-operations) [2021-11-15T21:19:46Z] <mutante> removing mediawiki font packages from remaining regular appservers globally (T294378)

Change 739012 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki/parsoid/wikitech: flip default for font install

https://gerrit.wikimedia.org/r/739012

Change 739012 merged by Dzahn:

[operations/puppet@production] mediawiki/parsoid/wikitech: flip default for font install

https://gerrit.wikimedia.org/r/739012

reopening

While we have purged all the font packages that were specifically in the config, these had also pulled in more font packages as dependencies that were not specifically in our list.

Since these unlisted fonts now had nothing depending on them anymore, an apt-get autoremove shows a bunch of font packages to be removed, as reported by @Joe today.

We should also purge those.

example:

a font that was in our list, fonts-alee is properly gone: https://debmonitor.wikimedia.org/packages/fonts-alee (except on thumbor, expected)

a font that was not in our list, fonts-kalapi is still there https://debmonitor.wikimedia.org/packages/fonts-kalapi

Mentioned in SAL (#wikimedia-operations) [2022-01-11T20:56:00Z] <mutante> mw1418 (lowest numbered canary appserver that we use for httpbb hourly tests on cumin1001) - apt-get autoremove - removed font* and python3* packages - reason: T294378

Doing the apt-get autoremove and accepting what it suggests also removes python packages in addition to font packages.

When running puppet afterwards the following python packages are getting reinstalled though:

Notice: /Stage[main]/Scap/Package[python-psutil]/ensure: created (corrective)
Notice: /Stage[main]/Scap/Package[python-yaml]/ensure: created (corrective)
Notice: /Stage[main]/Scap/Package[python-requests]/ensure: created (corrective)
Notice: /Stage[main]/Scap/Package[python-jinja2]/ensure: created (corrective)

That is unrelated to the font package issue but means we should not just run apt-get autoremove via cumin but more specifically wipe only font packages.

Mentioned in SAL (#wikimedia-operations) [2022-01-11T21:29:50Z] <mutante> mw1418 - apt-get remove --purge fonts*; apt-get remove --purge xfonts*; running puppet - nothing gets reinstalled and with --purge it means 'dpkg -l | grep fonts' is actually empty, not full of "rc" still - T294378

Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:34:57Z] <mutante> mw2271, mw2272, mw2251, mw2252 (appserver and API canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)

Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:40:29Z] <mutante> mw2278, mw2279, mw2374, mw2376 (API and jobrunner canaries codfw) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)

Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:46:28Z] <mutante> parse2001, parse2002, wtp1025, wtp1026 (all parsoid canaries - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)

Mentioned in SAL (#wikimedia-operations) [2022-01-17T15:54:29Z] <mutante> mw1414,mw1415,mw1416,mw1417,mw1418,mw1447,mw1448,mw1449,mw1450,mw1437,mw1438 (all canaries eqiad) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* (T294378)

I checked the "status code 5xx" graph in the "RED dashboard" for appservers, looked at the last 30 days, especially if anything went up since Jan 17 when I removed the packages above.

And I can't see any obvious change there. This was for appserver, API_appserver and parsoid.

Mentioned in SAL (#wikimedia-operations) [2022-01-26T19:53:37Z] <mutante> labweb1001, labweb1002, cloudweb2001-dev (wikitech hosts) - apt-get remove --purge fonts*; apt-get remove --purge xfonts* | purging font packages that had been installed as dependencies (T294378)

purged remaining font files on mw13*

Mentioned in SAL (#wikimedia-operations) [2022-01-27T19:36:18Z] <mutante> purging font* / xfont* packages from further eqiad appservers (mw14*) for T294378

purged from all of parsoid (wtp* and parse*) and the rest of eqiad (mw14*)

Mentioned in SAL (#wikimedia-operations) [2022-01-28T21:47:21Z] <mutante> purging font packages from remaining appservers in codfw mw23* ranges.. T294378

Mentioned in SAL (#wikimedia-operations) [2022-01-28T21:52:45Z] <mutante> purging font packages from mwdebug* and scandium* T294378

@Joe All the font* and xfont* packages are gone. from mw1, mw2, wtp, parse, labweb, scandium etc

For example https://debmonitor.wikimedia.org/packages/fonts-kalapi should be empty once debmonitor updates the next time.

Also:

[cumin1001:~] $  sudo cumin mw1* 'dpkg -l | grep fonts'
155 hosts will be targeted:
mw[1302-1456].eqiad.wmnet
Ok to proceed on 155 hosts? Enter the number of affected hosts to confirm or "q" to quit 155
===== NODE GROUP =====                                                                                                                                                                                             
(22) mw[1302-1311,1318,1334-1338,1437-1440,1445-1446].eqiad.wmnet                                                                                                                                                  
----- OUTPUT of 'dpkg -l | grep fonts' -----                                                                                                                                                                       
ii  libwoff1:amd64                       1.0.2-1                                                             amd64        library for converting fonts to WOFF 2.0                                                 
ii  ttf-bitstream-vera                   1.10-8                                                              all          The Bitstream Vera family of free TrueType fonts                                         
===== NODE GROUP =====                                                                                                                                                                                             
(133) mw[1312-1317,1319-1333,1339-1436,1441-1444,1447-1456].eqiad.wmnet                                                                                                                                            
----- OUTPUT of 'dpkg -l | grep fonts' -----                                                                                                                                                                       
ii  libwoff1:amd64                       1.0.2-1                                                             amd64        library for converting fonts to WOFF 2.0                                                 
rc  texlive-fonts-recommended            2018.20190227-2                                                     all          TeX Live: Recommended fonts                                                              
ii  ttf-bitstream-vera                   1.10-8                                                              all          The Bitstream Vera family of free TrueType fonts
================                                                                                                                                                                                                   
PASS |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 100% (155/155) [00:01<00:00, 87.71hosts/s]          
FAIL |                                                                                                                                                                 |   0% (0/155) [00:01<?, ?hosts/s]
100.0% (155/155) success ratio (>= 100.0% threshold) for command: 'dpkg -l | grep fonts'.
100.0% (155/155) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
[cumin1001:~] $  sudo cumin mw2* 'dpkg -l | grep fonts'
157 hosts will be targeted:
mw[2251-2255,2257-2279,2281-2339,2350-2419].codfw.wmnet
Ok to proceed on 157 hosts? Enter the number of affected hosts to confirm or "q" to quit 157
===== NODE GROUP =====                                                                                                                                                                                             
(21) mw[2259-2260,2263-2267,2278-2279,2281-2282,2351,2353,2355,2357,2381-2382,2394-2395,2410-2411].codfw.wmnet                                                                                                     
----- OUTPUT of 'dpkg -l | grep fonts' -----                                                                                                                                                                       
ii  libwoff1:amd64                       1.0.2-1                                                             amd64        library for converting fonts to WOFF 2.0                                                 
ii  ttf-bitstream-vera                   1.10-8                                                              all          The Bitstream Vera family of free TrueType fonts                                         
===== NODE GROUP =====                                                                                                                                                                                             
(128) mw[2251-2255,2257-2258,2261-2262,2268-2277,2283-2339,2350,2352,2354,2356,2358-2380,2383-2393,2396-2409].codfw.wmnet                                                                                          
----- OUTPUT of 'dpkg -l | grep fonts' -----                                                                                                                                                                       
ii  libwoff1:amd64                       1.0.2-1                                                             amd64        library for converting fonts to WOFF 2.0                                                 
rc  texlive-fonts-recommended            2018.20190227-2                                                     all          TeX Live: Recommended fonts                                                              
ii  ttf-bitstream-vera                   1.10-8                                                              all          The Bitstream Vera family of free TrueType fonts
================                                                                                                                                                                                                   
PASS |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Ž       |  95% (149/157) [00:02<00:00, 51.48hosts/s]          
FAIL |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–Š                                                                                                                                                 |   5% (8/157) [00:02<00:53,  2.76hosts/s]
5.1% (8/157) of nodes failed to execute command 'dpkg -l | grep fonts': mw[2412-2419].codfw.wmnet
94.9% (149/157) success ratio (< 100.0% threshold) for command: 'dpkg -l | grep fonts'. Aborting.: mw[2251-2255,2257-2279,2281-2339,2350-2411].codfw.wmnet
94.9% (149/157) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.: mw[2251-2255,2257-2279,2281-2339,2350-2411].codfw.wmnet

ttf-bitstream-vera is there because it gets installed if you remove fonts-dejavu-core.

The following packages will be REMOVED:
  fonts-dejavu-core*
The following NEW packages will be installed:
  ttf-bitstream-vera

I think we can close this again now.

I did not see a change in MW exceptions over the last couple days either.

@Joe The one package I was talking about is "ttf-bitstream-vera" which gets installed when you remove "fonts-dejavu-core*" and since I did "--purge fonts*" I caused this new install. But besides that..nothing called fonts* or xfonts* or ttf-* on any mw* etc. resolved?