Page MenuHomePhabricator

Assess Thumbor upgrade options
Closed, ResolvedPublic

Description

Our current Thumbor installation (debian jessie + librsvg 2.40.18-1) is not rendering properly SVGs -> PNG. We believe that librsvg >= 2.43 will fix most of our issues but:

  • Debian stretch's librsvg version is older (2.40.16). The highest version we can use on stretch is librsvg=2.40.20-3
  • We can't port librsvg >= 2.43 to stretch as we will go down a dependancy spiral
  • Debian buster uses librsvg >= 2.43 but is not stable yet (ETA this summer?)
  • (Also) Thumbor is a stateless service and a good candidate for our k8s cluster.

After assessing running Thumbor on stretch with librsvg 2.40.20-3, we have the following options:

1. Upgrade all Thumbor servers to Stretch during Q3 since there is some improvement with librsvg=2.40.20-3. Migrate Thumbor to Buster (probably Q4 or Q1 FY19/20?).

  • Rolling upgrade our 8 Thumbor servers (4 eqiad, 4 codfw)
  • Some puppet changes to support both jessie & stretch
  • Will offer some improvement in rendering untill librsvg >= 2.43
  • This will be temporary, we should put minimum effort in it

2. Hold the upgrade, wait for buster to become stable and migrate Thumbor directly to Buster (probably Q4 or Q1 FY19/20?).

  • This adds another 6 months of having SVG rendering in the current state
  • Thumbor could be the first service we migrate to buster

3. Hold the upgrade, work (part time) on Q3/Q4 to move Thumbor to k8s and buster.

  • This will take longer and an (currently) unknown amount of time due to availability
  • Moving Thumbor to k8s could be a goal for Q4

4. Upgrade servers to stretch (option 1) in Q3, aim to move Thumbor to k8s and buster on Q4.

  • We will be running a version with some improvements until we figure out the k8s transition
  • Moving Thumbor to k8s could be a goal for Q4

Possible test cases: https://phabricator.wikimedia.org/tag/wikimedia-svg-rendering/

Testing
librsvg 2.40.20-3:

  • Get stretch test VM on WMCS
  • Package python-libthumbor:1.3.2-0+wmf1 for stretch
  • Package python-thumbor-community-core: 0.4.0-0wmf1 for stretch
  • Package python-thumbor-wikimedia: 2.2-1 for stretch
  • Package Thumbor: 6.3.2+git20170607-1 for stretch
  • Package librsvg 2.40.20-3 for stretch and upload to component/thumbor
  • Deploy on Beta
  • Test how this works and what are our issues

Event Timeline

jijiki created this task.Nov 19 2018, 9:43 PM
jijiki triaged this task as Normal priority.
jijiki moved this task from Backlog/Radar to Misc on the User-jijiki board.Nov 20 2018, 11:43 AM
jijiki moved this task from Misc to In Progress on the User-jijiki board.

Mentioned in SAL (#wikimedia-operations) [2018-11-20T14:55:14Z] <jijiki> libthumbor_1.3.2-0+wmf1+stretch1 uploaded to stretch-wikimedia T209886

jijiki updated the task description. (Show Details)Nov 20 2018, 2:58 PM
jijiki updated the task description. (Show Details)Dec 10 2018, 10:23 AM

2018-12-05 12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
2018-12-05 16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
2018-12-05 18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia

jijiki updated the task description. (Show Details)Dec 10 2018, 10:27 AM

Looking good!

On Beta thumbnails are generated by the Thumbor instance running on deployment-imagescaler01.deployment-prep.eqiad.wmflabs and deployment-imagescaler02.deployment-prep.eqiad.wmflabs

I would suggest adding a third one with the same naming (which will make it inherit the "project" puppet config) running Stretch, with the appropriate puppet role (role::thumbor::mediawiki).

Once you have that running, the thumbor process should be up on that machine and we can test sending local requests to it to have it generate thumbnails (or fetch existing ones). This should already allow us to test rendering of a variety of files before putting that new server "into rotation" for Beta.

I have setup deployment-imagescaler03 and applied role::thumbor::mediawiki. Please find me on IRC and tell me how I can help :)

jijiki updated the task description. (Show Details)Dec 12 2018, 7:40 PM
jijiki moved this task from Backlog to Doing on the serviceops board.
jijiki updated the task description. (Show Details)Dec 17 2018, 1:40 PM

Change 480077 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] aptrepo: add component/thumbor to stretch-wikimedia

https://gerrit.wikimedia.org/r/480077

Change 480077 merged by Effie Mouzeli:
[operations/puppet@production] aptrepo: add component/thumbor to stretch-wikimedia

https://gerrit.wikimedia.org/r/480077

Mentioned in SAL (#wikimedia-operations) [2018-12-18T16:37:00Z] <jijiki> librsvg* 2.40.20-3+wmf1+stretch1 uploaded to components/thumor to stretch-wikimedia - T209886

jijiki updated the task description. (Show Details)Dec 18 2018, 4:59 PM

@kaldari We have deployed librsvg 2.40.20-3 on deployment-imagescaler03 under debian stretch, after some testing it does't look like SVG rendering is improved. Please have a go as well and let us know:)

@jijiki - Is there a way I can test it, other than locally?

If the files are already on Beta, purge them, make sure your browser cache is cleared for these images, and you'll get thumbnails generated with librsvg 2.40.20-3

I tried doing that on the 3 images you provided before.

The first 2 have improved a little in the sense that the font seems to be rendered the same (before the width of the text varied), but the location of the text is different between both files:

https://commons.wikimedia.beta.wmflabs.org/wiki/File:Rsvg_text_rendering_test_1.svg
https://commons.wikimedia.beta.wmflabs.org/wiki/File:Rsvg_text_rendering_test_2.svg

And there doesn't seem to be any improvement for your third test file:

https://commons.wikimedia.beta.wmflabs.org/wiki/File:Fonttest-Kerning.svg

jijiki added a comment.EditedDec 20 2018, 12:42 PM

@kaldari there is a thumbor debug log on deployment-imagescaler03 under /var/log which was generated as I was testing https://commons.wikimedia.beta.wmflabs.org/wiki/File:Rsvg_text_rendering_test_13.svg

If you don't see anything funny there, how do you think we should proceed?

@jijiki - The first test seems to be hugely improved:
On beta cluster:


On Commons:

It is frustrating though that the kerning bug has been replaced with a position change bug! Still seems to be a big improvement though.

The 2nd test case is also improved, although only when the glyphs are continuous (the left side of the SVG). The kerning for glyphs in separate tspans is still buggy (right side), but that's by far a less common use case.
On beta cluster:


On Commons:

Overall, it seems like a significant improvement, IMO. Of course it's still crap compared with any browser's native SVG rendering :(

There should also be a number of additional test cases in Phab: https://phabricator.wikimedia.org/tag/wikimedia-svg-rendering/

IMHO it makes sense to upgrade to Stretch next (which in the addition to the librsvg bugfixes also brings in refresh of the underlying multimedia library stack, two years worth of distro development). More significant improvements will be available when moving to buster and the Rust-based librsvg in the future as well.

jijiki changed the task status from Open to Stalled.Jan 9 2019, 3:42 PM
jijiki closed this task as Resolved.Jan 9 2019, 3:45 PM
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)Jan 14 2019, 8:53 AM
jijiki reopened this task as Open.
jijiki updated the task description. (Show Details)Jan 14 2019, 8:56 AM
jijiki moved this task from Doing to Externally Blocked on the serviceops board.Jan 18 2019, 12:03 PM
jijiki updated the task description. (Show Details)Jan 24 2019, 1:49 PM
jijiki closed this task as Resolved.Feb 12 2019, 11:10 AM