HTML document containing various real world test-cases side-by-side
While experimenting with the test page  I realized that for some PNG images quality of the new scaler was superior, for others much worse. Also checking the "bilinear" check-box yielded completely different results depending on content.
I therefore tried to find some sort of systematics by comparing the results for various real world examples (e.g. images used on Wikipedia, scaled to the size the are currently used in the respective articles). I created an HTML page including those test-cases side-by-side to be able to directly compare them (see attachment).
My results so far:
- First of all: Is checking "bilinear" (or the URL parameter bilinear=1) on the test page actually activating bilinear scaling? Since in all my test-cases the "bilinear" version yields a very sharp result, with sharp edges and noticeable steps - something which should actually be avoided by bilinear scaling and isn't visible in any other image editing software I'm using when setting a bilinear scaler (e.g. GIMP). So, if the switch is not inverted for a reason, VIPS incorporates the worst linear scaler I've ever seen.
- All test-cases have in common, that the old-scaler is always a bit smoother than VIPS (regardeless of bilinear setting) - maybe a bit too smooth.
- While the old scaler does not always provide the best quality, it provides always an acceptable quality without any visual flaws. Nothing one could say off any of the VIPS scaler settings.
- Choosing one of VIPS scaler settings is basically impossible. Sometimes "bilinear" looks better, sometimes worse. Sometimes there are serious visual flaws with "bilinear" enabled, sometimes there are serious visual flaws with it disabled.
I also gave some basic notes in the attached HTML document on the issues I see with every graphic. But porbably it's best if you have a look yourself.
In conclusion - judging from those test-cases - I'd still prefer imagemagick currently. I don't know how hard this is, but I'd say the scalers used by VIPS need to be improved before we can deploy them on the WMF Wikis. Otherwise we'd have to accept much worse quality in some cases which should only be considered if we really have serious performance issues currently (which I don't think we have?).