Sat, Jul 4
Fri, Jul 3
Because bots using OAUTH are not considered logged-in by Pywikibot, those bots will always ignore bot-specific exclusion templates. However, site.username() will return the username. page.botMayEdit should fail safe, which means that it should return False when the username matches, even if the bot is not logged in. That's a very easy fix, and I'll put a patch in momentarily.
Wed, Jul 1
I see two major quality issues with the rendered JPGs: compression artifacts around the text and missing pixels around the text.
Sun, Jun 28
[04:13:29] <AntiComposite> is everything supposed to be bold in new gerrit gitiles? [07:22:26] <andre__> AntiComposite: where exactly (URL) to see "everything in bold"? [07:38:50] <legoktm> andre__: I'm guessing it's on pages like https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master where the entire commit message looks bolded... [07:39:20] <legoktm> I can't tell if it's actually bolded or that's just how the font looks like [07:56:02] <Nemo_bis> Apparently it's because of the use of "Roboto Mono Medium". Once I disable the custom fonts, the page looks fine. [08:41:53] <qchris> AntiComposite: Yes, fonts on gitiles have changed. Before (on previous gerrit) fonts got loaded from Google. That's not what we want for various reasons (E.g.: T240264). So we switched gitiles to the fonts that Gerrit itself provides. [08:41:53] <stashbot> T240264: Gerrit loads font from fonts.googleapis.com and fonts.gstatic.com - https://phabricator.wikimedia.org/T240264 [08:42:04] <qchris> That way we can avoid the loading from 3rd party resources. [08:42:38] <qchris> But on the flip-side: Gerrit does not provide all fonts that we want in all the weights that we want. So gitiles looks a bit different. [08:43:10] <qchris> If it's a concern to you, please file a bug on phabricator so we can discuss with others. [08:49:26] <Nemo_bis> Very nice to avoid loading fonts from third parties. [08:50:05] <Nemo_bis> (I never loaded them in the first place so gitiles was presumably using system fonts for me.)
Was entirely removing CSP headers from $wmgUseCSPReportOnlyHasSession wikis intentional? Because rOMWC203f468dcdf3: Make CSP enforce on beta appears to have had that effect.
Sat, Jun 27
Thumbnail generation is independent of the upload process. Thumbnails are generated on demand by Thumbor after a client request, and MediaWiki is largely uninvolved in thumbnail generation on Wikimedia.
Wed, Jun 24
Tue, Jun 23
Fri, Jun 19
Based on your description, it sounds like you are probably hitting the concurrent thumbnail generation limits. When a thumbnail at a particular size for a particular file is requested for the first time, it is not cached and has to be generated. This can take anywhere from a few hundred milliseconds to a minute, and it's expected that TIF files can take longer to thumbnail. You're most likely hitting the per-IP ratelimit, which in the current configuration will allow 4 new thumbnails to be generated at the request of a specific IP address at a time. Up to 50 additional requests from that IP address will wait in a queue for up to 1 second before they are processed or dropped (returning a 429).
Tue, Jun 16
Sun, Jun 14
Disabling extension changes entirely is probably a bad idea, as it would prevent normalizing .ogg to .oga/.ogv. Unless, of course, MediaWiki decided to strip extensions off and set them to standard values based on MIME type.
Sat, Jun 13
This issue, at least the 503s, is reproducible with Special:Upload (not just the API).
Request from *** via cp1085 frontend, Varnish XID 40264702 Error: 503, Backend fetch failed at Sat, 13 Jun 2020 23:35:18 GMT
HTTP/2 503 Service Unavailable date: Sat, 13 Jun 2020 23:35:18 GMT server: Varnish content-type: text/html; charset=utf-8 age: 0 x-cache: cp1085 int x-cache-status: int-front server-timing: cache;desc="int-front" strict-transport-security: max-age=106384710; includeSubDomains; preload x-client-ip: **** cache-control: private, s-maxage=0, max-age=0, must-revalidate content-length: 1792 X-Firefox-Spdy: h2
Host: commons.wikimedia.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Referer: https://commons.wikimedia.org/wiki/Special:Upload Content-Type: multipart/form-data; boundary=---------------------------239181092836737455363224853218 Content-Length: 2924 Origin: https://commons.wikimedia.org Connection: keep-alive Cookie: **** Upgrade-Insecure-Requests: 1 DNT: 1 Pragma: no-cache Cache-Control: no-cache TE: Trailers
SVG and PDF rendering are both handled by Thumbor on Wikimedia sites. Font dependencies are managed through Puppet in rOPUP modules/mediawiki/manifests/packages/fonts.pp. This file also controls the fonts available on Proton (pdf creator), MediaWiki, and Graphoid (to be undeployed in T242855).
Confirming fixed in buster (rsvg-convert version 2.44.10).
Thu, Jun 11
We don't support TeX for the same base reason that we don't support uploading HTML, markdown, or reStructured Text files: They are all text markup formats, not media formats.
Wed, Jun 10
Tue, Jun 9
I'm also not sure that I agree with that assessment of Special:Upload. It's the only way (except for external tools or gadgets) to upload a file with a custom description template (very common for me), overwrite a file, or upload a file to a wiki other than Commons.
Sun, Jun 7
Jun 5 2020
Jun 3 2020
Jun 2 2020
$ rsvg-convert -w 512 -f png -u -o 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png test.svg $ ls -l 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png -rw-r--r-- 1 root root 0 Jun 2 17:02 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png
$ rsvg-convert -w 512 -f png -u -o 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg The SVG Cluse_de_Chambéry_-_Carte_de_l'occupation_des_sols_(CORINE).svg has no dimensions
Rsvg can't parse the SVG properly, so it generates a 0-byte image, causing ImageMagick to complain and raise a 500 error. The newer version of rsvg gives us an error message, which says something's off with the dimensions.
$ grep -P -o '<svg.*?>' Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg <svg baseProfile="tiny" version="1.2" xmlns:cc="http://creativecommons.org/ns#" width="535.686rmm" xmlns="http://www.w3.org/2000/svg" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" height="578.613mm" viewBox="0 0 2109 2278" xmlns:xlink="http://www.w3.org/1999/xlink">
It seems that somewhere along the line the width got set to 535.686rmm, not 535.686mm. "rmm" is not a valid CSS length unit, so rsvg can not parse the SVG. While web browsers have a series of fallbacks and assumptions to render arbitrary SVGs, librsvg intentionally does not.
Doh! That should be rc_this_oldid not rc_cur_id. rc_cur_id is a key to page_id, not rev_id.
https://quarry.wmflabs.org/query/45496 and the other queries should be correct now. The numbers are a lot lower (which is good) but still extend through the whole month.
Jun 1 2020
The issue seems to be roughly constant per day over the past month (https://quarry.wmflabs.org/query/45496). Doesn't really help figure out when it started, other than knowing it was at least a month ago.
Is there still work that needs to be done to prepare for this, or is it just waiting for someone to get around to handling the necessary configuration changes, testing, and deployment?
May 30 2020
Strike that, finally got a 500.
Request from *** via cp1076 frontend, Varnish XID 780802066 Upstream caches: cp1076 int Error: 500, Internal Server Error at Sat, 30 May 2020 04:52:58 GMT
I wasn't able to reproduce the issue previously because it's in exiv2, not ImageMagick. Testing locally, I get
2020-05-30 05:00:51 thumbor:ERROR ERROR: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 140, in get_image self.context.request.image_url File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run value = future.result() File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1024, in run yielded = self.gen.send(value) File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 609, in _fetch self.context.request.engine.load(fetch_result.buffer, extension) File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 119, in load self.lcl[enginename].load(buffer, extension) File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 167, in load image_or_frames = self.create_image(buffer) File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 77, in create_image self.read_exif(temp_file) File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 146, in read_exif self.exif['Pyexiv2Orientation'] = metadata.get('Exif.Image.Orientation').value File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 190, in _get_value self._compute_value() File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 185, in _compute_value self._value = self._convert_to_python(self._raw_value) File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 311, in _convert_to_python raise ExifValueError(value, self.type) ExifValueError: Invalid value for EXIF type [Short]: 
According to exiftool, there are a bunch of EXIF metadata keys with no values that shouldn't be there:
$ exiftool -verbose Smědavská_hornatina,_Sloupský_potok,_vodopád_01.jpg.1
May 29 2020
The PDF is encoded in such a way that the Identity-UTF16-H CMap is required for Ghostscript to process the text. Ghostscript normally provides that file, but in Debian it is provided by the poppler-data package. In Debian stretch, it does not include the Identity-UTF16-H file, but it is available in Debian buster (Debian bug #861363).
This looks like a bug that was fixed in librsvg 2.41.1:
- The feConvolveMatrix filter primitive wasn't being rendered at all; now it works.
That is not this bug.
May 26 2020
This is because of rounding, and is working as intended.
May 25 2020
Looks like the large-size issue was fixed by a librsvg update, but at small sizes there are still issues:
May 23 2020
https://commons.wikimedia.org/wiki/File:Romulo_Espaldon.jpg and https://commons.wikimedia.org/wiki/File:Jolo_Citation_Romulo_Espaldon.jpg were both uploaded after the patch was deployed, but are still showing the bug:
May 22 2020
The situation in vips is not much better. We use vips directly, which means that configuring it to work in linear space is not easy and probably requires multiple commands and more temporary files. vipsthumbnail has a command-line switch, but may cause other problems (I haven't tested it).
Looks like whatever happened to cause these files to output a blank thumbnail or throw an error is no longer occurring. I can't reproduce the problem locally, and purging the files causes new, working thumbnails to be generated. If there are files that can't be fixed with a purge and bypassing your cache, please file a new task.
This is one of the many known font rendering problems. Part of it is librsvg, but part of it is the font itself. The Wikimedia servers are using liberation 1.07.4-2, which doesn't always produce great results (as you've noticed). Liberation 2 works better, but that's not yet installed. The Noto fonts also work well, so you might want to them or one of the other fonts.