Because bots using OAUTH are not considered logged-in by Pywikibot, those bots will always ignore bot-specific exclusion templates. However, site.username() will return the username. page.botMayEdit should fail safe, which means that it should return False when the username matches, even if the bot is not logged in. That's a very easy fix, and I'll put a patch in momentarily.

I see two major quality issues with the rendered JPGs: compression artifacts around the text and missing pixels around the text.

https://wm-bot.wmflabs.org/logs/%23wikimedia-tech/20200628.txt

[04:13:29] <AntiComposite>	 is everything supposed to be bold in new gerrit gitiles?
[07:22:26] <andre__>	 AntiComposite: where exactly (URL) to see "everything in bold"?
[07:38:50] <legoktm>	 andre__: I'm guessing it's on pages like https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master where the entire commit message looks bolded...
[07:39:20] <legoktm>	 I can't tell if it's actually bolded or that's just how the font looks like
[07:56:02] <Nemo_bis>	 Apparently it's because of the use of "Roboto Mono Medium". Once I disable the custom fonts, the page looks fine.
[08:41:53] <qchris>	 AntiComposite: Yes, fonts on gitiles have changed. Before (on previous gerrit) fonts got loaded from Google. That's not what we want for various reasons (E.g.: T240264). So we switched gitiles to the fonts that Gerrit itself provides.
[08:41:53] <stashbot>	 T240264: Gerrit loads font from fonts.googleapis.com and fonts.gstatic.com - https://phabricator.wikimedia.org/T240264
[08:42:04] <qchris>	 That way we can avoid the loading from 3rd party resources.
[08:42:38] <qchris>	 But on the flip-side: Gerrit does not provide all fonts that we want in all the weights that we want. So gitiles looks a bit different.
[08:43:10] <qchris>	 If it's a concern to you, please file a bug on phabricator so we can discuss with others.
[08:49:26] <Nemo_bis>	 Very nice to avoid loading fonts from third parties.
[08:50:05] <Nemo_bis>	 (I never loaded them in the first place so gitiles was presumably using system fonts for me.)

Was entirely removing CSP headers from $wmgUseCSPReportOnlyHasSession wikis intentional? Because rOMWC203f468dcdf3: Make CSP enforce on beta appears to have had that effect.

Thumbnail generation is independent of the upload process. Thumbnails are generated on demand by Thumbor after a client request, and MediaWiki is largely uninvolved in thumbnail generation on Wikimedia.

In T237700#6248930, @Whatamidoing-WMF wrote:

@AntiCompositeNumber's signature-checking tool gives an error at https://signatures.toolforge.org/check/en.wikipedia.org/EEng The custom sig includes a link to a specific section on the user page. Will this be considered valid?

Based on your description, it sounds like you are probably hitting the concurrent thumbnail generation limits. When a thumbnail at a particular size for a particular file is requested for the first time, it is not cached and has to be generated. This can take anywhere from a few hundred milliseconds to a minute, and it's expected that TIF files can take longer to thumbnail. You're most likely hitting the per-IP ratelimit, which in the current configuration will allow 4 new thumbnails to be generated at the request of a specific IP address at a time. Up to 50 additional requests from that IP address will wait in a queue for up to 1 second before they are processed or dropped (returning a 429).

Disabling extension changes entirely is probably a bad idea, as it would prevent normalizing .ogg to .oga/.ogv. Unless, of course, MediaWiki decided to strip extensions off and set them to standard values based on MIME type.

This issue, at least the 503s, is reproducible with Special:Upload (not just the API).

Request from *** via cp1085 frontend, Varnish XID 40264702
Error: 503, Backend fetch failed at Sat, 13 Jun 2020 23:35:18 GMT

HTTP/2 503 Service Unavailable
date: Sat, 13 Jun 2020 23:35:18 GMT
server: Varnish
content-type: text/html; charset=utf-8
age: 0
x-cache: cp1085 int
x-cache-status: int-front
server-timing: cache;desc="int-front"
strict-transport-security: max-age=106384710; includeSubDomains; preload
x-client-ip: ****
cache-control: private, s-maxage=0, max-age=0, must-revalidate
content-length: 1792
X-Firefox-Spdy: h2

Host: commons.wikimedia.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://commons.wikimedia.org/wiki/Special:Upload
Content-Type: multipart/form-data; boundary=---------------------------239181092836737455363224853218
Content-Length: 2924
Origin: https://commons.wikimedia.org
Connection: keep-alive
Cookie: ****
Upgrade-Insecure-Requests: 1
DNT: 1
Pragma: no-cache
Cache-Control: no-cache
TE: Trailers

SVG and PDF rendering are both handled by Thumbor on Wikimedia sites. Font dependencies are managed through Puppet in rOPUP modules/mediawiki/manifests/packages/fonts.pp. This file also controls the fonts available on Proton (pdf creator), MediaWiki, and Graphoid (to be undeployed in T242855).

Confirming fixed in buster (rsvg-convert version 2.44.10).

We don't support TeX for the same base reason that we don't support uploading HTML, markdown, or reStructured Text files: They are all text markup formats, not media formats.

MediaWiki ensures an execution environment that is capable of methods introduced in JavaScript 1.8 and ECMAScript 5.

I'm also not sure that I agree with that assessment of Special:Upload. It's the only way (except for external tools or gadgets) to upload a file with a custom description template (very common for me), overwrite a file, or upload a file to a wiki other than Commons.

Testing locally:

$ rsvg-convert -w 512 -f png -u -o  512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png test.svg
$ ls -l 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png
-rw-r--r-- 1 root root 0 Jun  2 17:02  512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png

$ rsvg-convert -w 512 -f png -u -o 512px-Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg.png Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg 
The SVG Cluse_de_Chambéry_-_Carte_de_l'occupation_des_sols_(CORINE).svg has no dimensions

Rsvg can't parse the SVG properly, so it generates a 0-byte image, causing ImageMagick to complain and raise a 500 error. The newer version of rsvg gives us an error message, which says something's off with the dimensions.

$ grep -P -o '<svg.*?>' Cluse_de_Chambéry_-_Carte_de_l\'occupation_des_sols_\(CORINE\).svg
<svg baseProfile="tiny" version="1.2" xmlns:cc="http://creativecommons.org/ns#" width="535.686rmm" xmlns="http://www.w3.org/2000/svg" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" height="578.613mm" viewBox="0 0 2109 2278" xmlns:xlink="http://www.w3.org/1999/xlink">

It seems that somewhere along the line the width got set to 535.686rmm, not 535.686mm. "rmm" is not a valid CSS length unit, so rsvg can not parse the SVG. While web browsers have a series of fallbacks and assumptions to render arbitrary SVGs, librsvg intentionally does not.

@Peachey88, is there a reason you removed MediaWiki-Categories from this category-related task?

Doh! That should be rc_this_oldid not rc_cur_id. rc_cur_id is a key to page_id, not rev_id.
https://quarry.wmflabs.org/query/45496 and the other queries should be correct now. The numbers are a lot lower (which is good) but still extend through the whole month.

The issue seems to be roughly constant per day over the past month (https://quarry.wmflabs.org/query/45496). Doesn't really help figure out when it started, other than knowing it was at least a month ago.

Is there still work that needs to be done to prepare for this, or is it just waiting for someone to get around to handling the necessary configuration changes, testing, and deployment?

Strike that, finally got a 500.

Request from *** via cp1076 frontend, Varnish XID 780802066
Upstream caches: cp1076 int
Error: 500, Internal Server Error at Sat, 30 May 2020 04:52:58 GMT

I wasn't able to reproduce the issue previously because it's in exiv2, not ImageMagick. Testing locally, I get

2020-05-30 05:00:51 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 140, in get_image
    self.context.request.image_url
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 1024, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/thumbor/handlers/__init__.py", line 609, in _fetch
    self.context.request.engine.load(fetch_result.buffer, extension)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/proxy/proxy.py", line 119, in load
    self.lcl[enginename].load(buffer, extension)
  File "/usr/lib/python2.7/dist-packages/thumbor/engines/__init__.py", line 167, in load
    image_or_frames = self.create_image(buffer)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 77, in create_image
    self.read_exif(temp_file)
  File "/usr/lib/python2.7/dist-packages/wikimedia_thumbor/engine/imagemagick/imagemagick.py", line 146, in read_exif
    self.exif['Pyexiv2Orientation'] = metadata.get('Exif.Image.Orientation').value
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 190, in _get_value
    self._compute_value()
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 185, in _compute_value
    self._value = self._convert_to_python(self._raw_value)
  File "/usr/lib/python2.7/dist-packages/pyexiv2/exif.py", line 311, in _convert_to_python
    raise ExifValueError(value, self.type)
ExifValueError: Invalid value for EXIF type [Short]: []

According to exiftool, there are a bunch of EXIF metadata keys with no values that shouldn't be there:

$ exiftool -verbose Smědavská_hornatina,_Sloupský_potok,_vodopád_01.jpg.1 

The PDF is encoded in such a way that the Identity-UTF16-H CMap is required for Ghostscript to process the text. Ghostscript normally provides that file, but in Debian it is provided by the poppler-data package. In Debian stretch, it does not include the Identity-UTF16-H file, but it is available in Debian buster (Debian bug #861363).

This looks like a bug that was fixed in librsvg 2.41.1:

• The feConvolveMatrix filter primitive wasn't being rendered at all; now it works.

That is not this bug.

This is because of rounding, and is working as intended.

Looks like the large-size issue was fixed by a librsvg update, but at small sizes there are still issues:

https://commons.wikimedia.org/wiki/File:Romulo_Espaldon.jpg and https://commons.wikimedia.org/wiki/File:Jolo_Citation_Romulo_Espaldon.jpg were both uploaded after the patch was deployed, but are still showing the bug:
https://en.wikipedia.org/wiki/File:Romulo_Espaldon.jpg https://en.wikipedia.org/wiki/File:Jolo_Citation_Romulo_Espaldon.jpg

The situation in vips is not much better. We use vips directly, which means that configuring it to work in linear space is not easy and probably requires multiple commands and more temporary files. vipsthumbnail has a command-line switch, but may cause other problems (I haven't tested it).

Looks like whatever happened to cause these files to output a blank thumbnail or throw an error is no longer occurring. I can't reproduce the problem locally, and purging the files causes new, working thumbnails to be generated. If there are files that can't be fixed with a purge and bypassing your cache, please file a new task.

This is one of the many known font rendering problems. Part of it is librsvg, but part of it is the font itself. The Wikimedia servers are using liberation 1.07.4-2, which doesn't always produce great results (as you've noticed). Liberation 2 works better, but that's not yet installed. The Noto fonts also work well, so you might want to them or one of the other fonts.