I don't have a multi-wiki test set up, but looking at git log I'm guessing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/935143 (bf3cc4628ad4289bfe6d74caf3c1e07535fe0c6a).

Deployed, pages load, no errors in logs.

NB: The repo for the deployed webservice is https://gerrit.wikimedia.org/g/labs/tools/stewardbots-ui/+/refs/heads/master, not https://gerrit.wikimedia.org/r/plugins/gitiles/labs/tools/stewardbots/+/refs/heads/master/stewardbots-ui/ after T308256: Split stewardbots-ui into own repository. Probably should do something about that.

Yes, the documentation for StewardBot and the SULWatchers is now in the main stewardbots webservice.

As usual, this resolved itself without intervention after a few days. However, every time it happens it prevents us from appropriately responding to abuse. Given that there is no reason to think it will not reoccur, I am leaving this task open.

Superset is interesting, but I agree with Bawolff that simple is a good goal. Superset requires that I find the correct database server to query against, which is unnecessary and confusing.

Reason and new name definitely need to be suppressable. Notes are stored on public wiki pages, and should be handled by the existing system. I don't think a level below suppression is needed.

Looking at https://k8s-status.toolforge.org/namespaces/tool-bullseye/pods/bullseye-6bfb678845-6qh58/, it looks like bullseye runs off the toolforge-python39-sssd-web container. My tools, which also run python39, don't set the header, so I don't think it's a Toolforge issue. My tools run Flask, but bullseye is Django.

is there a specific reason the tile viewer at https://maps.wikimedia.org/ was broken?

In general, IP addresses should still be sent (or at least forms should be configurable to allow IP addresses to be sent). This information is necessary for the Stewards IP block appeal and exemption workflow.

Agreed that trying to minimize data collection beyond what was previously collected is counterproductive.

The right thing to do is avoid locales entirely and pass the langtag through --accept-languages, but that must wait for next version of the operating system.

I have no objections, though I have not tested this patch since I submitted it. I will note that there is only one XCF test case in the Thumbor integration tests. However, because of the limitations of both the xcftools and imagemagick implementations, I'd say our support for the format is already minimal, so any changes are low-risk.

We do scanning upon upload

Unfortunately, this is probably not sufficient. We know there are originals on Commons that predate the scanner that contain JavaScript.

re xcftools: see https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/619864, T260285: Use ImageMagick for XCF rendering instead of xcftools. That patch will probably need a few revisions because of the Thumbor changes in the last two years.

Writing to files, even in memory /tmp, is usually slower than writing directly to stdout. This is why upstream Thumbor, and our customizations, avoid it wherever possible.

action=purge on the file description page is *supposed* to work, it just sometimes doesn't for reasons that are impossible to determine from the outside.

In general, a 4XX response code would be incorrect for thumbnail requests for already-uploaded images. There is nothing that the HTTP client can do about a bad file having been uploaded.

I'm not sure this is the best idea. rel=me implies some level of control over the linking page, which generally shouldn't be applied to a wiki page. RealMe requires userpage links to be listed in a user preference for that reason.

I don't see anything wrong with https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/DEU_Freudenberg_COA.svg/146px-DEU_Freudenberg_COA.svg.png. That points to the thumbnail being stuck in the European caching cluster.

It was added in {D306}, I couldn't find any documentation for why, https://wikitech.wikimedia.org/wiki/Thumbor/PDF and the code comments just says that it does. I'm trying to think of what cases could require this, and the only thing I can think of would be cached page numbers/URLs after a reupload that changes the page count. However, I don't think that serving the first page makes more sense than sending an error. Someone more familiar with ProofreadPage
might be able to thing of a situation where this makes sense.

Any swiftclient.exceptions.ClientException while loading the file causes a 404 (see rTHMBREXT wikimedia_thumbor/loader/swift/__init__.py:152-158). The documentation vaguely indicates that a HTTP response code should be available in ClientException.http_status, so that could be used to adjust the Thumbor response.

In T5593#8664865, @Glrx wrote:

Translation is a small issue. Most users will not notice. If a French speaker just visits the fr.Wiki, then she will just see the French translations. It only gets weird when she visits the Japanese Wiki and starts seeing illustrations in French or English rather than Japanese. Technically, the issue can be avoided by localizing the SVG for each language wiki -- that's what is done now, but the localization is done just for the PNG and not for the SVG.

https://signatures.toolforge.org/reports/en.wikipedia.org says 35. That probably means 1-2 on other wikis.

IME most thumbnail failures are not caused by strictly invalid files, but by rendering that exceeds the 1 minute timeout or the ImageMagick resource limits. There are a few instances where this is not the case (T285875: Thumbor fails to render PNG with "Failed to convert image convert: IDAT: invalid distance too far back", returns 429 "Too Many Requests" comes to mind), but this should generally be solved on the MediaWiki side, not the Thumbor side. ImageMagick and the other converters are also often perfectly happy to thumbnail invalid files while failing on apparently-valid files.

I've also gotten some similar complaints from English Wikipedia editors who wish to hide the rollback link on the watchlist.

It looks like the DNS records got updated, as I can connect to the replicas now. However, meta_p did not get updated, so a rOPUP modules/profile/files/wmcs/db/wikireplicas/maintain-meta_p.py run is still required.

That seems like a good idea, as it would also disambiguate from WMCS more clearly.

In T216815#8472047, @Andrew wrote:

Huh, is anyone tasked with this? This is one of the few cases that's keeping Stretch alive in cloud-vps and prod.

xbuild is old, unmaintained, deprecated, and doesn't work with VS/MSBuild projects written in the last 5 years (at least). As far as I can tell, Debian doesn't package msbuild, I don't know why. The only discussion was this request (#959045).

In T5593#8433481, @Vollbracht wrote:

The only problem left seams to be the fear of security flaws that I still don't understand.

Not that we intend to support, at least. The WMF HTTPS requirements also prevent most of those browsers from connecting at all.

@Pols12, could you create two new tasks for that? I personally don't consider the first to be a bug though.

Sounds like this is probably a duplicate of T251049: Make extension directory more easily configurable at compile time.

MariaDB does not support REGEXP_LIKE. https://jira.mariadb.org/browse/MDEV-16599

Looks like we're not getting off quite that easy, as stashbot's been timing out all day and StewardBot timed out an hour ago.

Do you have debug logs to confirm what imagemagick command gets run?

I implemented this on Commons in https://commons.wikimedia.org/wiki/MediaWiki:Gadget-fuzzyeverywhere.js.

Looks like there's also nothing in your configuration loading the conditional sharpening filter. I don't know if anything's changed in that plumbing, but I would expect that it might have. The relevant line from my old config is

FILTERS = [
    'wikimedia_thumbor.filter.conditional_sharpen',
    'wikimedia_thumbor.filter.format',
    'wikimedia_thumbor.filter.lang',
    'wikimedia_thumbor.filter.page',
    'wikimedia_thumbor.filter.crop',
    'wikimedia_thumbor.filter.flip',
    'thumbor.filters.quality',
    'thumbor.filters.rotate'
]

You can try adding it and see what happens.

Looks like you don't have the conditional sharpening configuration. DEFAULT_FILTERS_JPEG = 'conditional_sharpen(0.0,0.8,1.0,0.0,0.85)' Here's the config I used for test and development (it may need to be updated for the recent work):

import thumbor
QUALITY = 87
RESPECT_ORIENTATION = True
LOADER = 'wikimedia_thumbor.loader.proxy'
#LOADER = 'wikimedia_thumbor.loader.video'
FILE_LOADER_ROOT_PATH = '/srv/thumbor-plugins/tests/integration/originals/'
STORAGE = 'thumbor.storages.no_storage'
ENGINE = 'wikimedia_thumbor.engine.proxy'
HTTP_LOADER_CA_CERTS = '/etc/ssl/certs/ca-certificates.crt'
HTTP_LOADER_REQUEST_TIMEOUT = 300
FFMPEG_PATH = '/usr/bin/ffmpeg'
FILTERS = [
    'wikimedia_thumbor.filter.conditional_sharpen',
    'wikimedia_thumbor.filter.format',
    'wikimedia_thumbor.filter.lang',
    'wikimedia_thumbor.filter.page',
    'wikimedia_thumbor.filter.crop',
    'wikimedia_thumbor.filter.flip',
    'thumbor.filters.quality',
    'thumbor.filters.rotate'
]
EXIF_FIELDS_TO_KEEP = [ 'Artist', 'Copyright', 'ImageDescription' ]
EXIF_TINYRGB_PATH = '/srv/thumbor-plugins/tests/integration/tinyrgb.icc'
EXIF_TINYRGB_ICC_REPLACE = 'sRGB IEC61966-2.1'
PROXY_ENGINE_ENGINES = [
    ('wikimedia_thumbor.engine.xcf', ['xcf']),
    ('wikimedia_thumbor.engine.djvu', ['djvu']),
    ('wikimedia_thumbor.engine.vips', ['tiff', 'png']),
    ('wikimedia_thumbor.engine.tiff', ['tiff']),
    ('wikimedia_thumbor.engine.ghostscript', ['pdf']),
    ('wikimedia_thumbor.engine.gif', ['gif']),
    ('wikimedia_thumbor.engine.stl', ['stl']),
    ('wikimedia_thumbor.engine.svg', ['svg']),
    ('wikimedia_thumbor.engine.imagemagick', ['jpg', 'png', 'webp']),
]
HTTP_LOADER_MAX_BODY_SIZE = 4*1024*1024*1024  # 4GB
PROXY_LOADER_LOADERS = [
    #'wikimedia_thumbor.loader.file',
    'wikimedia_thumbor.loader.https',
]
COMMUNITY_EXTENSIONS = [
    'wikimedia_thumbor.handler.images',
    'wikimedia_thumbor.handler.core',
]
SUBPROCESS_USE_TIMEOUT = True
SUBPROCESS_TIMEOUT = 59
VIPS_ENGINE_MIN_PIXELS = 10000000
CHROMA_SUBSAMPLING = '4:2:0'
QUALITY_LOW = 40
DEFAULT_FILTERS_JPEG = 'conditional_sharpen(0.0,0.8,1.0,0.0,0.85)'
LOADER_EXCERPT_LENGTH = 4096
MANHOLE_DEBUGGING = True
APP_CLASS = 'wikimedia_thumbor.app.App'
HTTP_LOADER_TEMP_FILE_TIMEOUT = 20
MAX_ANIMATED_GIF_AREA = 100000000

signatures, which uses a standard webservice configuration, is having the same issue and is 503ing now. The running pod was deleted, and new pods can't be started. Stopping and starting the webservice had no effect.

tools.signatures@tools-sgebastion-10:~$ kubectl get pods
NAME                                      READY   STATUS      RESTARTS   AGE
signatures.sigprobs-cron-27730335-l2x9x   0/1     Completed   0          18d
signatures.sigprobs-cron-27750495-2jczm   0/1     Completed   0          4d15h
tools.signatures@tools-sgebastion-10:~$ kubectl get all
NAME                                          READY   STATUS      RESTARTS   AGE
pod/signatures.sigprobs-cron-27730335-l2x9x   0/1     Completed   0          18d
pod/signatures.sigprobs-cron-27750495-2jczm   0/1     Completed   0          4d15h

Also affecting stewardbots, unable to restart the bots after they disconnected (Remote host closed the connection).

In T319135#8278278, @Func wrote:

The thing is there is no way to only mark the translation unit for deletion, <noinclude> would not work for the Translate extension.

Looks fine from here with a spot-check.

Task as stated is Declined per discussion.

Already fixed on twn: https://translatewiki.net/w/i.php?title=Wikimedia:Portals-language-name/fr&diff=10815811&oldid=10801664

(local test instance, no private data)