These tools are all affected by T280806, and the outreach seems to have targeted bots and user scripts. I personally never caught any of the outreach through the normal channels as the framing has been "these things we deprecated back in 2014"(although these API calls have been widely used after the 2014 change) nor have any "warnings" reached me even though Pattypan is one of the most popular batch upload tools to Commons used by many chapters and GLAMs.

Nor is this a quick change of "just replacing a few API queries", as many of these tools are affected through dependencies of various kinds.

In T293543#7487889, @Abbe98 wrote:

These tools are all affected by T280806, and the outreach seems to have targeted bots and user scripts. I personally never caught any of the outreach through the normal channels as the framing has been "these things we deprecated back in 2014"(although these API calls have been widely used after the 2014 change) nor have any "warnings" reached me even though Pattypan is one of the most popular batch upload tools to Commons used by many chapters and GLAMs.

General questions:

• Do you (or another pattypan developer) subscribe to the mediawiki-api-announce mailing list? Or Tech News? What "normal channels" are you typically following?
• Does pattypan use a distinct user-agent that makes it easy to identify in server logs? (https://meta.wikimedia.org/wiki/User-Agent_policy)
• Does pattypan log warnings that the API emits when you use deprecated functionality? (https://www.mediawiki.org/wiki/API:Errors_and_warnings#Warnings)

Do you (or another pattypan developer) subscribe to the mediawiki-api-announce mailing list? Or Tech News? What "normal channels" are you typically following?

I'm personally only actively follows Tech News, where I have seen this announced twice shortly before the deprecation. I did however not investigate this as it in both cases was described as the removal of features since long deprecated(before I maintained popular tools and before Pattypan was even created) rather than the removal of widely used features.

Does pattypan use a distinct user-agent that makes it easy to identify in server logs? (https://meta.wikimedia.org/wiki/User-Agent_policy)

Yes in using the following format: "pattypan/<version>(https://github.com/yarl/pattypan)"

Does pattypan log warnings that the API emits when you use deprecated functionality? (https://www.mediawiki.org/wiki/API:Errors_and_warnings#Warnings)

No, nor does Pattypan has any type of centralized logging or telemetry tracking.

There was direct outreach to Commonist by @Reedy with https://github.com/ritschwumm/commonist/issues/25 − August 24th (7 days before the days before the deadline) − which the volunteer maintainer acknowledged on September 12th. As far as I can tell, there was no such outreach to Pattypan and Vicuna.

(Even assuming that the maintainers had been / would have been super reactive to the warning and put out a release right away − these are desktop applications − it’s not over the minute a release is out. It could have taken several more weeks to get the upgrade through)

In T293543#7488401, @JeanFred wrote:

There was direct outreach to Commonist by @Reedy with https://github.com/ritschwumm/commonist/issues/25 − August 24th (7 days before the days before the deadline) − which the volunteer maintainer acknowledged on September 12th. As far as I can tell, there was no such outreach to Pattypan and Vicuna.

(Even assuming that the maintainers had been / would have been super reactive to the warning and put out a release right away − these are desktop applications − it’s not over the minute a release is out. It could have taken several more weeks to get the upgrade through)

I filed https://github.com/yarl/vicuna/issues/154 back in July.

There was outreach to the projects (on and off wiki) where I could see significant/numerous deprecated API calls where it was clear what program it was (from the User Agent). And out to various individual bot owners/maintainers starting back in July. And various ignored that too, repeatedly.

I had a look since the start of July in logstash... It's another yarl repo, and uses Wiki.java...

There's numerous Wiki.java useragents with nothing more than the version of Wiki.java in it, no supplemental information.

This is the reason we have https://meta.wikimedia.org/wiki/User-Agent_policy, and more specifically an example like:

User-Agent: CoolTool/0.0 (https://example.org/cool-tool/; cool-tool@example.org) generic-library/0.0

The generic format is <client name>/<version> (<contact information>) <library/framework name>/<version> [<library name>/<version> ...]. Parts that are not applicable can be omitted.

Vicuna has https://github.com/yarl/vicuna/commit/0dc286fb48a5c6d8a86c1673266b9d76cbeb84ad. And Pattypan looks to have https://github.com/yarl/pattypan/blob/2809ce78ff20ec13f7e418bd0e2ad1750bc6e085/src/pattypan/Settings.java#L46 (and other wiring of course) but when there's 0 results in logstash since July, I'm a little suspicious that this latter one is actually working properly.

I just filed https://github.com/yarl/pattypan/issues/146 for them to confirm they're actually setting the useragent that they're trying to do so.

If it doesn't appear in the logs, it's a little hard to know who to reach out to.

And as per T280806#7305738 and T280806#7399684... The number of deprecated API calls (relating to this issue) that were stopped was massive.

In T293543#7488987, @Reedy wrote:

In T293543#7488401, @JeanFred wrote:

As far as I can tell, there was no such outreach to Pattypan and Vicuna.

I filed https://github.com/yarl/vicuna/issues/154 back in July.

I stand corrected! Thanks for having reached out :)

Just to give people an idea of the size of the issue this is causing, Pattypan has been used to upload over 1.1 million images to Commons. Most of the people I know who are working with external organisations to share their content on Commons use this tool so all these projects are currently stopped until this issue is fixed. The only other option for mass uploads while this is down is using scripts which is way beyond the technical ability of most pople who work on partnerships. I'm personally working on a project which will upload up 50,000 to 100,000 images from a museum, without Pattypan I can't do it.

Thanks

Just to point out... There's two issues being bundled into one here in some ways.

If the issue is mostly due to the LetsEncrypt issues, there's nothing on the Wikimedia side to "fix". If it requires updates to Java versions etc, that's very much on the downstream developers of pattypan.

In T293543#7507232, @Reedy wrote:

Just to point out... There's two issues being bundled into one here in some ways.

If the issue is mostly due to the LetsEncrypt issues, there's nothing on the Wikimedia side to "fix". If it requires updates to Java versions etc, that's very much on the downstream developers of pattypan.

Thanks, is there a way to check which it is, and if it is the issue on the Wikimedia side who can fix it?

Based on https://github.com/yarl/pattypan/issues/138, it's related to the LetsEncrypt certs. Following that task is probably helpful.

Just wanted to +1 that Pattypan in particular being broken is a huge headache - it's the tool that we've been recommending to small (& indeed larger) GLAMs and Universities for the last few years, and I have one partnership whose engagement has now been completely interrupted until this is fixed. (They've been looking to do a test upload, and use that to train staff so that it can be implemented into their ongoing workflow, project was funded by an external body.)

I'm afraid we all know that, and maintainers need to fix issues in those code bases, as pointed out already... :)

Given how valuable/important Pattypan seems to be, I think it would benefit from some of https://wikitech.wikimedia.org/wiki/User:BryanDavis/Developing_community_norms_for_critical_bots_and_tools - whether it's more maintainers or better docs on how others can contribute to fix these kinds of issues.

@Legoktm This would be very much appreciated, this is really a big issue for most people working with partner organisations because there isn't a workaround or alternative and do not have any technical skills to maintain the tools.