In T280806#7312038, @Trizek-WMF wrote:

Side note: if you have a precise list of bot owners who should change the parameters for their bot, I think it would be more impactful to contact them directly.

I have been. To very mixed results.

I have also been targeting prolific users and fixing the scripts they are using

In T280806#7313070, @Reedy wrote:

In T280806#7312038, @Trizek-WMF wrote:

Side note: if you have a precise list of bot owners who should change the parameters for their bot, I think it would be more impactful to contact them directly.

I have been. To very mixed results.

I have also been targeting prolific users and fixing the scripts they are using

Krdbot is till one of the worst... https://commons.wikimedia.org/w/index.php?title=User_talk:Krd&oldid=587342912 doing 150K a day

And I've just followed up (due to no response) at https://commons.wikimedia.org/w/index.php?title=User_talk:Rudolphous&oldid=589964292#RudolphousBot_deprecated_API_calls doing around 100K a day.

The next worst offenders are doing like 1K a day...

Over the last 7 days:

TBH, I think I'm to the point we should just drop this... @Pchelolo thoughts?

It would be really nice to get it into the 1.37 release....

Agreed. Let's do it.

I'm just back from vacation today and catching up, will look into gerrit change tomorrow.

In T280806#7343246, @Pchelolo wrote:

Agreed. Let's do it.

I'm just back from vacation today and catching up, will look into gerrit change tomorrow.

Thanks

I'm struggling with what seems to be some weirdness with the Authority changes from rMWedb3a9a692b5: Allow passing mock Authority in API integration tests in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/681393, which is the only blocker to merging as things standard.

For some reason, with that patch applied MW seems to be behaving like the user is anon. But it isn't before, which is just odd.

i18n messages need removing, but I think doing that in a seperate patch is cleaner (incase we need to revert this soon after it's deployed). If we then need to cherry pick that early in the REL1_37 process, that WFM.

Change 714599 abandoned by Reedy:

[mediawiki/core@master] WIP: API: Add feature flag to turn off legacy token methods

Reason:

https://gerrit.wikimedia.org/r/714599

Change 720777 had a related patch set uploaded (by Ppchelko; author: Ppchelko):

[mediawiki/core@master] Drop i18n messages for removed token API

https://gerrit.wikimedia.org/r/720777

@Reedy, IMO, you can remove the tokens since both their removal has been publicized long enough and individual bot owners have been warned. You've done more than what is usually done.

If these old tokens are removed, what could happen with these bots? Would they stop working? Do bad actions? If risks are low, then remove the tokens. Even if risks are higher, bot's owners have been warned enough. It is their problem if they don't comply, or if a community has a bot with no active owner, not yours.

Change 681393 merged by jenkins-bot:

[mediawiki/core@master] Drop action api token methods deprecated in 1.24

https://gerrit.wikimedia.org/r/681393

In T280806#7349005, @Trizek-WMF wrote:

@Reedy, IMO, you can remove the tokens since both their removal has been publicized long enough and individual bot owners have been warned. You've done more than what is usually done.

If these old tokens are removed, what could happen with these bots? Would they stop working? Do bad actions? If risks are low, then remove the tokens. Even if risks are higher, bot's owners have been warned enough. It is their problem if they don't comply, or if a community has a bot with no active owner, not yours.

James has just +2'd the patch.

For most bots, if their error handling is reasonable, they'd probably just stop working (editing in most cases). Some might try and edit anonymously... They can probably still keep editing whilst they have a valid token, but at the rate many of them are requesting new tokens... It shouldn't ake them long to notice things failing... Whether the bots stop whatever they're doing, I'm sure users will start complaining... :)

I'm sure there'll be some stragglers that haven't seen (or have ignored) the messaging... And I guess that's on them.

We should probably do a followup email to mediawiki-l/wikitech-l and mediawiki-api-announce to say it is being dropped.

I don't see any obvious reason we should need to revert, but we'll wait and see what happens!

I'm putting it back into the Tech News loop. :)

Change 720777 merged by jenkins-bot:

[mediawiki/core@master] Drop i18n messages for removed token API

https://gerrit.wikimedia.org/r/720777

In T280806#7349193, @Trizek-WMF wrote:

I'm putting it back into the Tech News loop. :)

This was already announced in the 19 July and the 30 August editions of Tech News.
Q1. Does this require a 3rd announcement (within 3 months)?
(We can do so if warranted -- I.e. If we think or know there are bot/script/tool owners that urgently need a reminder of how to fix things.)
Q2. If yes, then is re-using the same message sufficient (with a past-tense tweak to sentence 2), or how else should it be tweaked?
Thanks!

Change 721539 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@master] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721539

Change 721540 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@wmf/1.37.0-wmf.23] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721540

So...based on T291202: Pywikibot: 'NoneType' object is not subscriptable in validate_tokens it seems this has broken Pywikibot's feature detection on which token module to use. While Pywikibot wasn't using action=tokens, the action=paraminfo query it was using is now broken.

Given that it wasn't obvious (though in hindsight it is) that doing this would break, and how widely used Pywikibot is, I'd like to revert this out of wmf.23 to give a few days for Pywikibot to fix and bot ops to update. It would still go out next week.

Change 721540 merged by jenkins-bot:

[mediawiki/core@wmf/1.37.0-wmf.23] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/721540

In T280806#7360429, @Legoktm wrote:

So...based on T291202: Pywikibot: 'NoneType' object is not subscriptable in validate_tokens it seems this has broken Pywikibot's feature detection on which token module to use. While Pywikibot wasn't using action=tokens, the action=paraminfo query it was using is now broken.

Given that it wasn't obvious (though in hindsight it is) that doing this would break, and how widely used Pywikibot is, I'd like to revert this out of wmf.23 to give a few days for Pywikibot to fix and bot ops to update. It would still go out next week.

T287819: Add automatic temporary feature disabling to encourage migration away from long deprecated feature would've been really useful for this sort of thing :)

BTW, is there anything of Cyberbot in the list of still offending bots, or did I fix this a while ago and forget?

Please do suggest specific content/wording that we can use in Tech News, if inclusion there is desired.

You might also consider a MassMessage to https://meta.wikimedia.org/wiki/Distribution_list/Bots

In T280806#7360797, @Quiddity wrote:

Please do suggest specific content/wording that we can use in Tech News, if inclusion there is desired.

If it's not too late, something like:

The [[mw:MediaWiki_1.37/Deprecation_of_legacy_API_token_parameters|previously announced]] change to how you obtain tokens from the API has been delayed to September 21 because of an incompatibility with Pywikibot. Bot operators using Pywikibot can follow [[phab:T291202|]] for progress on a fix and should plan to upgrade to 6.6.1 when it is released.

You might also consider a MassMessage to https://meta.wikimedia.org/wiki/Distribution_list/Bots

Ack, I think that would be appropriate once the Pywikibot release is out.

There are also plenty of other subtasks open...I'm not sure whether those should also be blockers or not.

Added to Tech News: https://meta.wikimedia.org/w/index.php?title=Tech/News/2021/38&diff=22029905&oldid=22027463&diffmode=source - Will be frozen for translations in ~1 hour.

In T280806#7362676, @Legoktm wrote:

There are also plenty of other subtasks open...I'm not sure whether those should also be blockers or not.

Personally, I don't really mind. I fixed various of the ones that were resulting in large amounts of log spam (like thousands or tens of thousands of log entries a day), or more clearly well used scripts.

Many of them are just rotten copies of scripts from other wikis, in various user subpages. And in many cases, the usages continued to grow, as people are/were copy pasting scripts to more wikis...

T286609: TMH still checking for intoken probably should be dealt with at least though. Though it shouldn't really block anything. I think it can just be removed, as I don't actually see anything explicitly making that API call either

I would've hoped that T286351: Update on-wiki GuidedTour JS pages to not use deprecated API calls might've been fixed by the team responsible, as again, the out of date patterns were clearly still being copied and proliferated...

Since it now has been announced in Tech News another time, I think you can safely remove the tokens from the API. If things break after this, it is not your fault. :)

Pywikibot 6.6.1 was released yesterday, I'd want to push this one more week so we have time to announce the release and give people time to upgrade.

I don't mind too much. We got it into MW-1.37-release which I think is good.

But it does highlight the importance of having something like T287819: Add automatic temporary feature disabling to encourage migration away from long deprecated feature

Change 722933 had a related patch set uploaded (by Legoktm; author: Legoktm):

[mediawiki/core@wmf/1.38.0-wmf.1] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/722933

Change 722933 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.1] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/722933

In T280806#7370239, @Legoktm wrote:

Pywikibot 6.6.1 was released yesterday, I'd want to push this one more week so we have time to announce the release and give people time to upgrade.

A week to have everyone upgrade all their bots? Are you kidding me?

This task is tagged as priority low. It would be completely ridiculous to push this through now knowing what amount of fallout this will cause.

In T280806#7372926, @Multichill wrote:

In T280806#7370239, @Legoktm wrote:

Pywikibot 6.6.1 was released yesterday, I'd want to push this one more week so we have time to announce the release and give people time to upgrade.

A week to have everyone upgrade all their bots? Are you kidding me?

Is that unreasonable? For me it's just running pip install -U pywikibot on the various tool accounts I have. If you're using the shared copy of Pywikibot on Toolforge it's already been updated.

On the Pywikibot project side somehow no one noticed that this would break until it actually did despite having months of advance notice, so I'm not really sure what the right thing to do is.

Further complicating or helping (depends how you see it) matters is that Pywikibot caches the API request in question for 30 days (by default), so in practice bot operators will have more than a week to upgrade, depending on when their API cache expires. This is why we didn't see any reports of mass breakage last week on Commons/Italian Wikipedia/etc. when it was deployed there for a few hours.

Is that unreasonable? For me it's just running pip install -U pywikibot on the various tool accounts I have. If you're using the shared copy of Pywikibot on Toolforge it's already been updated.

(People have real life, vacations, sick leaves, etc. Tool owners are not enterprise that can guarantee timely reaction times, but volunteers who have other things to do as well.)

Just to note, there is other bot code/frameworks than the pywikibot too. As the mainline pywikibot is tied to being scriptable bot by the login handling then the all other use cases are using custom login routines. Either on based on pywikibot, but most likely to something else. My point here is that the "pip install -U pywikibot" works only on limited number of use cases.

My 2 cents, this has been pending for months now, and advisories were sent out MULTIPLE times warning of this incoming change. Everyone had months to update their bots, no one chose to listen. I’m with @Legoktm here. There was plenty of time given to adjust to this change. Time’s up.

In T280806#7373247, @Zache wrote:

Just to note, there is other bot code/frameworks than the pywikibot too. As the mainline pywikibot is tied to being scriptable bot by the login handling then the all other use cases are using custom login routines. Either on based on pywikibot, but most likely to something else. My point here is that the "pip install -U pywikibot" works only on limited number of use cases.

Most other frameworks and bots were notified *months* ago, not to mention these modules have been deprecated and emitting warnings for years, Pywikibot slipped through the cracks because it was using it in a non-obvious way and is mostly getting a special extension because of how widespread its usage is. Reedy reached out to individual bot ops and tools that we could identify in the logs, some of which have fixed it, others said they'd fix it whenever it actually breaks :shrug:

In T280806#7373230, @Base wrote:

Is that unreasonable? For me it's just running pip install -U pywikibot on the various tool accounts I have. If you're using the shared copy of Pywikibot on Toolforge it's already been updated.

(People have real life, vacations, sick leaves, etc. Tool owners are not enterprise that can guarantee timely reaction times, but volunteers who have other things to do as well.)

Of course, I'm in that boat too. While it's a best practice to have multiple maintainers, I know the reality isn't like that. But I also have no clue what people want here, no one has proposed an alternative timeline or even said how much time they need to update their scripts/bots/tools. It's not like the notifications I sent out today were even the first notice, it was in this week's Tech News too that people would need to upgrade, but no one complained then...

There is no interface change or any other changed behaviour coming with Pywikibot 6.6.1 against the previous version except that APISite.validate_tokens() no longer changes old token keys to new csrf token because this is done by TokenWallet token handler. 6.6.1 solved a MW code change which was here with 1.37.0 for a short time. This only occurred due to this token switch an could have been avoided by giving up support of very old MW 1.23. Anyway this validate_tokens method hasn’t to be used by any private script and therefore I do not expect any further work except updating the framework.

Three groups of people are involved in this:

• MediaWiki API developers
• Pywikibot framework maintainers
• Pywikibot framework users

The MediaWiki API developers and the Pywikibot framework maintainers had a bit of miscommunication causing deprecated code to still be in use. By forcing this change through you're effectively punishing the Pywikibot framework users for a mistake made by other people.

Maybe we can rip out the deprecated code in parts so have a big disruption? Looking at Xqt's fix at https://gerrit.wikimedia.org/r/c/pywikibot/core/+/721621/14/pywikibot/site/_apisite.py#1302 :
https://en.wikipedia.org/w/api.php?action=paraminfo&format=json&modules=query+tokens should be used.
The bot crashes on the missing type in https://en.wikipedia.org/w/api.php?action=paraminfo&format=json&modules=query%2Binfo because it is removed in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/681393/30/includes/api/ApiQueryInfo.php . Would it be possible to just return an empty list instead for now?
Of course https://gerrit.wikimedia.org/r/c/mediawiki/core/+/681393/30/includes/api/ApiTokens.php would be the next bump. If that returns any empty list in the right location in the tree, you wouldn't hit the bug.

That way you can at least rip out most of the legacy code.

As for deprecation warnings. These are currently quite hidden, see https://en.wikipedia.org/w/api.php?action=paraminfo&format=json&modules=tokens and https://en.wikipedia.org/w/api.php?action=paraminfo&format=json&modules=query%2Binfo . Any way to make this more visible or some kind of strict mode where the API just drops everything deprecated? I'm sure Pywikibot will use more deprecated API parts and I wonder if this will help finding it now and in the future.

FWIW, the deprecated warnings on the actual queries when making use of the deprecated features are more visible - https://en.wikipedia.org/w/api.php?action=query&list=recentchanges&rctoken. But only to developers. Unless the framework/similar logs or displays them, the end users will be mostly oblivious.

"warnings": {
    "main": {
        "*": "Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application."
    },
    "recentchanges": {
        "*": "The parameter \"rctoken\" has been deprecated."
    }
},

Obviously they can still be ignored, but there's not much we can really do about that (unless we have a "confirm I know there's a warning" type response).

I'm not sure why on paraminfo we'd need to make it more visible, I don't think that quite makes sense. If you're querying these things programatically, you should probably be looking at things like depreacted if you care about these features. I will note I haven't actually looked what pywikibot is doing, or why it's querying paraminfo.

See also https://en.wikipedia.org/w/api.php?action=tokens

"warnings": {
    "main": {
        "*": "Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application."
    },
    "tokens": {
        "*": "\"action=tokens\" has been deprecated. Please use \"action=query&meta=tokens\" instead."
    }
},

And if you go on https://en.wikipedia.org/w/api.php?action=help&modules=tokens it does say "This module is deprecated." in red (yay, colour).

Using turnilo's webrequest 1/128 sampled logs, over the previous day there are 238k entries that have a user-agent with "Pywikibot" in them, If I filter it down to the regex of Pywikibot/(6\.6\.1|7) (either 6.6.1 or 7.0.0dev), that gives us 78.5k hits, or 33% of Pywikibot users (by request volume) having upgraded. At that level, I don't think it really makes sense to roll out this week, probably a massmessage to those bot owners individually would help.

But I also really shouldn't be doing this every week, someone else needs to own this. I already said last week I wasn't going to backport the revert again.

Change 724791 had a related patch set uploaded (by Ppchelko; author: Legoktm):

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

Change 724791 abandoned by Ppchelko:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

Reason:

https://gerrit.wikimedia.org/r/724791

Change 724791 restored by Ppchelko:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

Oh dang, apologies, I've totally missed this comment. Let's backport to wmf.2. I'd be very upset if we need to cherry-pick to wmf.3, will look again early next week.

Change 724791 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.2] Revert \"Drop action api token methods deprecated in 1.24\"

https://gerrit.wikimedia.org/r/724791

I did the same analysis as @Legoktm did last week, and the numbers didn't really change over the week 80k reqs/day upgraded out of 238k req/day.

I seems like the upgrades are not progressing any longer. There's been so many announcements related to this project already, that it's unlikely one more announcement will change anything. Let's try to roll it forward - I will not backport the revert just yet.

In T280806#7398805, @Pchelolo wrote:

I did the same analysis as @Legoktm did last week, and the numbers didn't really change over the week 80k reqs/day upgraded out of 238k req/day.

I seems like the upgrades are not progressing any longer. There's been so many announcements related to this project already, that it's unlikely one more announcement will change anything. Let's try to roll it forward - I will not backport the revert just yet.

So you want to break 150k reqs/day for a low priority change?

So you want to break 150k reqs/day for a low priority change?

Yes and no. wmf.3 will first be rolled out to group0/1 wikis, which receive way less traffic then group2. So some requests will get broken. With about 5 announcements that's been made regarding this, there seem to be no other way to encourage migration then temporary breaking things.

Before wmf.3 is rolled out to group2, we can backport the change. For this particular change we unfortunately don't have a way to enable/disable functionality per-wiki.

For future similar projects we should have T287819 and not do it all manually.

I think it makes sense to do another round of messaging (using bot username's from UA data) before trying to roll this out. At the very least it should be re-announced with a new date.

Just to point out, that with the fixes and outreach done... We're at a fraction of the count of deprecated requests we were ~3 months ago

(I know this doesn't account for pywikibot users that are making requests that would break).

Just to cross-reference, https://en.wikipedia.org/wiki/Wikipedia:Bots/Noticeboard#General_query is discussing which bots have gone offline (likely) because of this. I guess they were all low traffic (except Lowercase sigmabot III which is in one of Reedy's screenshots) to not make it onto dashboards.

@Quiddity this was mentioned in Tech News dated September 21, but I think it should have been included in the Tech News dated Oct 4 as well. The actual impact of this was perceived on the WMF wikis by Oct 7 (i.e. that is the date on which user scripts would break) and sending a notification just before this date would have been a good idea.

Do we have guidelines about when things should be announced on Tech News? There is a balance between announcing early (giving users time to prepare, modify code, etc.) and announcing just-in-time (so users who get impacted have an easy time finding relevant info in latest tech news). There is also a balance between announcing once or more than one time. Do we have guidelines for that?

I honestly think this has been announced plenty of times. Even I found the time to put in the 5 minutes it took to change up the API calls to get the token. I don’t do updates to Cyberbot often these days.

In T280806#7434073, @Huji wrote:

@Quiddity this was mentioned in Tech News dated September 21, but I think it should have been included in the Tech News dated Oct 4 as well. The actual impact of this was perceived on the WMF wikis by Oct 7 (i.e. that is the date on which user scripts would break) and sending a notification just before this date would have been a good idea.

Do we have guidelines about when things should be announced on Tech News? There is a balance between announcing early (giving users time to prepare, modify code, etc.) and announcing just-in-time (so users who get impacted have an easy time finding relevant info in latest tech news). There is also a balance between announcing once or more than one time. Do we have guidelines for that?

For what it's worth, this was announced 3 times in Tech News. 19 July and 30 August and 20 September.
We do generally try to announce at the optimal time, but without duplication so that readers don't become accustomed to ignoring entries.
Beyond that, I'd say it's context-dependent.

Change 721539 abandoned by Ppchelko:

[mediawiki/core@master] Revert \"Drop action api token methods deprecated in 1.24\"

Reason:

This landed, no need for revert anymore

https://gerrit.wikimedia.org/r/721539

In T280806#7376680, @Multichill wrote:

The MediaWiki API developers and the Pywikibot framework maintainers had a bit of miscommunication causing deprecated code to still be in use. By forcing this change through you're effectively punishing the Pywikibot framework users for a mistake made by other people.

I’m with Maarten on this. I use Pywikibot because it takes care of these sort of API communication for me. My expectation is that if I run a reasonably recent version of Pywikibot, then I generally will be fine.

So frankly I do not always pay much attention to these sort of API deprecation announcements, because I assume that pywikibot is well on top of it. Had I read that “Scripts, bots and tools that use the parameters from before the 2014 change need to be updated”, I would have assumed I had nothing to do because the pywikibot(s) I use is more recent than that.

So I maintain rTHER. I had updated pywikibot in August 2020, just about a year ago (I generally try to do it every year before Wiki-Loves-Monuments starts, I guess we forgot this year but it would have been the same).
Now the bot broke on October 27th − not quite the worst possible time (that would have been mid-September), but still quite disruptive, as some countries are still running their WLM.

I now have a patch up for upgrading − it was *not* a 5min job − I had to take care of this deprecated thing, and that removed method, and that signature change… Nothing to pull my hair out but still some time.

I don’t blame the pywikibot devs for this − it can happen, no worries. But I think that once it became apparent that pywikibot was affected by this, the whole “we have announced this plenty of times” was not quite true anymore − there was only the September 21st announcement, which I missed − and the whole thing should have been delayed much much longer. Some people do update their dependencies every once in a while, and for them the fix would have landed in their natural update cycle, without any disruption. Conversely, Expecting volunteers to update their bots on this kind of timeline is really not nice.

In T280806#7373254, @Cyberpower678 wrote:

My 2 cents, this has been pending for months now, and advisories were sent out MULTIPLE times warning of this incoming change. Everyone had months to update their bots, no one chose to listen.

See above. Pywikibot users certainly did not have *months* to update.

In T280806#7434100, @Cyberpower678 wrote:

Even I found the time to put in the 5 minutes it took to change up the API calls to get the token.

(Frankly, I find this “Even I” very dispensable − sounds like you are assuming to be the busiest tool developer around.)

I will note that I created T287819: Add automatic temporary feature disabling to encourage migration away from long deprecated feature as what would hopefully be a more user (and developer) noticeable way of becoming aware that functionality might be going away.

Because nearly 7 years of people ignoring the deprecated warnings in the output clearly shows a lot of people aren't paying any attention to those...

And there's definitely code that has been written after it was deprecated and deployed that still uses these deprecated calls, and/or has been copy/pasted around further...