In T274050#7152627, @Urbanecm wrote:

In T274050#7152521, @Cyberpower678 wrote:

[...]
I’m not convinced that this is something that can be fixed bot wide. As pointed out the API correctly identified the invoking user, and the consumer has all the correct grants selected that are needed to do the job, yet MW opted to error out with a blocked code. 172.16.*.* are likely tool UI invoked requests while the 185.15 one is the bot’s dedicated IP.

Would it be possible to log full API response at the bot's side? It'd be interesting to know the block ID, which should be included in the response. I think that would show us some way to go forward.

In T274050#7152537, @Cyberpower678 wrote:

@Urbanecm can you re-run that last query counting requests made omitting if the api_error_codes field is NULL or not? I’d like to see how those numbers change.

Like this?

spark-sql (default)> SELECT
                   >     http.client_ip, COUNT(*)
                   > FROM event.mediawiki_api_request
                   > WHERE
                   >     year=2021 AND
                   >     month=6 AND
                   >     day=11 AND
                   >     http.request_headers['user-agent'] LIKE "IABot/2.0%"
                   > GROUP BY http.client_ip
                   > ORDER BY COUNT(*) DESC;
client_ip       count(1)
185.15.56.22    7548910
172.16.1.7      8943
172.16.1.28     8104
172.16.1.228    51
172.16.1.4      36
[redacted]      14
Time taken: 65.786 seconds, Fetched 6 row(s)
spark-sql (default)>

@Urbanecm can you re-run that last query counting requests made omitting if the api_error_codes field is NULL or not? I’d like to see how those numbers change.

I can’t maybe have it pass back the whole API response ad-verbatim for users to report here. I’m absolutely certain that there is nothing wrong with the OAuth module of the bot, or it’s consumers. The many requests that @Urbanecm listed are from the bot’s main execution workers using the bot’s own owner-only consumer to edit with. These consumers have been largely the same since they were first created, as has been the OAuth engine the bot uses. I’m not convinced that this is something that can be fixed bot wide. As pointed out the API correctly identified the invoking user, and the consumer has all the correct grants selected that are needed to do the job, yet MW opted to error out with a blocked code. 172.16.*.* are likely tool UI invoked requests while the 185.15 one is the bot’s dedicated IP.

The problem is that I remain convinced this isn’t actually an error with the bot, but with the MediaWiki API. These are errors being passed back by the API itself, otherwise it would report a much more descriptive error, and you would also be barred from using the UI itself if you were actually blocked.

It's pretty much confirmed that this is not an issue with IABot. After testing multiple scenarios, a user that is legitimately blocked on Wikipedia will receive an entirely different worded error about their block. Specifically, the user will be greeted with "Blocked: You are currently blocked from using this interface. Check to make sure you are not blocked on Wikipedia and/or this interface." The current wording being returned to the user is "blocked: You have been blocked from editing.". This wording is consistent with what the MediaWiki API returns when a user attempts to submit an edit when MW thinks the user is blocked.

In T274050#7031542, @Chlod wrote:

The bot was working when I tried to archive the "SB19" (on enwiki) page normally. I tried to archive a page with IABot with the "Add archives to all non-dead references" option enabled, and it marked me as blocked. Likewise, I asked another editor (from a completely different part of the planet, which means there's little chance of an IP-based block to take effect) to archive that same page with that option disabled, and they were also blocked. I think this might be replicated by enabling the "Add archives to all non-dead references" option, however I fail to find what would be the possible underlying cause of why adding archives would cause the bot to interpret the user as blocked. I also can't determine how long this block lasts, and who is blocking who in the first place. IABot devs might find this information useful to some extent.

It is made very clear to the end user that this is a third party service, and that this service is opt-in. Users are not opted in by default. The use of Hotjar is not new to the admins of Toolforge.

None of those seem to apply as far as I can tell. It’s been discovered that this is only happening on the Kunernetes web service. The grid engine web service works fine.

Oops wrong ticket. This may need some more investigation.

https://phabricator.wikimedia.org/T280215

Alright. It will have to wait until I get the DB backend fixed. It's being repaired.

@AManWithNoPlan how would you like to help maintain the data contained within IABot? I can grant you the power to do this yourself going forward? Interested?

When v2.0.9 releases, IABot will no longer read LayURL values in the CS1 templates.

Then what exactly is the lay-url for?

Yes. The DB driving it is currently on fire, so to speak. While I’m putting it out, I had to shut down everything to prevent corruption.

This is implemented in v2.0.9. Use action=runpages to call it.

Your OAuth URL was borked.

This is fixed in v2.0.9 which will be released in a few days.

I need the full list you submitted and the articles that didn't make their way onto the submitted job.

No response.

This is fixed in v2.0.9 which will be released in a few days.

This is fixed in v2.0.9, which will be released in a few days.

Is this still happening?

This is related to a component of IABot Green develops.

I see it attached as /dev/vdb but where does it actually mount? Do I have to do that myself?

Is this still happening? I noticed a task that was commented out in my crontab file, was not commented out in the actual crontab.

Hmmm...it’s interesting. Not sure what is happening. Is there a timeframe when this happen? Is it continuous?

It’s hard to tell. It could also be other bots/tools getting blocked at the account level with the anon-flag disabled causing the IP to be effectively be hard-blocked. I don’t know how the inner workings of MediaWiki work. All I know is that this error message comes from the API itself, and not IABot.

These error messages are not originating from the bot, it’s being passed back by the MW API itself when requesting the edit on behalf of the user. All users edit from the IP 185.15.56.1 which is Toolforge. This is either a glitch in the API, or there are admins hardblocking bots originating from it.

I'm hoping this is resolved.

It'll take some time before the backlog of 229 pending jobs are completed.

Yes it turns out the bot had robust error handling for prod errors. It didn’t have robust handling for reused OAuth nonces. For some reason it got a lot of OAuth nonce already used errors. This took repeated running the last few days to diagnose.

Unable to reproduce.

Closing this as resolved as the Wayback Machine has gotten better over the years.

Unable to reproduce. Tool claims you are not blocked.

If you are getting the first error, make sure you are not set to a "Mixed" status. You need to firmly define the state of the URLs. Alive and Dead resets every individual link while Whitelist and Blacklist sets it on all the selected domains as a whole and the bot will no longer assess if they are alive or not.

This does indeed look like a legitimate bug.

You did not whitelist it. No log entry for that URL at all aside from the bot scans it did of it. I have gone ahead and whitelisted it for you.

In T273003#6784499, @Dreamy_Jazz wrote:

In T273003#6782955, @Joe wrote:

In T273003#6780172, @Dreamy_Jazz wrote:

Just to note that cyberbot I has been blocked because of the blanking issues on enwiki. The bot has also been blocked on two other wikis for the blanking issues. It would be really great if this issue is fixed, as this has stopped a fair number of tasks.

The solution to this problem is pretty simple:

make the bot handle failures of the API (they will *always* happen, even if in ridiculously low numbers, as shown above) and stop blanking pages when receiving an error.

Even if we get to the bottom of which events caused the few errors we found evidence of, we can definitely guarantee there will be more errors in the future, and at times even a ton of errors, be it for a wrong deployment or some catastrophic hardware failure or something else.

I think the bot should only be unblocked once error management is properly taken into account, not before.

I don't disagree with you, and the block of the bot is because the bot was blanking (and not because of the server side issue). Primarily, the bot should be ready for strange situations like this. But secondarily, when the bot has to deal with the issues the root cause of the issue on the server side should be fixed with some level of speed.

Timestamps of these entries seem to suggest that when the bot encounters these failures almost all API requests being made are erroring out during the timeframe of these batch errors.

Got an influx of blank, 500s and 502 overnight. Your errors may be "few" overall, but they are happening in batches.

In T273003#6782382, @Urbanecm wrote:

@Cyberpower678 This doesn't sound to be a complete dump of a raw request. I'm pretty confident the response has at least one line (the one starting with HTTP/ that denotes the status etc.).

Normalization is by default on, but can be turned off if the community wants it off.

I've added additional logging data to the framework.

Got another surge of bad responses from production.

Joe, not all requests are 502s. They are insignificant compared to the amount of requests returning null responses. This is a very problematic occurrence as Cyberbot has begun to malfunction badly because of it. Is blanking pages repeatedly, my talk page is littered with malfunction reports. Granted some extra robustness could be implemented, this shouldn’t have to be something I would need to guard against at this level. Cyberbot has never had these problems in the past until I started getting reports around 11 days ago. I see my communication log is starting to grow to a large size recording these failures and has grown to 5 GB.

It won’t. It’s not designed to do that. It will only recognize the alias and use what’s there, or add the default preferred alias if not defined.

No changes have been made to the bot whatsoever.

The IP 172.16.2.21, I don't know what the UA is. The bot is pretty old.

Project Name: cyberbot
Type of quota increase requested: 8 CPUs and 16 GB RAM
Reason: IABot is expanding on to more WMF wikis, which requires more processing power. The current exec VM is reaching its limit.

Especially to Wikidata.

Thursday, as in yesterday? I’m not aware of anything that should have been running to create that massive level of requests.

https://ru.wikipedia.org/wiki/Модуль:Citation/CS1/Configuration

I don’t have access to your Citation module on ruwiki. A local sysop will need to make the changes.