For now. Disk space has been freed by 163 GB

In T241493#5878765, @bd808 wrote:

Rather than closing this as invalid, I will assign to @Cyberpower678 and recommend:

• Truncation of the s51059__cyberbot.externallinks_profiledata to a reasonable size.
• Disabling of xhprof profiling OR only profiling 1/1000 runs && adding a cleaning process to discard profiling data regularly

I'm nuking everything not from 2020 right now.

Damn. I hadn't realized it had accumulated that much.

Disk space is now critically low.

This one's on you.

Sadly, I don't think the VM will last that long. I am able to make a dump of the DB and load that dump into a new VM quite easily. If the storage space made large enough to last until the new CEPH project deploys, that would be fine. I think it will only handle tops 2 more months, as there are new projects being worked on with new tables required for these projects.

This one's on you.

In T36928#5691865, @MGChecker wrote:

I don't understand the issue. We have abusefilters already that block additions of specific content by users matching specific criteria, and noone is complaining there. How is this different, albeit a bit more generalized?

I don't think it unreasonable that sysops should have it by default, and maybe bots.

Someone re-enabled the bot without authorization.

You might want to check the history again. It wasn't the bot.

IABot is seeing all of these as alive now, since you reported the links as dead. They were automatically reset. I don't see what else to do here?

Domain has been whitelisted.

@Anomie I can confirm that your suggestion to normalize the UTF text does the trick. Thank you.

I added dev-master to my composer json. This should fix the reported issue.

No further response

This was already implemented before this ticket was opened.

Because deadurl is no longer a supported alias on enwiki, and because the current definitions are being drawn from the enwiki definitions, deadurl is not acknowledged on the template at all. Thus they are ignored.

IABot cannot be expected to accurately detect these cases. Please use https://tools.wmflabs.org/iabot/index.php?page=manageurlsingle&wiki=enwiki to adjust the URL metadata of the links it rescues. You can disassociate the, or associate a different, archive there. Changes are applied immediately, and a bot job can be started on applicable pages afterwards.

Please use https://tools.wmflabs.org/iabot/index.php?page=reportfalsepositive&wiki=alswiki to report false positives or simply revert the bot.

The bot correctly matched the format being used in the template and applied the same format to the new dates it added. There is no bug here.

https://tools.wmflabs.org/iabot/index.php?page=manageurlsingle&url=https%3A%2F%2Fwww.bipm.org%2Fen%2Fmeasurement-units%2F suggests that it found the source with the access date of 2018-11-23 somewhere on some wiki. IABot picked the closest snapshot to that and saved it for future use as it is a guaranteed working snapshot with similar, if not the same, data as the original from that time frame. If you want the bot to use a different snapshot, you can use the provided link and change the archive URL the bot should use. Changes are applied immediately.

Seems to be working now on my end. :-)

@Anomie what's the latest about this bug? I see it's awaiting review, but when will it get merged and deployed? This is sadly holding up work on my end.

I see nothing to indicate you should be hitting a barrier. Please make sure the interface is set to the English Wikipedia. Please also report what username you are using. I found "Brown Chocolate" which is a member of "basic user" which has the permissions needed to run the bot.

Errors are populated here, which given the edit rate of the bot, and based off of a random sample of what IABot broke vs what humans broke, we see that of the millions of pages it edited, this all that it broke. We should certainly commission a bot to clean up the mess, however.

The error rate is still very, very low from what I'm seeing. I'm not using that as a reason to dismiss your concerns, I just need to look at what's happening. I'm not fond of patch jobs. I want to fix this at the root.

In T232922#5493994, @Mikeblas wrote:

Why can't IABot know that there are multiple definitions of the reference it's about to change?
It also seems possible for IABot to check its own work:

1. Read the article, uniquely identifying existing error messages
2. Edit the article as desired
3. preview (or submit then read) the article, again uniquely identifying error messages
4. discard (or revert) the edits if the number of error messages has increased

I don't think the bot should be making articles worse, but that's what it's doing with these edits (and the edits that T22434 tried to fix). If its behaviour isn't deterministic, then I don't think it should be trusted to run autonomously.

"Indeed, it might be best if IABot didn't edit articles where it can't predict the outcome of its own edits." <-- I'm referring to that. IABot can't know when an edit it makes might have an unpredictable outcome. That would require serious Machine Learning. Something that's not slated to be implemented until further down the road (IABot v3). That's still some years off at least.

That's also not feasible. There's no way for IABot to even know that the edit is unpredictable to begin with them. The solution is to try and establish why the bot only touches one ref and ignores the other.

@Mikeblas I honestly don't know if this is worth fixing in IABot now. I don't even know what reflist is doing to trigger this error. And I can't program IABot to predict what references inside templates will do on other wikis either. This will need a different solution.

The deduping has been disabled.

The domain is whitelisted because IABot cannot reliably assess the URL. As for it ignoring deadurl, it is no longer supported and is no longer acknowledged. A different bot should go around replace deadurl with url-status.

Looking more closely at the example, it looks like "ARIA News 28 Oct" was defined 3 times, where one had vastly different content, and the other was an identical duplicate. I don't see an error here other than IABot exposing that there was a ref error not being caught earlier.

Just looking at the first example, I'm already confused. It deduped a named reference but a different reference lit up with a red error. Can you help me understand what is going on here?

In T232662#5485429, @Anomie wrote:

Most likely the content you're submitting via the API is not NFC-normalized UTF-8, making this a duplicate of T144071.
What is the exact request you're submitting? You may have to encode it with base64 or the like in case Phab also normalizes UTF-8 input. Even better if you can narrow it down to a reasonable test case rather than a multi-kilobyte blob of article content.

I couldn't say. I can't even see which page it's trying to pull in that given batch.

This isn't the first time I've run my script against IABot's contributions, so I would say it is a recent issue.

On wiki template data could have recently been changed. The bot will read that as well.

If it's producing a blank parameter, than somewhere in the template data, that parameter was marked as required.

Added url-access for all book references not going to specific pages as the ones going to a specific page do not require registration to view.

I cannot establish what is going on here. When pulling up the URL I see one page being mentioned and it's the same one you mentioned.

First off there are no such things as URLs without a schema (HTTP/HTTPS/FTP). The https does not redirect anywhere.

Also just leaving this here as well. I have now changed my on wiki hash.

I'm still in Germany, so I will be going to bed now. :-). Thanks @bd808

In T231526#5452947, @bd808 wrote:

@Cyberpower678 check your irc bouncer and/or Telegram client for an invite from me to do a video confirmation. :)

@revi and @MF-Warburg met at Wikimania too. They know my face too.

Welp my phone got logged out somehow. So now I only have one device left with an active Phab session going.

In T231526#5452900, @Urbanecm wrote:

@Cyberpower678 can create a file in home in his toolforge account (he has LDAP connected to Phabricator account, so it's clear the accounts are both his), do an edit somewhere to confirm this or we can ask him for confirmation via IRC PM (it's common knowledge who owns the account there, and he has a cloak anyway). Besides, he is still logged in, so it's not a big problem. Plenty of ways to verify Cyberpower's request :). At worse, Cyberpower can have a quick videoconference with someone who recognizes him in person :).
I believe the rationale is to verify the identity somehow, not to require hash and only hash.
Cyberpower should change the hash anyway :D.

I have 2FA enabled and my identity recently confirmed multiple times on Wikipedia, most recently on this very thread. Can I just post a 2FA reset confirm post on my talk page?

In T231526#5452891, @mmodell wrote:

Damn, what a pain, I wish there was an easier process for this.

In T231526#5452676, @mmodell wrote:

@Cyberpower678: The procedure involves confirming your wiki identity via a private paste
see https://www.mediawiki.org/wiki/Phabricator/Help/Two-factor_Authentication_Resets
So I've created a private paste here: P9005
Please edit the paste and add you identity hash, I'll compare it and then remove your TOTP 2nd factor from phabricator.