Quotes with a level of 10 or above should not happen in first place. If case is difficult, summarise it. If parts of the case are resolved, remove them from the quote. If both side have threaded clients, which should be 99% of cases, quotes are courtesy only, and can be cut at any time.

Disregard my last entry. Tickets have been identified.

There are now some tickets where responses have not been sent out. How do we find these tickets?

There is a subqueue which counts for the headline but not for the queue view.

Numbers appear correct.

The bounces queue is at 292k now, and increasing. Please have a look.

All provided examples appear correct to me. The number for the main queue is always including the numbers for it's sub-queues.

Cannot reproduce that, looks good to me currently.

I think I have changed the setting and deployed it, but it still shows the old value, and now cannot be edited. Please check what is wrong.
Please change it to: volunteers-vrt@wikimedia.org

Please provide actual examples which counts are not accurate. I cannot reproduce it.

Thank you. I will change it.

It appears the password recovery is sent from "otrs-admins@lists.wikimedia.org", which is entire nonsense. Why is that, and how can it be changed?

All double checked now and appears correct, user reports that it still doesn't work. Please see: https://vrt-wiki.wikimedia.org/w/index.php?diff=134462

My personal opinion is that we should disable notifications completely, but this perhaps isn't consensus.

I just heard from one user that password recovery still doesn't work for them.

I have unsubscribed the mentioned user. This appears to be the only one, and I will monitor this from now on. It makes no sense to get copies of thousands of spams messages each day.

It appears my previous query must have been wrong. Please stand by an hour or two.

Please provide in private who that is and how you found the information.

If I checked correctly, nobody is subscribed to the Junk queue, so no notifications for that should have been created.
If there were any, I think that should be investigated further.

I think the focus should be to determine if the queue size is the cause of the impact or not. I.e. if there is another issue. The queue size itself is no issue as long as it stops growing.

We need an analysis what exactly happened, and perhaps a strategy not to accept such fake bounces at all.

The 219.240.37.89 looks like a common factor. Can we block this source IP for SMTP as a first measure?

Ir appears to me that we are accepting bounces from phishing e-mails sent with fake sender info@wikipedia.org.

I just observed this error at VRT wiki.

Again. Please unbreak this permanently.

Please unbreak again.

The problem seems to have disappeared-

After the weekend I suggest we just leave it as it is, as the deletion will be done in a few days.

The problem described in the previous entry has disappeared. Good.

We may have some database trouble already, the queue Probably-Spam in the overview shows to have 47 tickets but is actually empty.
{F58894214}

Let's wait until Monday.

I'd support that if we are sure that the database can handle it. I will try later today if the limit can be increased in the system config.

This for sure is because of the 1.4 million open Junk tickets. It will take approximate 30 hours to delete them. Please stand by.

Likely resolved in linked task.

OMG, you're so right, this is likely all related to too many tickets on one day, which is caused by T389079.

It appears to be an infinite loops of this:
https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=14332822

Of course templates could be fixed, but this is an unidentified numbers of templates in all wikis, and an unidentified number of gadgets and tools.

Broken template:
https://commons.wikimedia.org/wiki/File:Ehtoolehto_KeyArt_pysty_elisa.jpg

Examples:
2025031210000737 https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13518644
20250317104578578 https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=14329439

The issue could be the enormous amount of Junk tickets, normally 20k, currently 800k tickets.

The account was created in the meantime. I suggest I test this at the next opportunity and report here. Thank you!

Special:CreateAccount in a normal browser.

Request from <redacted>:1227:f5ff:fec7:7ec1 via cp3070 cp3070, Varnish XID 926533531
Upstream caches: cp3070 int
Error: 429, at Thu, 12 Dec 2024 10:46:40 GMT

Please unbreak now.

This looks much! better. Thank you.

I have the impression that it has changed something, but I still see a lot of RCVD_IN_VALIDITY_SAFE hits, example: https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13409476

I'd say this or any such problem should not occur again, as we definitely lost tickets, and the actual impact may never be determined. And I think it was pure luck that the issue was detected and identified so quickly. (Thank you very much revi for raising it!)

Can't we just check what was changed yesterday, and undo that?

That actually make VRTS fubar. Unbreak now please.

Just happend again:

Done.

As this is nearly complete and perhaps just needs activation, when are "we" going to do the final step?

Perfect. When will it be activated?

Every hour is perfect. Yes, please add the mentioned safeguard!

Looks good to me. Please continue.

Sounds good. Can you please provide a diff of the output of the current script version vs. the new one, just to be safe? If that looks good if should be activated as soon as possible.

The only addresses in the hardcoded lists, that is active on VRTS, is: otrs@wikimedia.org

We need to add "WHERE valid_id = 1" to the SQL query, and then sync then exemption list and OTRS active addresses (https://vrt-wiki.wikimedia.org/wiki/List_of_email_addresses).

I would like to see the whole content of vrts_aliases.py and discuss (perhaps at a different venue) if things can be cleanup up or otherwise improved.

Addresses can be disabled in VRTS.
I don't know how the exim integration works at all, but if somebody is willing to share some information, VRTS admins can perhaps add their perspective.

Why is this exemption hardcoded in a script while the addresses could have been just removed from VRTS? I'd appreciate if VRTS admins could be involved in such things, so that it can at least be considered to build as little legacy as possible.

The problem occurs just now, created one account, cannot create another one.

The list of all used e-mail addresses is defined in VRTS and mirrored to: https://vrt-wiki.wikimedia.org/wiki/List_of_email_addresses

With the curl command you are not logged in, are you?

With the curl command you are not logged in, are you?

Resolved by latest update.

Resolved by latest update.

In T314247#9905328, @Dzahn wrote:

Mailman migrated to a new server and a new version just now. Did this get faster?

Sadly I'm not competent enough, I don't even have a clue where to look. I'm of course happy to assist and to test.

Why is MW1.41.0 removed here when the issue started in MW1.41.0?

The list in Manual:32-bit consists of four entries, three of which are relevant for very large sites only, i.e. not relevant for nearly all sites, and one issue similar to this one.

I can confirm that in a 64bit environment the problem does not occur.

The problems happens even with a fresh default 1.41.0 installation on an empty database, so the German language isn't the factor because default is en-gb, and the same for short URLs.

What perhaps should be said: My story runs on a 32 bit Linux, and the MW installer says php uses 32 bit integers, which is not recommended. Any comment on that before a restart the whole testing on a 64 bit system?

Step by step:

• download MediaWiki 1.41.0 and unpack it in the Documentroot of an empty webserver
• setup empty MariaDB
• run the Mediawiki installation, leaving as much as possible on default, except: Wiki name, database password, user name, user passwort, enable Linter, enable Visual Editor, enable DiscussionTools
• notice that it complains that no default skin is selected, so choose Vector
• install the LocalSettings.php
• notice that DiscussionTools is not enabled in the generated LocalSettings.php, so enable it
• run update.php or not
• verify described behaviour

It may happen that more than 6 accounts per 24 hours have to be created,. Does the limit also apply to sysops?
Anyway, at the given day only two accounts were created, and the second creation showed the mentioned problem.

Sent by e-mail.

As I have created the ticket, it appeared to work. Anyway it should be checked, as this happened around a dozen times before and it's not acceptable to retry for hours to create an account.

Fri Mar 1 11:00:15 2024: error: OTRS-CGI-10: Can't write '/opt/otrs/var/tmp/CacheFileStorable/User/5/2/52f851d8292246f80d324eacb84f5bc5': No such file or directory