Yep! https://www.wikifunctions.org/wiki/Z36261?uselang=en&oldid=282025

I'd love to see this get taken care of soon. There's a Programs & Events Dashboard program that's trying to track uwikisource, and it won't be able to pull any revision data until Replica can connected to to it. (Incidentally, this led to flooding the Dashboard's logging system, as we hadn't had to deal with an existing wiki with replica connection errors before.)

In T423788#11839220, @Geertivp wrote:

It seems that the original error still appears when the user does not have an active Wikimedia session.

Fixed and deployed with https://github.com/WikiEducationFoundation/WikiEduDashboard/commit/76edcd4c408552d45d17974d83ed3a21668c8b4e

I prototyped something similar today (vibe coded), and took some ideas from yours: https://ragesoss.github.io/zblocks/

In T423781#11835994, @GrounderUK wrote:

I think this duplicates T419561 or, at least, it’s a subset.

Abstract Wikipedia team: I've tried to get everything in as near-complete shape as I could all at once, and do all the things I could think to do to make it usable, but I don't know whether all this AI code and description is worth your time. If it's not, please just close this and accept my apologies for the Phab and GitLab noise. I'm trying to scratch my own itch for what I want to do on Wikifunctions, and thought I'd give it a shot, since Claude did well enough with https://www.wikifunctions.org/wiki/Talk:Z16684

In T307878#11709728, @TheDJ wrote:

This makes sense. Thanks for the clear description! I've added an issue on the Dashboard's GitHub repo.

@LGoto I know that I missed the February 25 deadline, but I hope you can make an exception. I mistakenly thought the deadline was the Outreachy one (March 7). I've already posted this project on outreachy.org.

@LGoto done!

In T407660#11556351, @HMonroy wrote:

Hi @Ragesoss ! Is this something you are looking into? We are wondering if Community-Tech should take a look. Thank you for your help in advance :)

The vast majority of the storage is on the two mounted volumes, so I think a command like that would be hard to get useful info from unless it's scoped to /dev/sda1.

See also https://phabricator.wikimedia.org/T415603

@Joe checking my Sentry logs, I see we're still getting 429 for some types of queries, including Commons API queries and fetching page content (but not the OAuth login flow). Are there requests we're still making that don't meet the UA policy requirement? (We have several different libraries involved for requests, and I think I addressed these ones, which go through the mediawiki_api Ruby gem, but it's possible that I missed something.)

I did a quick simple intervention today, deleting a portion of the pickles that hadn't been accessed in 90 days or more. Any that get requested will be regenerated, so it should only cause some minor inconvenience when the data isn't available on first request. (I ran a simple script that was looping through the en directory, and deleting each file if its atime was earlier than 90 days ago. I was disconnected with a broken pipe before it finished, so I don't know how much more it would have deleted it it ran to completion.)

@ssingh I've just deployed an update that should fix it. Now the user agent is Wiki Education Dashboard/1.0 (dashboard.wikiedu.org; sage@wikiedu.org).

Thanks! Unfortunately, the OAuth library we use doesn't support setting the User Agent, so I'm going to have to figure out how to monkey patch it. :-(

Day 5: 2025-12-08

In T411890#11438103, @Cookroach wrote:

Thanks you very much Ragesoss for your restoration work. Which data records or which time period have actually been lost and cannot be restored? Background: We have a challenge that started on December 1. best regards

Day 2: 2025-12-05

Day 1: 2025-12-04

Excellent, thanks for creating the GitHub issue.

Sounds like a good idea. I've created an issue for it.

It's worth noting that we could enable additional languages without an initial processing run; in that case, every new request would trigger processing that particular article (and result in slower, but still usable, performance for the first request for each article for Who Wrote That and Dashboard end users). I think that strategy would be fine for enabling all additional languages; the only blocker would be making sure we have appropriate storage for handling the gradual accumulation of processed articles files for the additional languages (and clearing some headway for continued growth of storage requirements for the already-enabled ones).

@ccasilli @LGoto we're getting close to the deadline here; the community signup deadline is less than 24 hours away (September 17 at 4pm UTC) and the mentor project listing deadline is September 27.

When will we start listing projects?

I think I have a good guess at what happened, a race condition of sorts.

In T401873#11095223, @Daimona wrote:

@Ragesoss Hi! This course was associated to this event on-wiki on July 28th. However, it looks like it is no longer linked. This did not happen on our side AFAICT. Do you have an idea of how it might have happened, and if it could be prevented going forwards? Thanks!

Here's a nice visualization of the progress we made on performance problems since the start of the internship, in the form of the downtime monitor for Programs & Events Dashboard.

Thanks! I was unable to implement this last week, as I was travelling.

When does your edit-a-thon start?

Yes, that one seems like the same thing to me.

@Seb35 it seems closely related, but not quite the same. T280078 is about parameters that should be allowed to be empty without automatically getting removed, and this one is about preventing the addition of undocumented (and non-functional) parameters.

Thanks! I just processed the backlog of account requests on Programs & Events Dashboard, and things are working normally again.

Any update on this? We still have account requests coming in regularly on Programs & Events Dashboard for editathons and such, and can't process them.

Wiki Education continues to get reports of this error from new student editors. This is the most common error we hear about from users lately.

@Ladsgroup thanks! Do you have any more details about the ones still happening, like an example request?

@MusikAnimal thanks! I just pulled in the change and restarted all the services, so it should be live now.

I've implemented a fix here: https://github.com/wikimedia/wikiwho_api/pull/20/files

I think it's common for users who have lost/forgotten their password to attempt enough tries with password guesses to trigger the 'failed attempts' email, so some path that leads to a password reset makes sense there. The 'change password' link the email is easily interpreted to be a synonym for 'reset password'.

I mean, in the case of the user not seeing the password reset link from https://auth.wikimedia.org/enwiki/wiki/Special:UserLogin (where it appears for me) I assume it's because of the block.

I guess the 'Forgot password' link is not appearing on the login page for that reason as well, but it's very confusing for an affected user because there's no indication of why the link does not appear, if they are looking for it.

Further info from the user: When navigating directly to https://auth.wikimedia.org/enwiki/wiki/Special:PasswordReset, the user ended up reaching an error message indicating an IP block:

@Tgr is this related to SUL3?

The requests definitely include an email address (and no password). The account creation flow from the Dashboard was working at least as recently as March 28, so I assume it's something that changed in the April 1 deployment.

Hmm... I am getting this error with a test account request as well. My first guess was that it's related to T390437, which just deployed.

Thanks Esther! Great work!

Thanks Formasit! Great work!

I've just deploy this. Thanks to Betty Alagwu for implementing it!

Okay, I think I've finally found the major cause of the downtime that continued happening after the data rearchitecture deployment! I deployed a fix today, and I'm very optimistic that it will be much more stable going forward.

We've gotten through nearly all of the first updates now, and I've re-initiated update processing for the already-processed ones. The last few are still working through their first updates, but we're cycles through now with about 1 or 2 days of latency for the rest.

We continue chewing threw the backlog, down to 57 events remaining for their first update. We're on the last 20 of the medium queue (up to 10k edits). Once those are complete, I will start steady-state updates for the rest of the already-processed events.

Working through the initial backlog continues apace. We're down to 204 courses remaining, but that still includes the majority of the very large courses.

It looks like things are working smoothly so far. I have added extra workers, since the memory footprint of the update cycle is so much lower now, so there a bunch of processes working through the update backlog. We identified a number of inefficiencies that we can fix to speed up the update cycles, which I'll be deploying as we have them ready, but meanwhile we're down to 515 events in the queues awaiting their first update with the new system. The bigger ones that will take a long time to process are at the end, and may take a while to get through, but I will start steady-state processing for the ones that aren't going to take too long once we've done all the initial processing of the events with fewer than 10k edits.

The deployment is complete, and I've enqueued all the current events for their first cycle through the new update cycle. There are 804 current ones, and the low-revision count ones should get processed first. The others will take a while, I'll keep this issue updated as it progresses.

I'm deploying the major update to the statistics generation system today. I've paused all the data update workers, and I'm currently generating a database dump. After that, I'll deploy and then the system will begin processing all current courses using the new system; it will take quite a while to do the initial round of processing, as it will be pulling in data for all edits since the beginning of each still-active event. Once the initial round of processing is complete, updates should eventually reach a steady state with a better rate that currently.

Thanks! This is a good idea. I will write up an issue for it.

@Daimona the goal will be to get it pretty much completely stable. The Wiki Education instance (dashboard.wikiedu.org) has had ~100% uptime for more than year, and I think it's realistic to get very close to that for Programs & Events Dashboard. (The setup on labs is a little more complicated with system components spread across multiple instances, and also subject to occasional WM Cloud-wide issues, so it will probably not be quite 100%, but I want to get very close.) This update addresses the root cause of the current instability situation (which is that updates for very large events are too demanding of the database), but there are likely to be secondary bottlenecks that reveal themselves once we get it into production, so there might be some further work to do to get it really stable. The update will completely eliminate the bottleneck queries against the Dashboard's revisions database, which are the main culprits, so if all goes well, even if large events still cause problems in terms of the update cycle, it should stop taking down the web server along with it.

Yes, I'm very close to deploying a major update intended to get the system into a stable state. In the meantime, I will continue restarting the instance that runs the web server when it goes down (which typically happens because the database server is overloaded because of non-web requests from data updates).

@Samwalton9-WMF this time, it was good old-fashioned unresponsive server (so the homepage also was not loading). I believe this happens when the combination of web requests and the background workers use up all the database compute; without intervention, it typically comes back online within an hour, and restarting the peony-web instance via Horizon fixes it quickly.

Yes, I think it was related. It was hanging on fetching two Vega libraries that are hosted on toolforge (instead of a commercial CDN). I stopped and started the webservice for wikiedudashboard tool, and that fixed it.

Thanks for your contributions @Tunrayono

One case of it happened in Safari 16.4.1 .

In T381673#10387136, @matmarex wrote:

I guess the error message was 'deflate-invaliddeflate': "Content provided is not properly deflated"?

I've restarted the webserver and it's back up.

I received a screenshot of a console error from one of the affected users. It looks like it's "Remove does not match insert" from TransactionSquasher that is being thrown, but there's no user-facing indication of the problem or how to solve it.