In T284119#7275669, @MMiller_WMF wrote:

@Ragesoss -- thanks for the update! Did you end up implementing it such that all users have the features on? Or so that all users have the features off?

In T284119#7272798, @Urbanecm_WMF wrote:

In T284119#7272774, @Tgr wrote:

It would work for the account creation API as well, I think, although you'd get an invalid parameter warning as the parameter is not formally registered.

It's definitely not a behavior we tried to test -- so even if it works, I don't think it should be relied upon.

@kostajh thanks much!

Yesterday the system was broken for several hours because of a full disk on p-and-e-dashboard-database. This is because I hadn't configured the database to use Cinder volume for the tmpdir, so temporary tables filled up the small root partition.

I'm working on this now, and I have a few questions.

@MMiller_WMF : I haven't implemented it yet — it's near the top of my TODO list — but I think it should work fine. I'll report back once I do implement and deploy it, likely this week. Thanks!

Note that the same thing (with a slightly different href, https://#) also affects the New Post interface: https://diff.wikimedia.org/wp-admin/post-new.php

This is live now.

In T284119#7149725, @Urbanecm_WMF wrote:

In T284119#7149341, @Ragesoss wrote:

Is there a method available right now that I could implement to mark Wiki Education users?

I think that mostly depends on "do we want to force-enable the features, or force-disable". by reading this task, it's still not clear to me which one of those two things is desired here.

Thanks, this is helpful feedback.

Is there a method available right now that I could implement to mark Wiki Education users? Some of them are new accounts where we could set an additional param in the account creation URL, but some create their accounts on their own before joining a course on the Dashboard. But we do attempt to make an options edit immediately after a user self-enrolls in a course, so rolling some kind of identification setting into that will probably be the most reliable way to handle it. (This does not happen for cases where the instructor adds the username to the course, rather than the user doing it themselves while signed in to the Dashboard, but that's relatively rare... and those accounts won't get opted out of any experimental features anway.)

This happened to be a very simple change: https://github.com/WikiEducationFoundation/WikiEduDashboard/commit/1cbb98730efc2592ba9eaa5cc336b45c75bd08a4

This makes sense. I've added an issue on GitHub pointing here.

This is worth considering, but I'm not sure it's worth it to increase the complexity of the UI to add a timezone picker. We're currently storing all times as UTC, but converting to the user's local timezone on the frontend, which is a good solution for most users. But perhaps it could be done in a way that's unobtrusive enough to be worth it.

I've added a fix for this, and it should be live after the next deployment... likely next week sometime.

This is a good idea. I've added an issue for it: https://github.com/WikiEducationFoundation/WikiEduDashboard/issues/4526

@Urbanecm_WMF thank you! this is very helpful.

In T284119#7128463, @kostajh wrote:

I was thinking we could use a query parameter at registration time, something like geOptOut=1, that would be checked in the various LocalUserCreated implementations (help panel, homepage, mentor), and that would basically prevent the code in those hooks running for users created via that path. @Ragesoss would that work for your use case?

Thanks all! Martin helped me get it sorted out: I disabled the global block locally for en.wikipedia, and updated the local version of it to allow logged-in edits. Dashboard edits are working again now.

Thanks. I thought there was something else letting the dashboard IP get around a global block, because it looks like the old IP is also in a blocked range: https://meta.wikimedia.org/wiki/Special:Contributions/45.56.98.206

Thanks @Zabe! Is this live already? We're still having all our edits blocked like this:

2600:3c03::f03c:91ff:fe08:7973/128 is the ipv6 range that went with the 45.56.98.206 server, so I believe the ipv6 line from T151823 can be replaced as well.

Today I set up an additional node to process the long program update queue. My best guess is that running out of memory is the source of the problems we've had lately (ie, the server going down until an Apache restart), so moving that most resource-intensive process to a separate server should help.

I set up and switched over to a fresh VM today; with the separate database server, this was completely seamless as all I needed to do once it was set up was to change the Web Proxy on Horizon to point to the new VM.

My guess is that this is the same issue that affects users of the frontend when Apache gets into a state where a portion of the requests immediately return a 500. I still don't know exactly what causes it — likely a slow route using up most of the application threads. The system has been stable since we restarted the server yesterday. Have you had any more errors in the last ~21 hours?

I'm back into the system now after some help with restarting the server and fixing LDAP integration. My work preparing for a distributed architecture is still in progress, but it's looking good; I hope to have it much more stable and resilient a few months from now.

I'm not sure what is wrong, but it might be related to the recent NFS outage on cloud services... I can't currently log in to the server, as I'm getting Permission denied (publickey)

Couldn't find the immediate cause — didn't seem to be a problem at the application layer — but it's been up since the restart. We're to switch from Apache to nginx as part of the larger infrastucture update that's in the works in the coming months.

I restarted Apache and that brought it back up. Will investigate the cause.

Will do.

In T279956#6998872, @aborrero wrote:

Could you please detail how much RAM and CPU you would need in your quota?

@Ederporto I've enabled pt.wikiversity edits. Please test it out and confirm that it's working properly.

I've added the configuration for this: https://github.com/WikiEducationFoundation/WikiEduDashboard/commit/5130e8ae4033114259074a05ca61ad12d50c772d

I fixed the too-long-SQL-queries problem! After the switch to a separate database server, the Rails app was still connecting to the database using the mysql2 gem built against MariaDB 10.1, but the database server is now running MariaDB 10.3. I upgraded libmariadb-dev to 10.3 on the Rails server, rebuilt the gem, and the problem went away. :-D

This project hasn't been done; it might make a good project for a future round, although it's not a top priority.

My MySQL docs point to this possible problem:

There's also one more remaining case of too-long-SQL-queries that I can't work around as easily as the others; for an ArticleScopedProgram that is tracking based on a large category or PetScan query, we use the array of article titles for that category/query to find which edited articles are in scope, and if there are thousands of titles it makes the SQL too long and the database connection cuts off. Here's an example of a program being affected by this problem: https://outreachdashboard.wmflabs.org/courses/wikidata/Museum_Day_2021_-_Museums_(2021)/articles/edited

I've now re-enabled updates for all programs except for the following ones which are disproportionately responsible for high system load, which I need to study more to find possible bottlenecks before I'll feel safe re-enabling. Many of them are ones that track very large numbers of articles over long time periods; I may need to add some limitations such as not allowing programs that track activity for more than 1 or 2 years.

Thanks for getting to the bottom of this!

Oh, I see! I never noticed before that this example query was returning duplicates of the same revision; the intent was to return just one entry for a given revision, and have it report system: true if it was from any of the specified change tags. I'll update my test accordingly. Wiki Education Dashboard has been fetching duplicate revisions for a while now, which might explain why the system indication has been incosistent... it depended on which copy of the revision got written to the database first. (Programs & Events Dashboard just has one change tag specified in its config, so this won't have made an operational difference there.)

No, my code doesn't rely on the order of the results.

Here's a query:

In T277651#6960333, @Joalpe wrote:

Sage, have you checked if a memory module is failing in the respective host server?

Could you update the MariaDB server to the latest minor version ?

I've now disabled updates for all the 'long' programs, which are ones that have taken longer than 10 minutes for a data update. Many of these which take between 10 and 30 minutes are probably fine, and I'll start re-enabling updates for such pages. I'll be adding a message shortly to indicate when a particular program has updates disabled. For the longest ones, I plan to do more work to limit the time-consuming parts of the process (likely by disabling some features) so that updates can eventually be re-enabled.

I've finished reloading the revisions table, brought the web server back up, and turned on the background processes for short and medium events and 'constant' updates. I'll turn on the 'daily' updates process in a little while.

I've completed the revisions.sql dump (33G), dropped the revisions table, taken it out of recovery mode, and started restoring the table (mysql dashboard < revisions.sql).

I've taken the Dashboard offline, put the database in innodb_force_recoery=1 mode, and started mysqldump dashboard revisions > revisions.sql. It'll take perhaps a couple of hours to drop and reload the revisions table, after which I'll switch out of recovery bring the frontend back online while I work on isolating the events that take too long.

This error at 11:31 UTC March 28 happened right before the index corruption:

The system had looked quite stable, but about 4 hours ago, all the update processes once again started erroring due to a corrupted revisions index. So far, I haven't found any hints at what the acute cause is. Given that this happened again after moving the database to a new server and a Cinder volume, it seems like a hardware problem is a very unlikely reason.

The adjustments I made to avoid long query strings are working so far, and all update processes are currently back online. I'll continue monitoring closely for additional problems.

I'm still stumped on why larger query strings are causing the app to lose its database connection. I've started putting in place some workarounds to avoid needing such long query strings in the places that have failed so far.

I already tried increasing net_write_timeout as well, but that didn't help.

Yeah, I've been looking at that. It points to the problem most likely being network related, rather than server related. I ran tcpdump through a disconnection and then used strings to see what the traffic looked like, but it offers no useful hints that I can glean.

Also tried:
innodb_buffer_pool_size=12GB

I tried these settings as well, to no avail:

I've set log_warning=4 to surface logs from these connection errors on the mysqld side. They look like this: 2021-03-25 15:47:13 10 [Warning] Aborted connection 10 to db: 'dashboard' user: 'outreachdashboard' host: 'programs-and-events-dashboard.globaleducation.eqiad1.wikimed' (Got an error reading communication packets)

The web app is largely working now, but seems to be a problem with the database configuration when the system tries to run some of the larger queries, which trigger a connection error. Investigating it today.

The database population step is complete (using up 61G out of 177G on the Cinder volume). I've turned the Dashboard web server back on, and I'll gradually enable related services.

Database population seems to have sped up some, and is now estimated to complete in 2.5 more hours (which means I might be able to get the Dashboard up and running this evening).

Meanwhile, I'm setting up a pair of test VMs to try out running the system in this two-server configuration to identify any extra hurdles to expect once the database finishes loading.

In T277651#6942056, @Nattes wrote:

Hi can we have more details on when the dashbpard problem will be resolved? Just to know if it will be oprational this week end

I've got the backup file transferred to the new VM, I've configured mariadb to use /srv/mysql, and I've started loading the data (pv 2021-03-23_22h10m_12.sql.gz | gunzip | mysql dashboard). It looks like it's going to take all day to complete the load, so it will probably be tomorrow when I can try to bring the Dashboard back online with the new database server.

The backup is complete. I've spun up a new VM to put the database on. Setting up the Cinder volume was a breeze; I used 180 of the 200GiB quota for this volume (an uncompressed backups is huge, so this doesn't actually provide a huge amount of headroom, but it should be fine for now).

As before, running a dump errors out in the revisions table, so I've edited /etc/mysql/mariadb.conf.d/50-server.cnf again to set innodb recover mode 1. This time I'll do a complete backup and then use that .sql file to populate a new database on a new VM (with MariaDB 10.3, instead of 10.1 which it's currently using).

In T278132#6939532, @bd808 wrote:

+1 from me for granting 200GiB of Cinder quota today.

One of the very nice things about Cinder volumes is that they are resizable, so if this space becomes well used additional quota can be granted to grow the volume(s) in the future with minimal disruption to the project.

Okay.

Thanks @Legoktm . I just took the system down to run a dump. Not sure sure exactly how to hook up the database to a separate VM, but I think I can figure it out. I'll attempt to go with the move @bd808 suggests.

I'm currently unsure what the cause of the corruption is, but I've read this "Most InnoDB corruptions are hardware-related. Corrupted page writes can be caused by power failures or bad memory. The issue also can be caused by using network-attached storage (NAS) and allocating InnoDB databases on it." (source)

Although it's mostly only affecting the update processes for slow-updating programs, it appears that problems with the Revisions index are back. I'm not sure what to do about this in the short term. I'm investigating the idea of moving more processing outside of the Dashboard, but that's not something that could be done quickly.

Can you provide some additional details? What category did you try to track, and what did you enter as the name of it? What project are you doing this for, and what do you want to be able to selectively track?

The disk was full due to a large leftover file from the previous repair effort. Looks like things went back to normal as soon as I freed up space.

This was caused by a full disk.

In T277651#6925855, @Legoktm wrote:

Note that there's nothing stopping you from querying the wikireplicas directly from a VPS project, you just need to create a Toolforge tool to get credentials (replica.my.cnf), which you can then stick in your VM.

I've been working with Rails performance expert Nate Berkopec this month, and we've come to a clearer understanding of the usual cause of recurring outages. In most cases (except for the most recent one), the system gets overloaded through the combination of the background processes (which are responsibile for updating stats for courses and tie up a significant portion of system resources on essentially a continual basis) and multiple users requesting particularly slow pages (ie, ones that load a lot of data). The web server uses up to 6 "Passenger" processes that handle a web request by sending it to Ruby on Rails application. If all 6 of these are tied up with very slow requests, then the system can't process any more requests in the meantime.

To wrap up, my best guess at what happened is:

• Sometime on March 16, the index of the revisions table became corrupted.
• Whenever the database looked at the affected index entry, the database server would crash, interrupting any web requests and background stats jobs that were running at the time.
• By the time the database could restart, the web server had a full queue of unprocessed requests waiting.
• For reasons I don't understand yet, this also tied up the system such that it stopped processing web requests altogether and also became unresponsive via SSH.
• Upon restarting the server, it would run fine for a few minutes until once again encountering the corrupted database index entry.

The system is still stable, so I've just enabled the final Sidekiq processes for long stats updates and the daily job. I've also re-enabled the weekly database backup script (which runs on Sunday), and deployed a minor change to help with logging slower courses.

Still going fine, so I've enabled the short and medium sidekiq processes. The slowest programs, and the daily_update job, are still disabled.

Things are going fine so far. I've just re-enabled the constant_update and schedule_course_updates cron jobs, and the constant_update Sidekiq process that handles them.

The Dashboard is back up and running. I haven't enabled or restarted any of the background processes yet except for the 'default' one that handles transactional background jobs (like wiki edits and error reporting), so it hasn't started pulling in new data yet.

Reloading the revisions table has completed. I'm going to restart the frontend.

@Theklan done. (I added .htaccess, enabled mod_rewrite, and set a Directory stanza with AllowOverride All in the site config.)

After dumping the revisions table, I've done drop table revisions;. Then I took it mysqld out of recovery mode and restarted it. Now I'm running mysql dashboard < revisions.sql to restore the revisions table from the dump.

The backup script ran without issue is recovery mode, so I have an up-to-date, possibly valid backup of the whole DB now. I'm currently running a dump of just the revisions table (mysqldump dashboard revisions > revisions.sql) in preparation for dropping and importing it.