I'm noticing on mediawiki.org and test.wikipedia.org that my user account is now using the default site skin (Vector) instead of the skin I previously specified in my user preferences (MonoBook).
Later investigation in this task revealed that this was the result of a buggy maintenance script.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T340453 [Epic] FY 2023-24 Growth Maintenance Work | |||
| Resolved | KStoller-WMF | T347840 [Epic] Q2 FY 2023-24 Growth Maintenance Work | |||
| Open | None | T54777 user_properties table bloat | |||
| Resolved | • demon | T114208 User skin preference for MonoBook changed to default (Vector) on small and medium wikis (about 95% of wikis) | |||
| Duplicate | • demon | T114899 Recover missing values from user_properties tables |
To also note that the change occurred on private wikis too. Not sure whether that is expected or not, just feedback.
No need for highest urgency/priority here without providing reasons. Any user can fix their skin settings with a few clicks and the site has not faced functionality regressions or such.
I would argue this is UBN...more than a few people have thought their account was compromised due to the most visible preference changing without them doing it.
Is there anything to be done here at this point? Do we intend to correct the skin user preferences of users who had previously chosen a valid skin on small and medium wikis and whose preferences were inadvertently changed by this buggy script?
It seems to me that the time for un-breaking would have been four days ago, so I'm not sure there's much value in keeping this task open if it's only going to serve as an albatross.
I hope there are backups for the user_properties table.
If not, I guess we'll need to re-enable PrefSwitch :O
In thinking about ways to clean this up, it seems pretty tricky and it only gets worse with time. One potential approach is that you'd restore the user_properties table for about 850 wikis from 2015-09-28 offline somewhere and then repeatedly query it.
More to the point, though, has the underlying bug (root cause) been identified here? I saw a follow-up changeset in Gerrit (https://gerrit.wikimedia.org/r/242374/), but it wasn't clear to me whether that changeset fixed the issue we hit here. The incident report currently makes it sound as though the root cause still hasn't been identified.
I'm wondering whether it makes sense to require that maintenance scripts make a log of their actions?
Relatedly, I'm wondering whether MariaDB has binlogs that could maybe be used here? @jcrespo: any thoughts?
Hi all.
Quick update on what has already happened (sorry this wasn't already said on this task yet):
@demon and @jcrespo are working on restoring the information.
After this happened they both worked to get the data from the 24 hour binlog (and they have it). Early next week (in a few days) they will work on fixing the user_properties table.
Yea! The notification is valuable. Thanks.
Having so many wikis affected has been assaulting my visual cortex when.
Shame that I so dislike vector.
I've created T114208 to coordinate the recovery. @demon will work on it. Due to the time it has passed since the issue, we may be doing a safe recovery only (restoring preferences that has been deleted), and not updating existing ones due to the risk of annoying users again.
After that, we will gather feedback and decide if some extra work is needed.
The only part that needs cleaning up here is still the linked patch above to prevent this from happening again. Per T114899, due to preference table drift and the lack of any sort of timestamp to clarify when columns were last touched make it far too difficult for the risk.
For the record, we should have EventLogging data in the PrefUpdate schema whenever a user changes their preferences. I don't know if its worth trying to look into using it.
What was the bug that caused the rows to go missing or to be incorrectly updated? That's not clear from https://gerrit.wikimedia.org/r/#/c/242374/.
jcrespo changed the task status from "Open" to "Stalled".
What is this stalled on? I still expect my preferences to be fixed on all wikis, FWIW.
I truly would just like to see some efforts to get it fixed. I have been to more 200 wikis this last week or so deleting stuff and the PITA that is vector and all that travel distance to delete a file is driving me nutty. I wanted my monobook back.
You could set that preference automatically by adding some code to meta:Special:MyPage/global.js. E.g.:
mw.loader.using( 'mediawiki.api.options' ) .done( function() { new mw.Api().saveOption( 'skin', 'monobook' ); } )
See also mw:Help:Extension:GlobalCssJs#Example: Set a global interface language.
Yes, I could if I wanted a universal approach to my components. OR we could have the situation fixed, and fixed for all the users effected.
To quote the child task:
After inspecting the current state of the table, we believe that insert will not have any noticible effect, due to changes done since the change took effect. At this time, recovering the tables may have a worse effect in terms of user affection than not doing anything.
We can recover preferences on an individual bases, though, if they have not been changed since the commit.
@demon converting that to layman-speak? ...
if a person wants theirs fixed they have to lodge a Phabricator? Noting it
will replicate that point in time.
@demon at a convenient time (in the short term) would you please recover
the @Billinghurst skins settings xwiki. Thanks.
If they comment here it can be done.
This is not a responsible way to proceed. The population of users who will find this task is not the main population affected by the bug. We have to take care of those who will think their account has been compromised and so on, especially the inactive users. To start with, preferences should be restored for all users who had no user_properties updates since the bug.
@Nemo_bis Please do not add me back. My work here as a DBA concluded the moment I recovered the preferences from the backup.
Has the main bug here been identified and resolved? The only purpose I see to this task at this point is to adequately answer the questions:
The script sucks and the dataset is horrible.
- how can this type of data loss be prevented from happening again?
Don't bother trying to clean it up anymore.
Just to recap here:
I completely understand that shit happens. People make mistakes, code has bugs, etc. The data loss here is an unfortunate mistake and is forgivable.
However, in my mind what is absolutely unacceptable is that we're approaching nearly two months since this incident occurred and there's been no progress made in diagnosing and addressing what went wrong.
The action, inaction, and reaction here are so unbecoming that I'm strongly inclined to petition the Wikimedia Foundation operations team to revoke your shell access.
@MF-Warburg: Even with frustration around, please criticize ideas or prioritization of this specific issue instead of a group of people. Thank you for keeping Phabricator a friendly place!
To those that have contacted @Aklapper and/or me about @MZMcBride's comment in T114208#1814338: we have sent him a request to remove or edit his comment, as a contribution to a respectful and friendly environment for everyone.
If you have additional concerns, please contact us directly.
Interesting. The only person to contact me with this type of request was you, Quim. I'm genuinely very curious which specific parts of my comment here you feel I should censor.
The (somewhat outside) feedback I've received regarding this task is that there are deep and serious flaws in the Wikimedia Foundation's postmortem process.
I still agree with myself from October 3, 2015.
Except I still have my request above (as per explained process) for the restoration of my skin preference settings. Still hoping that it can be done.
Would you like me to set the preference to MonoBook (or some other skin) across all wikis on which you have a local account? Or did you have different skin preferences for different wikis?
@ori They were predominantly "monobook" though not exclusively, there were cases of "vector" (<10). If it is just easier to reset to monobook universally, then just do it, and I will adjust the handful that weren't.
Great, thanks. It is technically possible to recover your exact skin preferences from a backup, but merging those into the current state of the databases is just risky enough to make me uncomfortable. I am ready to reset your preference to MonoBook globally, though. Please confirm, and I'll go ahead.
All: I want to apologize for the response on this task on behalf of everyone involved from the WMF.
The original mistake happened, and that is what it is (outages, bad code, etc happens). The follow-up response was sub-par.
I am speaking with all WMF people involved that I can (privately first) to work out any issues that they may have.
The public portion of this response will be the actions we all take to continue to rectify the situation as much as possible from where we are now (eg: reseting individual users' preferences by request).
Around this point is also when people reported that the small and medium wikis...lost their monobook skin preferences. The cause...not immediately clear...looks to be a bug in the cleanupSkinPrefs code.
@ori Looks good, check 10 wikis, including 2 private, and situation is now all monobook. Thanks. Patience is cheap, and frustrations are a weakness that I (imperfectly) try to better manage. Thanks for your attention to the matter. Blame might be fun for some, however, in my experience it is usually a short-sighted view of a complex set of matters gone wrong, and the approach to repatriation are usually the best indication of wanting things to be better.
From my point of view this is complete and should be closed, and suggest that a new ticket be set up to manage future requests, and one that links back to here (for the intrepid)
Make the code in https://gist.github.com/atdt/b742e8ba56456f82427d more generic and adapt it for inclusion in the #WikimediaMaintenance extension.
Done, up for review: https://gerrit.wikimedia.org/r/#/c/254407/
Change 242374 had a related patch set uploaded (by Greg Grossmeier):
Fix cleanupSkinPrefs
Change 254407 had a related patch set uploaded (by Greg Grossmeier):
Add changeSkinPref
Almost, let's get those two changes merged first (that I just linked here) before closing this task. But I agree with the new task for managing reseting requests. Created as T119206.