Page MenuHomePhabricator

User skin preference for MonoBook changed to default (Vector) on small and medium wikis (about 95% of wikis)
Closed, ResolvedPublic

Description

I'm noticing on mediawiki.org and test.wikipedia.org that my user account is now using the default site skin (Vector) instead of the skin I previously specified in my user preferences (MonoBook).

Later investigation in this task revealed that this was the result of a buggy maintenance script.

Details

Related Gerrit Patches:
mediawiki/extensions/WikimediaMaintenance : wmf/1.27.0-wmf.7Add changeSkinPref
mediawiki/extensions/WikimediaMaintenance : masterAdd changeSkinPref
mediawiki/extensions/WikimediaMaintenance : masterRemove cleanupSkinPrefs

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

To also note that the change occurred on private wikis too. Not sure whether that is expected or not, just feedback.

Nemo_bis raised the priority of this task from High to Unbreak Now!.Oct 2 2015, 8:07 AM
Aklapper lowered the priority of this task from Unbreak Now! to High.Oct 2 2015, 9:17 AM

No need for highest urgency/priority here without providing reasons. Any user can fix their skin settings with a few clicks and the site has not faced functionality regressions or such.

No need for highest urgency/priority here without providing reasons. Any user can fix their skin settings with a few clicks and the site has not faced functionality regressions or such.

I would argue this is UBN...more than a few people have thought their account was compromised due to the most visible preference changing without them doing it.

demon raised the priority of this task from High to Unbreak Now!.Oct 2 2015, 2:46 PM

Is there anything to be done here at this point? Do we intend to correct the skin user preferences of users who had previously chosen a valid skin on small and medium wikis and whose preferences were inadvertently changed by this buggy script?

It seems to me that the time for un-breaking would have been four days ago, so I'm not sure there's much value in keeping this task open if it's only going to serve as an albatross.

Nemo_bis added a comment.EditedOct 4 2015, 10:16 AM

I hope there are backups for the user_properties table.
If not, I guess we'll need to re-enable PrefSwitch :O

MZMcBride renamed this task from User skin preference changed to default skin (Vector) on mediawiki.org and test.wikipedia.org when it was set to MonoBook to User skin preference changed to default skin (Vector) on when it was set to MonoBook on small and medium wikis (851 of 888 wikis).Oct 4 2015, 3:23 PM
MZMcBride updated the task description. (Show Details)

I hope there are backups for the user_properties table.
If not, I guess we'll need to re-enable PrefSwitch :O

In thinking about ways to clean this up, it seems pretty tricky and it only gets worse with time. One potential approach is that you'd restore the user_properties table for about 850 wikis from 2015-09-28 offline somewhere and then repeatedly query it.

More to the point, though, has the underlying bug (root cause) been identified here? I saw a follow-up changeset in Gerrit (https://gerrit.wikimedia.org/r/242374/), but it wasn't clear to me whether that changeset fixed the issue we hit here. The incident report currently makes it sound as though the root cause still hasn't been identified.

I'm wondering whether it makes sense to require that maintenance scripts make a log of their actions?

Relatedly, I'm wondering whether MariaDB has binlogs that could maybe be used here? @jcrespo: any thoughts?

MZMcBride renamed this task from User skin preference changed to default skin (Vector) on when it was set to MonoBook on small and medium wikis (851 of 888 wikis) to User skin preference changed to default skin (Vector) on when it was set to MonoBook on small and medium wikis (about 95% of wikis).Oct 4 2015, 3:47 PM
greg added a comment.Oct 4 2015, 4:05 PM

Hi all.

Quick update on what has already happened (sorry this wasn't already said on this task yet):

@demon and @jcrespo are working on restoring the information.

After this happened they both worked to get the data from the 24 hour binlog (and they have it). Early next week (in a few days) they will work on fixing the user_properties table.

Yea! The notification is valuable. Thanks.

Having so many wikis affected has been assaulting my visual cortex when.
Shame that I so dislike vector.

Nemo_bis assigned this task to jcrespo.Oct 5 2015, 5:58 AM

Thanks. There can only be one assignee, hence putting jcrespo in the slot. :)

jcrespo removed jcrespo as the assignee of this task.Oct 5 2015, 8:34 AM

Will send a private mail to @greg why.

Tgr updated the task description. (Show Details)Oct 6 2015, 8:38 PM

I've created T114208 to coordinate the recovery. @demon will work on it. Due to the time it has passed since the issue, we may be doing a safe recovery only (restoring preferences that has been deleted), and not updating existing ones due to the risk of annoying users again.

After that, we will gather feedback and decide if some extra work is needed.

greg added a comment.Oct 7 2015, 4:05 PM

Thanks for the update @jcrespo.

demon lowered the priority of this task from Unbreak Now! to Normal.Oct 7 2015, 10:22 PM

The only part that needs cleaning up here is still the linked patch above to prevent this from happening again. Per T114899, due to preference table drift and the lack of any sort of timestamp to clarify when columns were last touched make it far too difficult for the risk.

For the record, we should have EventLogging data in the PrefUpdate schema whenever a user changes their preferences. I don't know if its worth trying to look into using it.

[...] the linked patch above to prevent this from happening again.

What was the bug that caused the rows to go missing or to be incorrectly updated? That's not clear from https://gerrit.wikimedia.org/r/#/c/242374/.

demon added a comment.Oct 7 2015, 10:51 PM

[...] the linked patch above to prevent this from happening again.

What was the bug that caused the rows to go missing or to be incorrectly updated? That's not clear from https://gerrit.wikimedia.org/r/#/c/242374/.

I don't know.

I've created T114208 to coordinate the recovery.

You mean T114899.

What was the bug that caused the rows to go missing or to be incorrectly updated? That's not clear from https://gerrit.wikimedia.org/r/#/c/242374/.

I don't know.

This is an unsatisfactory answer.

demon added a comment.Oct 7 2015, 11:38 PM

I've created T114208 to coordinate the recovery.

You mean T114899.

What was the bug that caused the rows to go missing or to be incorrectly updated? That's not clear from https://gerrit.wikimedia.org/r/#/c/242374/.

I don't know.

This is an unsatisfactory answer.

I don't know...yet? Better? Not knowing doesn't mean we won't find out.

jcrespo changed the task status from Open to Stalled.Nov 4 2015, 7:22 PM

jcrespo changed the task status from "Open" to "Stalled".

What is this stalled on? I still expect my preferences to be fixed on all wikis, FWIW.

jcrespo removed a subscriber: jcrespo.

I truly would just like to see some efforts to get it fixed. I have been to more 200 wikis this last week or so deleting stuff and the PITA that is vector and all that travel distance to delete a file is driving me nutty. I wanted my monobook back.

You could set that preference automatically by adding some code to meta:Special:MyPage/global.js. E.g.:

mw.loader.using( 'mediawiki.api.options' )
.done( function() {
    new mw.Api().saveOption( 'skin', 'monobook' );
} )

See also mw:Help:Extension:GlobalCssJs#Example: Set a global interface language.

Yes, I could if I wanted a universal approach to my components. OR we could have the situation fixed, and fixed for all the users effected.

demon added a comment.Nov 5 2015, 2:50 PM

To quote the child task:

After inspecting the current state of the table, we believe that insert will not have any noticible effect, due to changes done since the change took effect. At this time, recovering the tables may have a worse effect in terms of user affection than not doing anything.
We can recover preferences on an individual bases, though, if they have not been changed since the commit.

Yes, individual preferences should be recovered.

@demon converting that to layman-speak? ...
if a person wants theirs fixed they have to lodge a Phabricator? Noting it
will replicate that point in time.

demon added a comment.Nov 5 2015, 8:34 PM

@demon converting that to layman-speak? ...
if a person wants theirs fixed they have to lodge a Phabricator? Noting it
will replicate that point in time.

If they comment here it can be done.

@demon at a convenient time (in the short term) would you please recover
the @Billinghurst skins settings xwiki. Thanks.

Nemo_bis added subscribers: hashar, gerritbot, Matanya, jcrespo.

If they comment here it can be done.

This is not a responsible way to proceed. The population of users who will find this task is not the main population affected by the bug. We have to take care of those who will think their account has been compromised and so on, especially the inactive users. To start with, preferences should be restored for all users who had no user_properties updates since the bug.

@Nemo_bis Please do not add me back. My work here as a DBA concluded the moment I recovered the preferences from the backup.

jcrespo removed a subscriber: jcrespo.Nov 6 2015, 7:45 AM
Catrope removed a subscriber: Catrope.Nov 13 2015, 8:07 PM
hashar removed a subscriber: hashar.Nov 16 2015, 12:27 PM

Shameful how WMF employees here remove themselves from the bug when it is not fixed.

I don't know...yet? Better? Not knowing doesn't mean we won't find out.

Has the main bug here been identified and resolved? The only purpose I see to this task at this point is to adequately answer the questions:

  • why were user preferences lost?
  • how can this type of data loss be prevented from happening again?
demon added a comment.Nov 18 2015, 7:43 AM

I don't know...yet? Better? Not knowing doesn't mean we won't find out.

Has the main bug here been identified and resolved? The only purpose I see to this task at this point is to adequately answer the questions:

  • why were user preferences lost?

The script sucks and the dataset is horrible.

  • how can this type of data loss be prevented from happening again?

Don't bother trying to clean it up anymore.

  • why were user preferences lost?

The script sucks and the dataset is horrible.

  • how can this type of data loss be prevented from happening again?

Don't bother trying to clean it up anymore.

Just to recap here:

  • You ran an untested and damaging script (cleanupSkinPrefs.php) in production across hundreds of Wikimedia wikis.
  • You didn't log the use of the script in the production server admin log.
  • The script caused temporary performance problems in production.
  • The script didn't keep its own logs of what destructive actions it was taking.
  • You committed to cleaning up the mess you made, but then abandoned that plan after realizing it would be difficult.
  • You committed to fixing individual users' preferences, but have ignored the repeated requests on this task to do so.
  • The underlying issue that caused this data loss has still not been found and addressed.
  • When pressed, the explanation provided is that "the script sucks" and you've seemingly washed your hands of this whole mess, without resolving any of it.

I completely understand that shit happens. People make mistakes, code has bugs, etc. The data loss here is an unfortunate mistake and is forgivable.

However, in my mind what is absolutely unacceptable is that we're approaching nearly two months since this incident occurred and there's been no progress made in diagnosing and addressing what went wrong.

The action, inaction, and reaction here are so unbecoming that I'm strongly inclined to petition the Wikimedia Foundation operations team to revoke your shell access.

demon added a comment.Nov 18 2015, 7:10 PM

The action, inaction, and reaction here are so unbecoming that I'm strongly inclined to petition the Wikimedia Foundation operations team to revoke your shell access.

No need to petition, I'll volunteer: https://gerrit.wikimedia.org/r/#/c/253951/

Aklapper added a subscriber: Qgil.Nov 18 2015, 8:24 PM

@MF-Warburg: Even with frustration around, please criticize ideas or prioritization of this specific issue instead of a group of people. Thank you for keeping Phabricator a friendly place!

Qgil added a comment.Nov 18 2015, 11:17 PM

To those that have contacted @Aklapper and/or me about @MZMcBride's comment in T114208#1814338: we have sent him a request to remove or edit his comment, as a contribution to a respectful and friendly environment for everyone.

If you have additional concerns, please contact us directly.

To those that have contacted @Aklapper and/or me about @MZMcBride's comment in T114208#1814338: we have sent him a request to remove or edit his comment, as a contribution to a respectful and friendly environment for everyone.

Interesting. The only person to contact me with this type of request was you, Quim. I'm genuinely very curious which specific parts of my comment here you feel I should censor.

The (somewhat outside) feedback I've received regarding this task is that there are deep and serious flaws in the Wikimedia Foundation's postmortem process.

[...] I'm not sure there's much value in keeping this task open if it's only going to serve as an albatross.

I still agree with myself from October 3, 2015.

Except I still have my request above (as per explained process) for the restoration of my skin preference settings. Still hoping that it can be done.

Just to recap here:

This recap seems useful, I suggest to add it to the post-mortem.

ori added a subscriber: ori.Nov 19 2015, 9:50 AM

Except I still have my request above (as per explained process) for the restoration of my skin preference settings. Still hoping that it can be done.

Would you like me to set the preference to MonoBook (or some other skin) across all wikis on which you have a local account? Or did you have different skin preferences for different wikis?

@ori They were predominantly "monobook" though not exclusively, there were cases of "vector" (<10). If it is just easier to reset to monobook universally, then just do it, and I will adjust the handful that weren't.

ori added a comment.Nov 19 2015, 4:57 PM

If it is just easier to reset to monobook universally, then just do it, and I will adjust the handful that weren't.

Great, thanks. It is technically possible to recover your exact skin preferences from a backup, but merging those into the current state of the databases is just risky enough to make me uncomfortable. I am ready to reset your preference to MonoBook globally, though. Please confirm, and I'll go ahead.

greg added a comment.Nov 19 2015, 6:05 PM

All: I want to apologize for the response on this task on behalf of everyone involved from the WMF.

The original mistake happened, and that is what it is (outages, bad code, etc happens). The follow-up response was sub-par.

I am speaking with all WMF people involved that I can (privately first) to work out any issues that they may have.

The public portion of this response will be the actions we all take to continue to rectify the situation as much as possible from where we are now (eg: reseting individual users' preferences by request).

ori added a comment.Nov 19 2015, 9:50 PM

TODOs

Around this point is also when people reported that the small and medium wikis...lost their monobook skin preferences. The cause...not immediately clear...looks to be a bug in the cleanupSkinPrefs code.

  • On the next edition of Tech News, include a notice that invites users who were affected by this bug to get in touch if they would like a skin preference to be set globally for their account.

If it is just easier to reset to monobook universally, then just do it, and I will adjust the handful that weren't.

Great, thanks. It is technically possible to recover your exact skin preferences from a backup, but merging those into the current state of the databases is just risky enough to make me uncomfortable. I am ready to reset your preference to MonoBook globally, though. Please confirm, and I'll go ahead.

@ori Make it so. Thanks.

ori added a comment.Nov 20 2015, 9:10 AM

@ori Make it so. Thanks.

Done. Please confirm, if you can. Thanks for your patience with this.

@ori Looks good, check 10 wikis, including 2 private, and situation is now all monobook. Thanks. Patience is cheap, and frustrations are a weakness that I (imperfectly) try to better manage. Thanks for your attention to the matter. Blame might be fun for some, however, in my experience it is usually a short-sighted view of a complex set of matters gone wrong, and the approach to repatriation are usually the best indication of wanting things to be better.

From my point of view this is complete and should be closed, and suggest that a new ticket be set up to manage future requests, and one that links back to here (for the intrepid)

demon added a comment.Nov 20 2015, 3:30 PM

Make the code in https://gist.github.com/atdt/b742e8ba56456f82427d more generic and adapt it for inclusion in the #WikimediaMaintenance extension.

Done, up for review: https://gerrit.wikimedia.org/r/#/c/254407/

Change 242374 had a related patch set uploaded (by Greg Grossmeier):
Fix cleanupSkinPrefs

https://gerrit.wikimedia.org/r/242374

Change 254407 had a related patch set uploaded (by Greg Grossmeier):
Add changeSkinPref

https://gerrit.wikimedia.org/r/254407

From my point of view this is complete and should be closed, and suggest that a new ticket be set up to manage future requests, and one that links back to here (for the intrepid)

Almost, let's get those two changes merged first (that I just linked here) before closing this task. But I agree with the new task for managing reseting requests. Created as T119206.

Change 254407 merged by jenkins-bot:
Add changeSkinPref

https://gerrit.wikimedia.org/r/254407

Keegan added a subscriber: Keegan.Nov 20 2015, 9:11 PM

Change 242374 merged by Ori.livneh:
Remove cleanupSkinPrefs

https://gerrit.wikimedia.org/r/242374

ori closed this task as Resolved.Nov 20 2015, 9:59 PM
ori assigned this task to demon.
ori removed a project: Patch-For-Review.

Change 256031 had a related patch set uploaded (by Chad):
Add changeSkinPref

https://gerrit.wikimedia.org/r/256031

Change 256031 merged by jenkins-bot:
Add changeSkinPref

https://gerrit.wikimedia.org/r/256031

Krinkle renamed this task from User skin preference changed to default skin (Vector) on when it was set to MonoBook on small and medium wikis (about 95% of wikis) to User skin preference for MonoBook changed to default (Vector) on small and medium wikis (about 95% of wikis).Dec 2 2016, 3:10 AM