Page MenuHomePhabricator

Unexpected DB insert from DiscussionTools::determineUserABTestBucket during GET request
Closed, ResolvedPublic

Description

Whilst preparing an unrelated deployment and testing something on nl.wikipedia.org, I noticed the following warning in the application logs:

DBPerformance warning
Expectation (writes <=) 0 by MediaWiki::main not met (actual: {actual}) in trx #831889b42a:
query-m: INSERT /* MediaWiki\User\UserOptionsManager::saveOptionsInternal */ IGNORE INTO `user_properties` (up_user,up_property,up_value) VALUES (N,'X')
trace
#8 /srv/mediawiki/php-1.38.0-wmf.22/includes/user/UserOptionsManager.php(465): Wikimedia\Rdbms\DBConnRef->insert(string, array, string, array)
#9 /srv/mediawiki/php-1.38.0-wmf.22/includes/user/UserOptionsManager.php(390): MediaWiki\User\UserOptionsManager->saveOptionsInternal(User, Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.38.0-wmf.22/extensions/DiscussionTools/includes/Hooks/HookUtils.php(185): MediaWiki\User\UserOptionsManager->saveOptions(User)
#11 /srv/mediawiki/php-1.38.0-wmf.22/extensions/DiscussionTools/includes/Hooks/HookUtils.php(106): MediaWiki\Extension\DiscussionTools\Hooks\HookUtils::determineUserABTestBucket(User, string)
#12 /srv/mediawiki/php-1.38.0-wmf.22/extensions/DiscussionTools/includes/Hooks/PreferenceHooks.php(79): MediaWiki\Extension\DiscussionTools\Hooks\HookUtils::isFeatureAvailableToUser(User, string)
#13 /srv/mediawiki/php-1.38.0-wmf.22/includes/HookContainer/HookContainer.php(160): MediaWiki\Extension\DiscussionTools\Hooks\PreferenceHooks->onGetPreferences(User, array)
#14 /srv/mediawiki/php-1.38.0-wmf.22/includes/HookContainer/HookRunner.php(1913): MediaWiki\HookContainer\HookContainer->run(string, array)
#15 /srv/mediawiki/php-1.38.0-wmf.22/includes/preferences/DefaultPreferencesFactory.php(249): MediaWiki\HookContainer\HookRunner->onGetPreferences(User, array)

Source: DiscussionTools.git:HookUtils.php#165

	$optionsManager->setOption( $user, 'discussiontools-abtest2', $abstate );
	$optionsManager->saveOptions( $user );

Impact

The code is synchronously writing to the database regardless of request context. This violates DBPerformance restrictions and is incompatible with performance and multi-dc constraints.

I'd consider it high priority to be written a different way. I don't know if this is intended to go to other wikis, but I'd also consider this a blocker for wider enablement of $wgDiscussionToolsABTest, which I believe to be controlling this particular code.

Consider reviewing how other extensions use this core API (Codesearch). If a viable pattern isn't available or clear, I'm happy to help find or create one.

From Logstash: mediawiki-errors:

Screenshot 2022-02-24 at 18.02.02.png (700×2 px, 174 KB)

Event Timeline

Krinkle updated the task description. (Show Details)

The one that looks somewhat-equivalent is BetaFeatures, which has a very similar "sometimes the getPreferences hook causes DB writes" pattern (auto-enrollment in new features for it). It handles it by using DeferredUpdates::addCallableUpdate to do the actual saveOptions call -- does that pattern work for you? (It also does some ObjectCache locking in there.)

Our reason for the write occurring, incidentally, is that your eligibility for being in the test will change quickly and we don't want to unenroll people in that case -- otherwise we'd not need any storage and could just do it off of the user's ID.

The reason this stands out in production traffic is that it is placed in the onGetPreferences hook, which is run during nearly every possible web request for every kind of account, including page views and bot edits. If I understand the use case correctly, DT has to bucket basically everyone on the wiki. For that use case, we generally use a more fine-grained hook to avoid unleashing so much load at once, especially if its effect is not immediately visible during login (or does it change how pageviews look as well?).

Based on previous experiments of this kind, I believe it tended to be done in one of the following ways:

  1. Bucket the user during the sign up process, e.g. on account creation. This is a write action, and therefore naturally has access to the primary DB, and is also naturally more spread out in terms of traffic, and is also before DT is visible and thus can be fully async.
  2. Bucket during login, for existing users. Similarly ahead of time, and can be fully async.
  3. Bucket on-demand during a specific interaction, for existing users. During an early hook so that it is affecting the current page render as well, with the actual write re-applied from a deferred callback for persistence.
  4. Bucket during the sign up process for new accounts, and backfill from a one-off maintenance script for existing accounts.

The BetaFeatures auto-enroll feature unfortunately doesn't fall into any of these methods, and very un-narrowly uses the onGetPreferences hook indeed. However if I understand correctly, it has one key difference which is that it is only active for people that have opted in to the auto-enroll feature, which is itself an opt-in beta feature. Thus its database writes are far and few between both due to the relatively low number of people in the group, and due to how its writes are naturally spread out (with most of its writes happening during the act of enabling it, which is a POST request).

Using a deferred update would make for an easy and quick improvement. Depending on quickly the traffic drops off, and whether there are other wikis to be tested on, I would recommend option 3 or 4 in the future. Option 4 is likely the most convenient to use for you as it takes very little code, removes the DB risk entirely, and also means you never have to worry about whether you have covered all relevant interactions/hooks.

I've asked what Product's plans are for future A/B tests, and depending on the answer I'll either do a larger change to enrollment or add the DeferredUpdate calls.

Let me know if I can help on anything.

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team.
ppelberg subscribed.

I've asked what Product's plans are for future A/B tests, and depending on the answer I'll either do a larger change to enrollment or add the DeferredUpdate calls.

@DLynch I'm assigning this task over to you to "close the loop" considering we will NOT be deploying the A/B test to more wikis.

We disabled the A/B test a few weeks ago (T291873), but we're planning another one (T304030), so we should work on this before that.

It looks like for T304030, we want to include every logged-in user in the test, so we might be able to just remove the preference stuff entirely, and compute the bucket from the user ID every time.

It looks like for T304030, we want to include every logged-in user in the test, so we might be able to just remove the preference stuff entirely, and compute the bucket from the user ID every time.

Yeah, for once there's no requirements based on changing properties (edit count / has-not-used-DT), so we won't need to store anything.

Change 793359 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/DiscussionTools@master] Ready A/B test code for topic subscriptions

https://gerrit.wikimedia.org/r/793359

Change 793359 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Ready A/B test code for topic subscriptions

https://gerrit.wikimedia.org/r/793359

DLynch added a project: Skipped QA.

This is server-side warning-logging, so QA can't really check it easily.