Page MenuHomePhabricator

Reconsider GrowthExperiments feature flags
Closed, ResolvedPublic5 Estimated Story Points

Description

Because the Growth team tends to deploy features to pilot wikis only, we have a fairly large set of feature flags. Many of those are set to true at all wikis already (since the features scaled up sufficiently), but we still need to spend some time ensuring they are properly respected in the code, in case we ever decide to set the flag to false again. Most recently, this came up in T367330: GrowthExperiments: Do not expose configuration for unavailable features from Community Configuration, where we implemented a special case for "levelling up disabled", even though we do not have levelling up anywhere.

If we remove some of our feature flags, we will be able to simplify our code, making it easier to maintain in the future. For this reason, we should review that all of our feature flags are still needed, and consider removing them if possible. The full list of our feature flags is below:

Feature flag nameFeature flag descriptionKeep? (YES/NO)Notes
GEHelpPanelEnabledControls whether the Help panel is available for newcomers (whether they can enable it in the preferences, or whether A/B testing might enable it for them))No
WelcomeSurveyEnabledWhether the Welcome survey is served for newcomersNo
GEHomepageEnabledWhether the Homepage is available for newcomers (whether they can enable it in the preferences, or whether A/B testing might enable it for them)No
GEHomepageLoggingEnabledWhether telemetry data is collected for Homepage via EventLoggingNo
GEHomepageSuggestedEditsEnabledWhether the Suggested edits are a part of the Homepage (set to false for frwiktionary; true everywhere else)Yes*
GEPersonalizedPraiseBackendEnabled / GEPersonalizedPraiseNotificationsEnabledWhether Personalised praise is available for mentorsYES
GELevelingUpFeaturesEnabledControls whether Levelling up features are available for newcomersNo
GEMentorDashboardEnabledWhether the Mentor dashboard is available for mentorsNo
GEConfirmEmailEnabledWhether the "check your email" hint is added to Special:CreateAccountNo
GENewcomerTasksGuidanceEnabledWhether guidance is added to the Help Panel for Suggested editsNo
GENewcomerTasksImageRecommendationsEnabledWhether Add Image is available for newcomers (for whole articles)Yes*
GENewcomerTasksSectionImageRecommendationsEnabledWhether Add Image is available for newcomers (for sections)Yes*
GEStructuredTaskRejectionReasonTextInputEnabledWhether newcomers are allowed to add raw text when they reject a suggestion for other reasonsNo
GELinkRecommendationsFrontendEnabledWhether Add Link is enabledYes*
GETopicsMatchModeEnabledWhether there is the AND/OR topic matching mode in the HomepageYes*

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Urbanecm_WMF added a subscriber: KStoller-WMF.

I added in a table that covers what each of the feature flags does – hopefully that makes it easier for others to chime in whether each of the flags is needed or not. When making a decision, please note that we should probably only remove feature flags that we're fully certain we will never need again (in other words: we should only remove feature flags for features that became an essential part of Growth features at all wikis, including French Wiktionary). This is because potentially re-adding the feature flag again (after we remove it in this task) would be more costly than maintaining it along the way.

@KStoller-WMF I think this now needs you to take a look and fill the "Keep?" column with decisions. I excluded technical feature flags from the list, so hopefully all of the flags make sense to you, but if not, happy to clarify any of them either here or in our next 1:1.

Urbanecm_WMF added a project: Technical-Debt.
Urbanecm_WMF updated the task description. (Show Details)

Thanks for the explanation.
I'll review, check in with @nettrom_WMF, and update the Keep column by the end of the week.

I've added decisions in the task description.

Some of the Yes* decisions are based on the fact that we don't have these features scaled to all wikis that have Growth Features, so I assume we still need the feature flag. Please let me know if that's an incorrect assumption.

Some of the Yes* decisions are based on the fact that we don't have these features scaled to all wikis that have Growth Features, so I assume we still need the feature flag. Please let me know if that's an incorrect assumption.

Related to that the question: Is the plan to give new wikis (either newcly created wikipedias or maybe sister projects to which we might want to expand to) access to full feature suite out of the box when we enable GE for them? That would make sense to me, but I'm not sure if that decisions has been made.

Related to that the question: Is the plan to give new wikis (either newcly created wikipedias or maybe sister projects to which we might want to expand to) access to full feature suite out of the box when we enable GE for them? That would make sense to me, but I'm not sure if that decisions has been made.

No final decision, but my assumption is that new wikis should have access to the full feature suite with a few exceptions:

I hope to release further improvements to these features before enabling them more widely:

  • GETopicsMatchMode
  • GEPersonalizedPraiseBackendEnabled & GEPersonalizedPraiseNotifications

And I think we first need to check task availability and suggestion quality before enabling these tasks at new / very small wikis:

  • GENewcomerTasksImageRecommendations
  • GENewcomerTasksSectionImageRecommendations
  • GELinkRecommendationsFrontend

That makes sense to me, thank you! 🙏

KStoller-WMF set the point value for this task to 5.Sep 17 2024, 2:53 PM
Sgs updated the task description. (Show Details)

Looking through this, I notice that GEStructuredTaskRejectionReasonTextInputEnabled seems to be false everywhere!?! Is that actually a feature that is intended to be removed?

Looking through this, I notice that GEStructuredTaskRejectionReasonTextInputEnabled seems to be false everywhere!?! Is that actually a feature that is intended to be removed?

At this point, quite probably! This is a perfect example of the first Tim Starling's law :/. We originally added this in 2022 as part of T304099: Structured tasks: temporary free text for "other" rejection reason, to research what reasons we might be missing. Users enter surprising things, so we turned it off as fast as we could. Removing the feature seems like a good idea, unless @KStoller-WMF thinks otherwise. Let's have a separate task to make+implement that decision?

For reference, this is how it used to look like:

Task typeDesktopMobile
Add image
Screen Shot 2022-06-14 at 9.47.28 AM.png (864×1 px, 114 KB)
Screen Shot 2022-06-14 at 9.31.55 AM.png (1×752 px, 168 KB)
Add link
Screen Shot 2022-06-14 at 9.45.56 AM.png (764×1 px, 126 KB)
Screen Shot 2022-06-14 at 9.33.12 AM.png (1×752 px, 174 KB)

Looking through this, I notice that GEStructuredTaskRejectionReasonTextInputEnabled seems to be false everywhere!?! Is that actually a feature that is intended to be removed?

At this point, quite probably! This is a perfect example of the first Tim Starling's law :/. We originally added this in 2022 as part of T304099: Structured tasks: temporary free text for "other" rejection reason, to research what reasons we might be missing. Users enter surprising things, so we turned it off as fast as we could. Removing the feature seems like a good idea, unless @KStoller-WMF thinks otherwise. Let's have a separate task to make+implement that decision?

Interesting, thanks! With confirmation by @KStoller-WMF, I can create a task to not just remove that config but the entire associated custom-text-input feature.

Are the original inputs still available somewhere? Also, where can I see the tracking for what people choose as the rejection-reason?

so we turned it off as fast as we could

Though when looking at the timestamps in T304099, it seemed to have run until (T304099#8083516) its planned end-date (T304099#8041128)

Interesting, thanks! With confirmation by @KStoller-WMF, I can create a task to not just remove that config but the entire associated custom-text-input feature.

Sounds good!

Are the original inputs still available somewhere?

Not in full. Analysis is available here. Parts of the data (w/o the identifying parts) are available in this spreadsheet too. Full dataset cannot be available at this point, because it is covered by the standard 90 days retention period (after that, we can only keep non-identifying datasets). Hopefully both documents are visible to you, but let me know if that is not the case.

Also, where can I see the tracking for what people choose as the rejection-reason?

The data is flowing to the Data Lake, available as mediawiki_structured_task_article_link_suggestion_interaction. Rejection reasons are logged under action_data, see schema for details. That said, you wouldn't find any plaintext under "other" at this point, because we stopped collecting that data (you'd only see the percentage of "other" responses, but no details). Hope this helps!

Interesting, thanks! With confirmation by @KStoller-WMF, I can create a task to not just remove that config but the entire associated custom-text-input feature.

Yes, please! Let's simplify and remove that config as it was meant to be temporary and I don't think we want to ever enable it again.

It has now several dedicated sub-tasks, so let's call it an Epic.

Michael claimed this task.

As far as I can tell, this is done. Let's close it.