Page MenuHomePhabricator

Watchlist Expiry: Release plan [rough schedule]
Closed, ResolvedPublic

Description

NOTE: Dates in this ticket have been updated to reflect the conversation in the comments below.

Hello, DBAs! We have decided to share our current plans for the release of watchlist expiry. This way, we can ensure that we have reached a common agreement and that we are properly monitoring potential risks. Here’s what we have in mind:

  • In September, we begin the process of enabling the Watchlist Expiry feature. We have chosen to first release to wikis in Groups 0 and 1. This way, we can collect feedback within smaller communities before releasing to larger wikis. Our plan is as follows:
    • Group 0 -- Week of Sept 21: We'll enable the feature on Officewiki and Mediawikiwiki.
    • Group 1 -- Week of Sept 28: We'll enable the feature on Metawiki, Enwikisource, Enwikivoyage, Enwikiversity, Eswiktionary, Eswikisource, Eswikivoyage, Hewiki, Hewikisource, Hewiktionary, and dewikisource.
  • After about two weeks of collecting feedback, we will begin the second stage of the release.
    • Group 2 -- October 13: We'll enable the feature on Dewiki, Frwiki, Fawiki, and Cswiki. These wikis were chosen due to the conclusion of our discussion in T252812#6311084 with @Marostegui, as well as further discussion in the comments below.
  • Once we have finished the secondary release, the DBA team will be able to begin analyzing the impact of Watchlist Expiry. It is our hope that the analysis can be completed within the first half of Q2 (roughly, by November 15th or so), so the Community Tech can release to all wikis by the end of Q2 (December 30th). This way, if issues should arise, we can try to address them before the end of the quarter.

How do you all feel about our release plan? Do you have any major reservations or concerns? Would you feel comfortable with us proceeding with this plan in September?

Thank you!

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ifried updated the task description. (Show Details)
ifried updated the task description. (Show Details)
Marostegui triaged this task as Medium priority.Aug 24 2020, 7:22 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Thanks @ifried for reaching out.
Overall, I feel good with that plan, especially the order and the set of wikis it will be developed too.
However, I would like to see if we can change the dates, especially the first one, 7th September.

We'll be doing our DC switchover the 1st of September (Tuesday) and while we are confident about it, we're going to spend the rest of the week monitoring and tackling things as they arise.
The DC switchover is a great opportunity for us to perform big and costly maintenances on eqiad (our, normally, active datancenter) as it becomes stand by. We are expecting to start that on Monday 7th.
Whilst we expect the switchover to go smooth, I would like to see if we can delay the enablement of the new watchlist feature by a week, so 14th instead of 7th, so we don't have more variables added to the mix if this big switchover.

Let me know if that works.

Regarding the second date, 15th November, I am not sure we can commit to that date as of today, let's keep it a bit flexible and we can see how everything goes within the next few weeks. How does that sound?
Thanks!

@Marostegui Thank you for your prompt response! We are very happy to read that you feel good about our plan. Yes, we can push the release schedule a week later in order to accommodate the DC switchover. In that case, we'll begin the release of the feature on September 14th instead. Finally, regarding the question of November 15th for an analysis deadline: Yes, let's stay in touch about this, and we can see how things go over the next couple of week, as you wrote. Thank you again for your flexibility and assistance during this process!

My final question is: What sort of communication would you like from us as we begin to release the feature, beginning September 14th? Should I ping you on this ticket after we have released to some wikis, just so you can stay in the loop? Overall, I'm just curious what next steps, if any, we should take. Thanks!

Let's use this task for communication, that works for me

Great, thanks! In that case, I'll post an update when we begin the process of deploying on September 14th. Thank you!

Aaaah, typo (the days sure do blend together lately)! I have corrected it to be September 14 :)

@ifried is this still scheduled for next week?

@Marostegui Yes, we're planning for a Tuesday, 9/15 release. Thanks for checking in! I'll update you on Tuesday and let you know if the release is still happening or if anything has changed.

@Marostegui Hello! I'm sending an update that we now plan to postpone the release to a week later, so we can address a UX issue. This means that Group 0 release will now happen the week of Sept 21st (most likely, on September 22nd), and the Group 1 release would happen one week afterward (most likely, beginning on September 29th).

Overall, how does this sound? Any concerns? Thanks in advance!

That sounds good @ifried - let me know when the final date is clear
Thank you for the heads up

Hello, @Marostegui! We are scheduled to begin the release process this week, as per the schedule. This means that we plan to enable Watchlist Expiry on Officewiki and MediawikiWiki on Tuesday, September 22nd. We'll let you know when it has been enabled, and progress can be tracked in T261249.

Also, a small update: We have received a request to include Czech wiki in our Group 2 release (scheduled for later) that will include Dewiki, Frwiki, and Fawiki. Would this be okay?

Thanks in advance!

ifried updated the task description. (Show Details)

Thanks for the heads up @ifried. Those wikis are ok by me!

ifried updated the task description. (Show Details)

@Marostegui Thank you for the approval of Czech Wikipedia! I'm sending an update that Watchlist Expiry is now up on Officewiki and Mediawiki.org, and we're still on schedule to release to the next batch of pilot wikis (Group 1) next week. I'll send an update when that next release occurs. Thanks again!

@Marostegui Watchlist Expiry has now been enabled on Metawiki, Enwikisource, Enwikivoyage, Enwikiversity, Eswiktionary, Eswikisource, Eswikivoyage, Hewiki, Hewikisource, Hewiktionary, and Dewikisource. We will collect feedback from users for about 2 weeks, and we will then begin our Group 2 release. Thanks!

Thank you @ifried - I will take a look at the graphs for the servers where those wikis live in case there're strange things.

ifried updated the task description. (Show Details)
ifried updated the task description. (Show Details)

@Marostegui Hello! We are planning to enable the feature on Persian, French, German, and Czech Wikipedia on Tuesday, October 13. You can follow the release in T264780. Once we have finished the release, I'll update this ticket again.

Thanks for the heads up @ifried - from those Wikipedias, the German and the French ones will be interesting to monitor, as those are quite big already.
Thank you!

@Marostegui Hello! We have enabled the feature on Persian, French, German, and Czech Wikipedia.

Excellent, thanks for the heads up.
I haven't noticed anything on s5 (dewiki) or s6 (frwiki) - so we are good there

@Marostegui Thank you for your help so far in the release process! Now that it has been a few weeks, we would like to proceed with enabling the feature on all wikis. Below, I have shared our proposal:

  • November 10: Enable the feature on Wikimedia Commons and Wikidata
  • Monitor for a week and confirm that no major issues arise
  • November 17: Enable the feature on all wikis

How does this plan sound? Do you have any major concerns? Overall, we would love to get this feature out to folks, so it is available to everyone, rather than just a small batch of wikis. Thanks in advance and we look forward to your response!

@Marostegui Thank you for your help so far in the release process! Now that it has been a few weeks, we would like to proceed with enabling the feature on all wikis. Below, I have shared our proposal:

  • November 10: Enable the feature on Wikimedia Commons and Wikidata
  • Monitor for a week and confirm that no major issues arise
  • November 17: Enable the feature on all wikis

How does this plan sound? Do you have any major concerns? Overall, we would love to get this feature out to folks, so it is available to everyone, rather than just a small batch of wikis. Thanks in advance and we look forward to your response!

We need to keep in mind that 11th, 12th and 13th will be days off, and thus there will be limited availability so enabling Commons and Wikidata can be a bit dangerous if something happens.
Can we do that a week after your initial proposal?
The rest of the plan sounds good.

@Marostegui Thanks for your response, and it's great news that we have general approval to now release to all wikis!

One note about the calendar dates: Yes, great point about staff being off on the 11th-13th. We have just discussed potential calendar dates as a team, and we realized that most weeks in November are problematic due to various holidays. For example, we have next Tuesday off (so it wouldn't make sense to do a release that week, since we try to do releases for the feature on Monday or Tuesday). The week after the Nov 16-20 week is a holiday week (Thanksgiving), so releases probably cannot happen that week (or at least not recommended). Then, of course, the following week brings us into December -- and, by that point, the team will be heavily focused on reviewing proposals from the 2021 Community Wishlist Survey. In December, we'll have much less capacity to properly coordinate and monitor a release, as we'll be busy with wishlist work until our Christmas holiday break.

So, here is a new idea: What do you think of enabling the feature on all wikis on November 17th? Once the feature is enabled on all wikis, we can closely communicate with you & monitor any potential impact. The feature has been enabled on a total of 17 wikis, with a few large ones (such as dewiki & frwiki) for the last few weeks, and things seem to be going smoothly. Let us know what you think. Thanks in advance & we look forward to your response!

@ifried can we do: Wikidata on 17th Nov and then the 24th Nov on all wikis?
I prefer to have the feature enabled on 1 big wiki first, before going full on.

If we run into issues, we can disable the feature, is that correct?

@Marostegui Thanks for the reply! Nov 24th is a holiday week, so how about we do the second release to all wikis on Tuesday, Dec 1st? This would mean that the schedule would be:

  • Nov 17: Enable the feature on Commons & Wikidata
  • Dec 1: Enable the feature on all wikis

How does that sound? Thanks!

That sounds good to me!
Thank you

@Marostegui Fantastic; thank you! We'll update you when the Nov 17 release is complete.

@Marostegui Watchlist Expiry has been enabled on Wikidata & Commons.

Thanks, this should be monitored for a few weeks in a similar way than we are doing for other tables (ie: T267275)

@Marostegui Hello! We are still planning to release to all wikis on December 1st. Is that still okay? Any major concerns or blockers? Thanks in advance!

@Marostegui Hello! We are still planning to release to all wikis on December 1st. Is that still okay? Any major concerns or blockers? Thanks in advance!

So far I haven't seen anything super worrying. I am closely monitoring the sizes of them at T268096 and while there was a 300MB increase on the table from one week to another, let's see if that stabilizes or if that keeps growing 300MB per week, which would be worrying.
I think you might have missed my question at T261005#6587447, if we run into issues, can this feature be disabled easily?

Thanks for the update, @Marostegui! Yes, if there are serious issues, we can easily disable the feature on wikis. Everything is behind a config flag that we can turn off at anytime.

@Marostegui No problem!

In that case, are we good to go for our release on 12/1? Or shall I check in the day before, on 11/30? You can follow our release via T266875. Thanks!

Let's go ahead for the 01/12/2020

@Marostegui Fantastic, thank you so much! We'll update you when the release is complete.

@Marostegui The feature has now been enabled on all wikis.

Thanks @ifried
For the record and for future reference when checking graphs/logs this is when it was enabled:

Mentioned in SAL (#wikimedia-operations) [2020-12-02T00:14:48Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c73f0bf0d1cdc1c7441261ffb1ad8ae12aa92ec9: Enable watchlist expiry feature on all wikis (T266875) (duration: 01m 07s)

I will keep an eye on the graphs to see if we notice some strange patterns. Let's leave this task open for a few more days

@Marostegui As the feature has been enabled on all wikis for a week now, shall we close this ticket? Thanks in advance!

Is there a task for enabling this in MediaWiki by default (and possibly even dropping the feature flag)?

Is there a task for enabling this in MediaWiki by default (and possibly even dropping the feature flag)?

There is now T270058 (:

@Marostegui As the feature has been enabled on all wikis for a week now, shall we close this ticket? Thanks in advance!

Let's close it.
We also have T268096 to keep track of the sizes

ifried claimed this task.

Fantastic, @Marostegui ! Thank you for your help & collaboration throughout the course of the release process. It's really appreciated :) I'll now mark this ticket as Resolved and we can keep track of the sizes in T268096.

Filed T270481, not yet UBN, but could become one as usage increases.

Thanks Jaime for filing that task. I have added it as a subtask for this one. I am going to re-close this one, so we can follow up on the specific one for the possible issue and discuss next steps there