Page MenuHomePhabricator
Create Task
Maniphest T281212

Restart x1 database master (db1103)
Closed, ResolvedPublic
Actions

Assigned To
• Marostegui
Authored By
• Marostegui
Apr 27 2021, 4:33 AM
Tags
Referenced Files
None
Subscribers
Aklapper
BEANS-X2
Johan
• Marostegui
RhinosF1
SD_hehua
sgrabarczuk
View All 9 Subscribers
Tokens
"Like" token, awarded by Legoktm.

Description

Due to T279281: Upgrade 10.4.13 hosts to a higher version we need to restart x1 primary database master.
x1 currently has:

  • Databases: flowdb, cognate_wiktionary and wikishared database,
  • Tables on the x1 wikis are:
+-------------------+
| Tables_in_enwiki  |
+-------------------+
| aft_feedback      |
| echo_email_batch  |
| echo_event        |
| echo_notification |
| echo_target_page  |
+-------------------+
bounce_records
cx_corpora
cx_lists
cx_suggestions
cx_translations
cx_translators
echo_unread_wikis
reading_list
reading_list_entry
reading_list_project
urlshortcodes
wikimedia_editor_tasks_counts
wikimedia_editor_tasks_edit_streak
wikimedia_editor_tasks_keys
wikimedia_editor_tasks_targets_passed

The last time we did an x1 maintenance ( T273758 T226358 and T250701) we used a bunch of tags, so I am tagging the same and subscribing the same people.

Impact: x1 will be on read-only for around 1 minute. Writes will not go through but reads will remain unaffected
When: Wednesday 5th May at 06:00 AM UTC

Related Objects
Search...

Event Timeline

• Marostegui created this task.Apr 27 2021, 4:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2021, 4:33 AM
• Marostegui triaged this task as Medium priority.Apr 27 2021, 4:33 AM
• Marostegui moved this task from Triage to Ready on the DBA board.
Legoktm awarded a token.Apr 27 2021, 4:52 AM
RhinosF1 subscribed.Apr 27 2021, 7:02 AM
Tgr added a project: GrowthExperiments.Apr 27 2021, 4:21 PM
Tgr subscribed.

GrowthExperiments also uses x1 (not for enwiki but for a number of others). As far as I'm aware it won't be nontrivially impacted by a short readonly period.

Johan subscribed.EditedApr 28 2021, 12:44 AM

Making sure I read the task correctly:

Only English Wikipedia and Growth features (trivially) will be affected.
Echo, CX and a few other things will not allow writes.

Does this mean that during this minute, edits will fail to cause notifications at English Wikipedia? That is, editors will never be notified for the things that happened during this restart?

• Marostegui added a comment.Apr 28 2021, 4:25 AM

@Johan all wikipedias will be affected, as x1 holds echo for all the projects.

Johan added a comment.Apr 28 2021, 10:27 AM

@Marostegui Thanks! And notifications for this minute will be lost, not just delayed?

• Marostegui added a comment.Apr 28 2021, 11:41 AM

I am not fully sure about that, as I don't know whether they get re-tried or not.

Trizek-WMF added subscribers: sgrabarczuk, Trizek-WMF.Apr 28 2021, 2:30 PM

IIRC, notifications work on a one-time attempt. I'll ask around.

This read-only is a big one, affecting multiple wikis and cross-wiki services. We should treat is as such. @Marostegui, do you plan to have a task to inform communities about it?

• Marostegui added a comment.Apr 28 2021, 2:33 PM

From what I can see on the last time we did this, it was just announce on tech news: https://phabricator.wikimedia.org/T273758#6832250

• Marostegui added a comment.Apr 28 2021, 2:33 PM

@Trizek-WMF if you want me to create a task for that I can do that too, whatever works best for you.

• Marostegui added a comment.Apr 28 2021, 2:46 PM

@Trizek-WMF this is what we listed as impact last time: T273762

Trizek-WMF mentioned this in T281375: Read only time for extension 1 (x1) primary database on 2021-05-05.EditedApr 28 2021, 2:46 PM

Last time, we setup a banner. :)

It is not a big deal to create this banner. I prefer to have our communities being informed, to prevent messages from unhappy people who haven't been warned about the read-only.

Tgr mentioned this in T281382: Make sure all GrowthExperiments DB writes handle readonly mode well.Apr 28 2021, 3:53 PM
Johan moved this task from To Triage to Announce in next Tech/News on the User-notice board.Apr 29 2021, 11:48 AM
Johan moved this task from Announce in next Tech/News to In current Tech/News draft on the User-notice board.Apr 30 2021, 3:45 PM
MSantos moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.May 3 2021, 3:04 PM
• Marostegui moved this task from Ready to In progress on the DBA board.May 4 2021, 5:43 AM
Stashbot added a comment.May 4 2021, 11:56 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T11:56:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1120 to upgrade its mysql T281212', diff saved to https://phabricator.wikimedia.org/P15710 and previous config saved to /var/cache/conftool/dbconfig/20210504-115634-marostegui.json

Stashbot added a comment.May 4 2021, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T11:58:42Z] <marostegui> Upgrade mysql and kernel on db1120 T281212

Stashbot added a comment.May 4 2021, 12:49 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:49:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 to upgrade its mysql T281212', diff saved to https://phabricator.wikimedia.org/P15717 and previous config saved to /var/cache/conftool/dbconfig/20210504-124937-marostegui.json

Stashbot added a comment.May 4 2021, 12:50 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T12:50:19Z] <marostegui> Upgrade mysql and kernel on db1137 T281212

• Marostegui added a comment.EditedMay 4 2021, 12:53 PM

All slaves have been upgraded to 10.4.18, so the master is ready for the operation tomorrow.

kostajh moved this task from Inbox to Triaged on the Growth-Team board.May 4 2021, 1:57 PM
• Marostegui added a comment.May 5 2021, 5:15 AM

All hosts silenced.
Master binary's upgraded, waiting now to perform the restart at 06:00 AM UTC

SD_hehua subscribed.May 5 2021, 5:54 AM
Stashbot added a comment.May 5 2021, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T06:00:03Z] <marostegui> Restart mysqld on x1 database primary master (db1103) T281212

• Marostegui closed this task as Resolved.May 5 2021, 6:02 AM

This was done.
RO starts: 06:00:15
RO stops: 06:00:46

Total RO time: 31 seconds

Trizek-WMF closed subtask T281375: Read only time for extension 1 (x1) primary database on 2021-05-05 as Resolved.May 5 2021, 10:18 AM
BEANS-X2 subscribed.May 6 2021, 7:28 AM
Johan moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.May 6 2021, 11:04 AM
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 13 2022, 2:15 PM
Maintenance_bot added a project: Wikidata.Aug 13 2022, 2:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits