Page MenuHomePhabricator
Create Task
Maniphest T179417

Migrate Parsoid from legacy to new storage
Closed, ResolvedPublic
Actions

Description

Migrate Parsoid (incl. stashing) from legacy storage (Cassandra 2.x), to the new strategy and cluster (Cassandra 3.x). This is planned to happen in two steps, first all non-Wikipedia groups, then the Wikipedias. Between these two steps, cluster reshaping will occur, with capacity decommissioned from the legacy cluster, and bootstrapped into the new.

Details

ProjectBranchLines +/-Subject
mediawiki/services/restbase/deploymaster+0 -9[Config] Remove new_storage_enabled_parsoid config stanza
mediawiki/services/restbase/deploymaster+1 -10[Config] Enable new storage for Parsoid for all domains
mediawiki/services/mobileappsmaster+2 -2Bump mobile-sections and definitions version
mediawiki/services/restbase/deploymaster+1 -0Config: Add wikidata.org to the next-gen Parsoid storage
mediawiki/services/restbase/deploymaster+9 -4Config: Parsoid: Switch all but WPs to the next-gen storage
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans created this task.Oct 31 2017, 4:52 PM
Pchelolo added a subscriber: Pchelolo.EditedOct 31 2017, 4:53 PM

We've decided to migrate stashing together with the rest of parsoid tables.

Eevans renamed this task from Migrate Parsoid stashing from legacy to new storage to Migrate Parsoid from legacy to new storage.Oct 31 2017, 4:55 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Oct 31 2017, 4:58 PM
Stashbot added a subscriber: Stashbot.Nov 2 2017, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-02T11:21:05Z] <mobrovac@tin> Started deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417

Stashbot added a comment.Nov 2 2017, 11:29 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-02T11:29:21Z] <mobrovac@tin> Finished deploy [restbase/deploy@f6c4e2d]: Parsoid module: use the Cassandra 2 tables as fallback when needed - T179417 (duration: 08m 15s)

gerritbot added a subscriber: gerritbot.Nov 2 2017, 11:55 AM

Change 388036 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Config: Parsoid: Switch all but WPs to the next-gen storage

https://gerrit.wikimedia.org/r/388036

gerritbot added a project: Patch-For-Review.Nov 2 2017, 11:55 AM

Change 388036 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Config: Parsoid: Switch all but WPs to the next-gen storage

https://gerrit.wikimedia.org/r/388036

Stashbot added a comment.Nov 2 2017, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-02T11:57:57Z] <mobrovac@tin> Started deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417

Stashbot added a comment.Nov 2 2017, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-02T12:06:39Z] <mobrovac@tin> Finished deploy [restbase/deploy@9314cf6]: Parsoid: Switch all but WPs to use the next-generation storage - T179417 (duration: 08m 43s)

mobrovac added a subscriber: mobrovac.Nov 2 2017, 12:27 PM

All but WPs (and the global domain, technically) have been switched to use the new storage schema with Cassandra 3. We need to keep the old contents around for the next 24h, though, before we can get rid of the data in Cassandra 2. data tables in the following keyspaces can be truncated (not dropped!) after that period elapses:

"local_group_default_T_parsoid_dataLpBGD5XFAMFsTr"
"local_group_default_T_parsoid_htmliZ1mueNVmW9MlJ"
"local_group_default_T_parsoid_section_okYE_jMlE6"
"local_group_default_T_parsoid_stash_datU81yvllO3"
"local_group_default_T_parsoid_stash_datWH8IDUS9S"
"local_group_default_T_parsoid_stash_html"
"local_group_default_T_parsoid_stash_htmmXxc_uDhg"
"local_group_default_T_parsoid_stash_sec2ACMDK1DR"
"local_group_default_T_parsoid_stash_wikf0PBY8UXq"
"local_group_default_T_parsoid_stash_wikitext"
"local_group_default_T_parsoid_wikitext"
"local_group_phase0_T_parsoid_dataLpBGD5XFAMFsTr8"
"local_group_phase0_T_parsoid_htmliZ1mueNVmW9MlJq"
"local_group_phase0_T_parsoid_section_ofkYE_jMlE6"
"local_group_phase0_T_parsoid_stash_dataU81yvllO3"
"local_group_phase0_T_parsoid_stash_dataWH8IDUS9S"
"local_group_phase0_T_parsoid_stash_html"
"local_group_phase0_T_parsoid_stash_htmlmXxc_uDhg"
"local_group_phase0_T_parsoid_stash_sect2ACMDK1DR"
"local_group_phase0_T_parsoid_stash_wikif0PBY8UXq"
"local_group_phase0_T_parsoid_stash_wikitext"
"local_group_phase0_T_parsoid_wikitext"
"local_group_wikibooks_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikibooks_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikibooks_T_parsoid_section_kYE_jMlE"
"local_group_wikibooks_T_parsoid_stash_daU81yvllO"
"local_group_wikibooks_T_parsoid_stash_daWH8IDUS9"
"local_group_wikibooks_T_parsoid_stash_html"
"local_group_wikibooks_T_parsoid_stash_htmXxc_uDh"
"local_group_wikibooks_T_parsoid_stash_se2ACMDK1D"
"local_group_wikibooks_T_parsoid_stash_wif0PBY8UX"
"local_group_wikibooks_T_parsoid_stash_wikitext"
"local_group_wikibooks_T_parsoid_wikitext"
"local_group_wikimedia_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikimedia_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikimedia_T_parsoid_section_kYE_jMlE"
"local_group_wikimedia_T_parsoid_stash_daU81yvllO"
"local_group_wikimedia_T_parsoid_stash_daWH8IDUS9"
"local_group_wikimedia_T_parsoid_stash_html"
"local_group_wikimedia_T_parsoid_stash_htmXxc_uDh"
"local_group_wikimedia_T_parsoid_stash_se2ACMDK1D"
"local_group_wikimedia_T_parsoid_stash_wif0PBY8UX"
"local_group_wikimedia_T_parsoid_stash_wikitext"
"local_group_wikimedia_T_parsoid_wikitext"
"local_group_wikinews_T_parsoid_dataLpBGD5XFAMFsT"
"local_group_wikinews_T_parsoid_htmliZ1mueNVmW9Ml"
"local_group_wikinews_T_parsoid_section_kYE_jMlE6"
"local_group_wikinews_T_parsoid_stash_daU81yvllO3"
"local_group_wikinews_T_parsoid_stash_daWH8IDUS9S"
"local_group_wikinews_T_parsoid_stash_html"
"local_group_wikinews_T_parsoid_stash_htmXxc_uDhg"
"local_group_wikinews_T_parsoid_stash_se2ACMDK1DR"
"local_group_wikinews_T_parsoid_stash_wif0PBY8UXq"
"local_group_wikinews_T_parsoid_stash_wikitext"
"local_group_wikinews_T_parsoid_wikitext"
"local_group_wikiquote_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikiquote_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikiquote_T_parsoid_section_kYE_jMlE"
"local_group_wikiquote_T_parsoid_stash_daU81yvllO"
"local_group_wikiquote_T_parsoid_stash_daWH8IDUS9"
"local_group_wikiquote_T_parsoid_stash_html"
"local_group_wikiquote_T_parsoid_stash_htmXxc_uDh"
"local_group_wikiquote_T_parsoid_stash_se2ACMDK1D"
"local_group_wikiquote_T_parsoid_stash_wif0PBY8UX"
"local_group_wikiquote_T_parsoid_stash_wikitext"
"local_group_wikiquote_T_parsoid_wikitext"
"local_group_wikisource_T_parsoid_dataLpBGD5XFAMF"
"local_group_wikisource_T_parsoid_htmliZ1mueNVmW9"
"local_group_wikisource_T_parsoid_sectionkYE_jMlE"
"local_group_wikisource_T_parsoid_stash_dU81yvllO"
"local_group_wikisource_T_parsoid_stash_dWH8IDUS9"
"local_group_wikisource_T_parsoid_stash_hmXxc_uDh"
"local_group_wikisource_T_parsoid_stash_html"
"local_group_wikisource_T_parsoid_stash_s2ACMDK1D"
"local_group_wikisource_T_parsoid_stash_wf0PBY8UX"
"local_group_wikisource_T_parsoid_stash_wikitext"
"local_group_wikisource_T_parsoid_wikitext"
"local_group_wikiversity_T_parsoid_dataLpBGD5XFAM"
"local_group_wikiversity_T_parsoid_htmliZ1mueNVmW"
"local_group_wikiversity_T_parsoid_sectiokYE_jMlE"
"local_group_wikiversity_T_parsoid_stash_2ACMDK1D"
"local_group_wikiversity_T_parsoid_stash_f0PBY8UX"
"local_group_wikiversity_T_parsoid_stash_html"
"local_group_wikiversity_T_parsoid_stash_mXxc_uDh"
"local_group_wikiversity_T_parsoid_stash_U81yvllO"
"local_group_wikiversity_T_parsoid_stash_WH8IDUS9"
"local_group_wikiversity_T_parsoid_stash_wikitext"
"local_group_wikiversity_T_parsoid_wikitext"
"local_group_wikivoyage_T_parsoid_dataLpBGD5XFAMF"
"local_group_wikivoyage_T_parsoid_htmliZ1mueNVmW9"
"local_group_wikivoyage_T_parsoid_sectionkYE_jMlE"
"local_group_wikivoyage_T_parsoid_stash_dU81yvllO"
"local_group_wikivoyage_T_parsoid_stash_dWH8IDUS9"
"local_group_wikivoyage_T_parsoid_stash_hmXxc_uDh"
"local_group_wikivoyage_T_parsoid_stash_html"
"local_group_wikivoyage_T_parsoid_stash_s2ACMDK1D"
"local_group_wikivoyage_T_parsoid_stash_wf0PBY8UX"
"local_group_wikivoyage_T_parsoid_stash_wikitext"
"local_group_wikivoyage_T_parsoid_wikitext"
"local_group_wiktionary_T_parsoid_dataLpBGD5XFAMF"
"local_group_wiktionary_T_parsoid_htmliZ1mueNVmW9"
"local_group_wiktionary_T_parsoid_sectionkYE_jMlE"
"local_group_wiktionary_T_parsoid_stash_dU81yvllO"
"local_group_wiktionary_T_parsoid_stash_dWH8IDUS9"
"local_group_wiktionary_T_parsoid_stash_hmXxc_uDh"
"local_group_wiktionary_T_parsoid_stash_html"
"local_group_wiktionary_T_parsoid_stash_s2ACMDK1D"
"local_group_wiktionary_T_parsoid_stash_wf0PBY8UX"
"local_group_wiktionary_T_parsoid_stash_wikitext"
"local_group_wiktionary_T_parsoid_wikitext"
Eevans added a comment.Nov 6 2017, 10:02 PM
In T179417#3729287, @mobrovac wrote:

All but WPs (and the global domain, technically) have been switched to use the new storage schema with Cassandra 3. We need to keep the old contents around for the next 24h, though, before we can get rid of the data in Cassandra 2. data tables in the following keyspaces can be truncated (not dropped!) after that period elapses:

"local_group_default_T_parsoid_dataLpBGD5XFAMFsTr"
"local_group_default_T_parsoid_htmliZ1mueNVmW9MlJ"
"local_group_default_T_parsoid_section_okYE_jMlE6"
"local_group_default_T_parsoid_stash_datU81yvllO3"
"local_group_default_T_parsoid_stash_datWH8IDUS9S"
"local_group_default_T_parsoid_stash_html"
"local_group_default_T_parsoid_stash_htmmXxc_uDhg"
"local_group_default_T_parsoid_stash_sec2ACMDK1DR"
"local_group_default_T_parsoid_stash_wikf0PBY8UXq"
"local_group_default_T_parsoid_stash_wikitext"
"local_group_default_T_parsoid_wikitext"
"local_group_phase0_T_parsoid_dataLpBGD5XFAMFsTr8"
"local_group_phase0_T_parsoid_htmliZ1mueNVmW9MlJq"
"local_group_phase0_T_parsoid_section_ofkYE_jMlE6"
"local_group_phase0_T_parsoid_stash_dataU81yvllO3"
"local_group_phase0_T_parsoid_stash_dataWH8IDUS9S"
"local_group_phase0_T_parsoid_stash_html"
"local_group_phase0_T_parsoid_stash_htmlmXxc_uDhg"
"local_group_phase0_T_parsoid_stash_sect2ACMDK1DR"
"local_group_phase0_T_parsoid_stash_wikif0PBY8UXq"
"local_group_phase0_T_parsoid_stash_wikitext"
"local_group_phase0_T_parsoid_wikitext"
"local_group_wikibooks_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikibooks_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikibooks_T_parsoid_section_kYE_jMlE"
"local_group_wikibooks_T_parsoid_stash_daU81yvllO"
"local_group_wikibooks_T_parsoid_stash_daWH8IDUS9"
"local_group_wikibooks_T_parsoid_stash_html"
"local_group_wikibooks_T_parsoid_stash_htmXxc_uDh"
"local_group_wikibooks_T_parsoid_stash_se2ACMDK1D"
"local_group_wikibooks_T_parsoid_stash_wif0PBY8UX"
"local_group_wikibooks_T_parsoid_stash_wikitext"
"local_group_wikibooks_T_parsoid_wikitext"
"local_group_wikimedia_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikimedia_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikimedia_T_parsoid_section_kYE_jMlE"
"local_group_wikimedia_T_parsoid_stash_daU81yvllO"
"local_group_wikimedia_T_parsoid_stash_daWH8IDUS9"
"local_group_wikimedia_T_parsoid_stash_html"
"local_group_wikimedia_T_parsoid_stash_htmXxc_uDh"
"local_group_wikimedia_T_parsoid_stash_se2ACMDK1D"
"local_group_wikimedia_T_parsoid_stash_wif0PBY8UX"
"local_group_wikimedia_T_parsoid_stash_wikitext"
"local_group_wikimedia_T_parsoid_wikitext"
"local_group_wikinews_T_parsoid_dataLpBGD5XFAMFsT"
"local_group_wikinews_T_parsoid_htmliZ1mueNVmW9Ml"
"local_group_wikinews_T_parsoid_section_kYE_jMlE6"
"local_group_wikinews_T_parsoid_stash_daU81yvllO3"
"local_group_wikinews_T_parsoid_stash_daWH8IDUS9S"
"local_group_wikinews_T_parsoid_stash_html"
"local_group_wikinews_T_parsoid_stash_htmXxc_uDhg"
"local_group_wikinews_T_parsoid_stash_se2ACMDK1DR"
"local_group_wikinews_T_parsoid_stash_wif0PBY8UXq"
"local_group_wikinews_T_parsoid_stash_wikitext"
"local_group_wikinews_T_parsoid_wikitext"
"local_group_wikiquote_T_parsoid_dataLpBGD5XFAMFs"
"local_group_wikiquote_T_parsoid_htmliZ1mueNVmW9M"
"local_group_wikiquote_T_parsoid_section_kYE_jMlE"
"local_group_wikiquote_T_parsoid_stash_daU81yvllO"
"local_group_wikiquote_T_parsoid_stash_daWH8IDUS9"
"local_group_wikiquote_T_parsoid_stash_html"
"local_group_wikiquote_T_parsoid_stash_htmXxc_uDh"
"local_group_wikiquote_T_parsoid_stash_se2ACMDK1D"
"local_group_wikiquote_T_parsoid_stash_wif0PBY8UX"
"local_group_wikiquote_T_parsoid_stash_wikitext"
"local_group_wikiquote_T_parsoid_wikitext"
"local_group_wikisource_T_parsoid_dataLpBGD5XFAMF"
"local_group_wikisource_T_parsoid_htmliZ1mueNVmW9"
"local_group_wikisource_T_parsoid_sectionkYE_jMlE"
"local_group_wikisource_T_parsoid_stash_dU81yvllO"
"local_group_wikisource_T_parsoid_stash_dWH8IDUS9"
"local_group_wikisource_T_parsoid_stash_hmXxc_uDh"
"local_group_wikisource_T_parsoid_stash_html"
"local_group_wikisource_T_parsoid_stash_s2ACMDK1D"
"local_group_wikisource_T_parsoid_stash_wf0PBY8UX"
"local_group_wikisource_T_parsoid_stash_wikitext"
"local_group_wikisource_T_parsoid_wikitext"
"local_group_wikiversity_T_parsoid_dataLpBGD5XFAM"
"local_group_wikiversity_T_parsoid_htmliZ1mueNVmW"
"local_group_wikiversity_T_parsoid_sectiokYE_jMlE"
"local_group_wikiversity_T_parsoid_stash_2ACMDK1D"
"local_group_wikiversity_T_parsoid_stash_f0PBY8UX"
"local_group_wikiversity_T_parsoid_stash_html"
"local_group_wikiversity_T_parsoid_stash_mXxc_uDh"
"local_group_wikiversity_T_parsoid_stash_U81yvllO"
"local_group_wikiversity_T_parsoid_stash_WH8IDUS9"
"local_group_wikiversity_T_parsoid_stash_wikitext"
"local_group_wikiversity_T_parsoid_wikitext"
"local_group_wikivoyage_T_parsoid_dataLpBGD5XFAMF"
"local_group_wikivoyage_T_parsoid_htmliZ1mueNVmW9"
"local_group_wikivoyage_T_parsoid_sectionkYE_jMlE"
"local_group_wikivoyage_T_parsoid_stash_dU81yvllO"
"local_group_wikivoyage_T_parsoid_stash_dWH8IDUS9"
"local_group_wikivoyage_T_parsoid_stash_hmXxc_uDh"
"local_group_wikivoyage_T_parsoid_stash_html"
"local_group_wikivoyage_T_parsoid_stash_s2ACMDK1D"
"local_group_wikivoyage_T_parsoid_stash_wf0PBY8UX"
"local_group_wikivoyage_T_parsoid_stash_wikitext"
"local_group_wikivoyage_T_parsoid_wikitext"
"local_group_wiktionary_T_parsoid_dataLpBGD5XFAMF"
"local_group_wiktionary_T_parsoid_htmliZ1mueNVmW9"
"local_group_wiktionary_T_parsoid_sectionkYE_jMlE"
"local_group_wiktionary_T_parsoid_stash_dU81yvllO"
"local_group_wiktionary_T_parsoid_stash_dWH8IDUS9"
"local_group_wiktionary_T_parsoid_stash_hmXxc_uDh"
"local_group_wiktionary_T_parsoid_stash_html"
"local_group_wiktionary_T_parsoid_stash_s2ACMDK1D"
"local_group_wiktionary_T_parsoid_stash_wf0PBY8UX"
"local_group_wiktionary_T_parsoid_stash_wikitext"
"local_group_wiktionary_T_parsoid_wikitext"

I've sampled a couple of hosts (one in eqiad and one in codfw), and see no write activity on the data tables of these keyspaces; I propose the following to truncate them:

P6271 truncate_non-wp_parsoid.cql
1TRUNCATE "local_group_default_T_parsoid_dataLpBGD5XFAMFsTr".data;
2TRUNCATE "local_group_default_T_parsoid_htmliZ1mueNVmW9MlJ".data;
3TRUNCATE "local_group_default_T_parsoid_section_okYE_jMlE6".data;
4TRUNCATE "local_group_default_T_parsoid_stash_datU81yvllO3".data;
5TRUNCATE "local_group_default_T_parsoid_stash_datWH8IDUS9S".data;
6TRUNCATE "local_group_default_T_parsoid_stash_html".data;
7TRUNCATE "local_group_default_T_parsoid_stash_htmmXxc_uDhg".data;
8TRUNCATE "local_group_default_T_parsoid_stash_sec2ACMDK1DR".data;
9TRUNCATE "local_group_default_T_parsoid_stash_wikf0PBY8UXq".data;
10TRUNCATE "local_group_default_T_parsoid_stash_wikitext".data;
11TRUNCATE "local_group_default_T_parsoid_wikitext".data;
12TRUNCATE "local_group_phase0_T_parsoid_dataLpBGD5XFAMFsTr8".data;
13TRUNCATE "local_group_phase0_T_parsoid_htmliZ1mueNVmW9MlJq".data;
14TRUNCATE "local_group_phase0_T_parsoid_section_ofkYE_jMlE6".data;
15TRUNCATE "local_group_phase0_T_parsoid_stash_dataU81yvllO3".data;
16TRUNCATE "local_group_phase0_T_parsoid_stash_dataWH8IDUS9S".data;
17TRUNCATE "local_group_phase0_T_parsoid_stash_html".data;
18TRUNCATE "local_group_phase0_T_parsoid_stash_htmlmXxc_uDhg".data;
19TRUNCATE "local_group_phase0_T_parsoid_stash_sect2ACMDK1DR".data;
20TRUNCATE "local_group_phase0_T_parsoid_stash_wikif0PBY8UXq".data;
21TRUNCATE "local_group_phase0_T_parsoid_stash_wikitext".data;
22TRUNCATE "local_group_phase0_T_parsoid_wikitext".data;
23TRUNCATE "local_group_wikibooks_T_parsoid_dataLpBGD5XFAMFs".data;
24TRUNCATE "local_group_wikibooks_T_parsoid_htmliZ1mueNVmW9M".data;
25TRUNCATE "local_group_wikibooks_T_parsoid_section_kYE_jMlE".data;
26TRUNCATE "local_group_wikibooks_T_parsoid_stash_daU81yvllO".data;
27TRUNCATE "local_group_wikibooks_T_parsoid_stash_daWH8IDUS9".data;
28TRUNCATE "local_group_wikibooks_T_parsoid_stash_html".data;
29TRUNCATE "local_group_wikibooks_T_parsoid_stash_htmXxc_uDh".data;
30TRUNCATE "local_group_wikibooks_T_parsoid_stash_se2ACMDK1D".data;
31TRUNCATE "local_group_wikibooks_T_parsoid_stash_wif0PBY8UX".data;
32TRUNCATE "local_group_wikibooks_T_parsoid_stash_wikitext".data;
33TRUNCATE "local_group_wikibooks_T_parsoid_wikitext".data;
34TRUNCATE "local_group_wikimedia_T_parsoid_dataLpBGD5XFAMFs".data;
35TRUNCATE "local_group_wikimedia_T_parsoid_htmliZ1mueNVmW9M".data;
36TRUNCATE "local_group_wikimedia_T_parsoid_section_kYE_jMlE".data;
37TRUNCATE "local_group_wikimedia_T_parsoid_stash_daU81yvllO".data;
38TRUNCATE "local_group_wikimedia_T_parsoid_stash_daWH8IDUS9".data;
39TRUNCATE "local_group_wikimedia_T_parsoid_stash_html".data;
40TRUNCATE "local_group_wikimedia_T_parsoid_stash_htmXxc_uDh".data;
41TRUNCATE "local_group_wikimedia_T_parsoid_stash_se2ACMDK1D".data;
42TRUNCATE "local_group_wikimedia_T_parsoid_stash_wif0PBY8UX".data;
43TRUNCATE "local_group_wikimedia_T_parsoid_stash_wikitext".data;
44TRUNCATE "local_group_wikimedia_T_parsoid_wikitext".data;
45TRUNCATE "local_group_wikinews_T_parsoid_dataLpBGD5XFAMFsT".data;
46TRUNCATE "local_group_wikinews_T_parsoid_htmliZ1mueNVmW9Ml".data;
47TRUNCATE "local_group_wikinews_T_parsoid_section_kYE_jMlE6".data;
48TRUNCATE "local_group_wikinews_T_parsoid_stash_daU81yvllO3".data;
49TRUNCATE "local_group_wikinews_T_parsoid_stash_daWH8IDUS9S".data;
50TRUNCATE "local_group_wikinews_T_parsoid_stash_html".data;
51TRUNCATE "local_group_wikinews_T_parsoid_stash_htmXxc_uDhg".data;
52TRUNCATE "local_group_wikinews_T_parsoid_stash_se2ACMDK1DR".data;
53TRUNCATE "local_group_wikinews_T_parsoid_stash_wif0PBY8UXq".data;
54TRUNCATE "local_group_wikinews_T_parsoid_stash_wikitext".data;
55TRUNCATE "local_group_wikinews_T_parsoid_wikitext".data;
56TRUNCATE "local_group_wikiquote_T_parsoid_dataLpBGD5XFAMFs".data;
57TRUNCATE "local_group_wikiquote_T_parsoid_htmliZ1mueNVmW9M".data;
58TRUNCATE "local_group_wikiquote_T_parsoid_section_kYE_jMlE".data;
59TRUNCATE "local_group_wikiquote_T_parsoid_stash_daU81yvllO".data;
60TRUNCATE "local_group_wikiquote_T_parsoid_stash_daWH8IDUS9".data;
61TRUNCATE "local_group_wikiquote_T_parsoid_stash_html".data;
62TRUNCATE "local_group_wikiquote_T_parsoid_stash_htmXxc_uDh".data;
63TRUNCATE "local_group_wikiquote_T_parsoid_stash_se2ACMDK1D".data;
64TRUNCATE "local_group_wikiquote_T_parsoid_stash_wif0PBY8UX".data;
65TRUNCATE "local_group_wikiquote_T_parsoid_stash_wikitext".data;
66TRUNCATE "local_group_wikiquote_T_parsoid_wikitext".data;
67TRUNCATE "local_group_wikisource_T_parsoid_dataLpBGD5XFAMF".data;
68TRUNCATE "local_group_wikisource_T_parsoid_htmliZ1mueNVmW9".data;
69TRUNCATE "local_group_wikisource_T_parsoid_sectionkYE_jMlE".data;
70TRUNCATE "local_group_wikisource_T_parsoid_stash_dU81yvllO".data;
71TRUNCATE "local_group_wikisource_T_parsoid_stash_dWH8IDUS9".data;
72TRUNCATE "local_group_wikisource_T_parsoid_stash_hmXxc_uDh".data;
73TRUNCATE "local_group_wikisource_T_parsoid_stash_html".data;
74TRUNCATE "local_group_wikisource_T_parsoid_stash_s2ACMDK1D".data;
75TRUNCATE "local_group_wikisource_T_parsoid_stash_wf0PBY8UX".data;
76TRUNCATE "local_group_wikisource_T_parsoid_stash_wikitext".data;
77TRUNCATE "local_group_wikisource_T_parsoid_wikitext".data;
78TRUNCATE "local_group_wikiversity_T_parsoid_dataLpBGD5XFAM".data;
79TRUNCATE "local_group_wikiversity_T_parsoid_htmliZ1mueNVmW".data;
80TRUNCATE "local_group_wikiversity_T_parsoid_sectiokYE_jMlE".data;
81TRUNCATE "local_group_wikiversity_T_parsoid_stash_2ACMDK1D".data;
82TRUNCATE "local_group_wikiversity_T_parsoid_stash_f0PBY8UX".data;
83TRUNCATE "local_group_wikiversity_T_parsoid_stash_html".data;
84TRUNCATE "local_group_wikiversity_T_parsoid_stash_mXxc_uDh".data;
85TRUNCATE "local_group_wikiversity_T_parsoid_stash_U81yvllO".data;
86TRUNCATE "local_group_wikiversity_T_parsoid_stash_WH8IDUS9".data;
87TRUNCATE "local_group_wikiversity_T_parsoid_stash_wikitext".data;
88TRUNCATE "local_group_wikiversity_T_parsoid_wikitext".data;
89TRUNCATE "local_group_wikivoyage_T_parsoid_dataLpBGD5XFAMF".data;
90TRUNCATE "local_group_wikivoyage_T_parsoid_htmliZ1mueNVmW9".data;
91TRUNCATE "local_group_wikivoyage_T_parsoid_sectionkYE_jMlE".data;
92TRUNCATE "local_group_wikivoyage_T_parsoid_stash_dU81yvllO".data;
93TRUNCATE "local_group_wikivoyage_T_parsoid_stash_dWH8IDUS9".data;
94TRUNCATE "local_group_wikivoyage_T_parsoid_stash_hmXxc_uDh".data;
95TRUNCATE "local_group_wikivoyage_T_parsoid_stash_html".data;
96TRUNCATE "local_group_wikivoyage_T_parsoid_stash_s2ACMDK1D".data;
97TRUNCATE "local_group_wikivoyage_T_parsoid_stash_wf0PBY8UX".data;
98TRUNCATE "local_group_wikivoyage_T_parsoid_stash_wikitext".data;
99TRUNCATE "local_group_wikivoyage_T_parsoid_wikitext".data;
100TRUNCATE "local_group_wiktionary_T_parsoid_dataLpBGD5XFAMF".data;
101TRUNCATE "local_group_wiktionary_T_parsoid_htmliZ1mueNVmW9".data;
102TRUNCATE "local_group_wiktionary_T_parsoid_sectionkYE_jMlE".data;
103TRUNCATE "local_group_wiktionary_T_parsoid_stash_dU81yvllO".data;
104TRUNCATE "local_group_wiktionary_T_parsoid_stash_dWH8IDUS9".data;
105TRUNCATE "local_group_wiktionary_T_parsoid_stash_hmXxc_uDh".data;
106TRUNCATE "local_group_wiktionary_T_parsoid_stash_html".data;
107TRUNCATE "local_group_wiktionary_T_parsoid_stash_s2ACMDK1D".data;
108TRUNCATE "local_group_wiktionary_T_parsoid_stash_wf0PBY8UX".data;
109TRUNCATE "local_group_wiktionary_T_parsoid_stash_wikitext".data;
110TRUNCATE "local_group_wiktionary_T_parsoid_wikitext".data;

Once complete, snapshots will need to cleared to see the space actually freed.

Stashbot added a comment.Nov 7 2017, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T11:43:26Z] <mobrovac> restbase truncating cassandra 2 non-WP tables for T179417

gerritbot added a comment.Nov 7 2017, 11:47 AM

Change 389699 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Config: Add wikidata.org to the next-gen Parsoid storage

https://gerrit.wikimedia.org/r/389699

gerritbot added a comment.Nov 7 2017, 11:49 AM

Change 389699 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Config: Add wikidata.org to the next-gen Parsoid storage

https://gerrit.wikimedia.org/r/389699

Stashbot added a comment.Nov 7 2017, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T11:52:02Z] <mobrovac@tin> Started deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417

Stashbot added a comment.Nov 7 2017, 12:00 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T12:00:16Z] <mobrovac@tin> Finished deploy [restbase/deploy@eab2948]: Use the new storage for wikidata.org - T179417 (duration: 08m 14s)

mobrovac removed a project: Patch-For-Review.Nov 7 2017, 12:04 PM

All but the default group tables have been truncated. default still had activity due to the wikidata.org domain which had not been switched. We switched that one too, so we should be able to truncate that group tomorrow as well.

Stashbot added a comment.Nov 8 2017, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-08T17:43:53Z] <mobrovac> restbase truncate the default parsoid storage group's tables for T179417

mobrovac added a comment.Nov 8 2017, 5:45 PM

All of the non-WP tables have been truncated now, snapshots can be cleared.

Stashbot added a comment.Nov 8 2017, 5:51 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-08T17:51:48Z] <urandom> Clearing snapshots in RESTBase legacy Cassandra cluster (T179417)

Eevans added a comment.Nov 8 2017, 5:52 PM
In T179417#3745181, @mobrovac wrote:

All of the non-WP tables have been truncated now, snapshots can be cleared.

Snapshots cleared.

mobrovac added a project: Parsoid.EditedDec 8 2017, 8:35 AM
mobrovac added a subscriber: ssastry.

We are now ready to switch the WP Parsoid tables to Cassandra 3. I propose to do a coordinated deploy of Parsoid (bumping the HTML content version to 1.6.0) and RESTBase (moving to Cassandra 3) on Tuesday, 2017-12-12 during the regular services deploy window. Specifically, we need to:

  • prepare the bump of the Parsoid and MCS versions in RESTBase
  • prepare the storage switch to Cassandra 3 in RESTBase
  • deploy Parsoid
  • deploy MCS
  • deploy RESTBase
  • start HTML dumps for at least the biggest WPs to minimise client impact of the deployment

Furthermore, for RESTBase we need to ensure the transition period of 24h during which contents is read from both new and old storage to avoid edit failures.

ssastry added a comment.Dec 8 2017, 5:55 PM
In T179417#3822208, @mobrovac wrote:

Furthermore, for RESTBase we need to ensure the transition period of 24h during which contents is read from both new and old storage to avoid edit failures.

Just in case others are confused like I was, RB is assuming a very generous 24-hour visual editor session, i.e. a VE edit session is started before deploy and is active upto 24 hours after deploy. They want to ensure that a save at any point in that 24+ hour period does the right thing by fetching data-parsoid from cassandra-2 for the HTML that was fetched from cassandra-2 before deploy.

gerritbot added a comment.Dec 8 2017, 9:00 PM

Change 396473 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] [Config] Enable new storage for Parsoid for all domains

https://gerrit.wikimedia.org/r/396473

bearND added a subscriber: bearND.Dec 9 2017, 3:41 AM
In T179417#3823603, @ssastry wrote:
In T179417#3822208, @mobrovac wrote:

Furthermore, for RESTBase we need to ensure the transition period of 24h during which contents is read from both new and old storage to avoid edit failures.

Just in case others are confused like I was, RB is assuming a very generous 24-hour visual editor session, i.e. a VE edit session is started before deploy and is active upto 24 hours after deploy. They want to ensure that a save at any point in that 24+ hour period does the right thing by fetching data-parsoid from cassandra-2 for the HTML that was fetched from cassandra-2 before deploy.

I hope that means that RB picks the latest one of the two.

gerritbot added a comment.Dec 11 2017, 9:41 PM

Change 397633 had a related patch set uploaded (by BearND; owner: BearND):
[mediawiki/services/mobileapps@master] Bump mobile-sections and definitions version

https://gerrit.wikimedia.org/r/397633

gerritbot added a comment.Dec 11 2017, 9:54 PM

Change 397633 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Bump mobile-sections and definitions version

https://gerrit.wikimedia.org/r/397633

gerritbot added a comment.Dec 12 2017, 6:54 PM

Change 396473 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] [Config] Enable new storage for Parsoid for all domains

https://gerrit.wikimedia.org/r/396473

Stashbot added a comment.Dec 12 2017, 9:23 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-12T21:23:51Z] <mobrovac@tin> Started deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417

Stashbot added a comment.Dec 12 2017, 9:28 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-12T21:28:07Z] <mobrovac@tin> Finished deploy [restbase/deploy@dceab2e]: Switch to Parsoid content v1.6.0 and switch to Cassandra 3 storage - T179417 (duration: 04m 16s)

mobrovac added a comment.Dec 12 2017, 9:43 PM

The storage back-end has been switched. I started dumps of en, de, fr, it and es WPs in a screen session on praseodymium targeting restbase.svc.codfw.wmnet. Once they are done, we need to merge and deploy PR #920 at which point this task can be considered done.

mobrovac added a comment.Dec 12 2017, 10:03 PM
In T179417#3832531, @mobrovac wrote:

[...] I started dumps of en, de, fr, it and es WPs in a screen session on praseodymium targeting restbase.svc.codfw.wmnet.

I had to stop them because they were hammering Cassandra 3 too much. We'll start them slowly and consecutively at a later point, probably tomorrow.

gerritbot added a comment.Dec 13 2017, 7:14 PM

Change 398101 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] [Config] Remove new_storage_enabled_parsoid config stanza

https://gerrit.wikimedia.org/r/398101

gerritbot added a comment.Dec 13 2017, 7:22 PM

Change 398101 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] [Config] Remove new_storage_enabled_parsoid config stanza

https://gerrit.wikimedia.org/r/398101

Stashbot added a comment.Dec 13 2017, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-13T19:30:57Z] <ppchelko@tin> Started deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417

Stashbot added a comment.Dec 13 2017, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-13T19:35:40Z] <ppchelko@tin> Finished deploy [restbase/deploy@3f4bedc]: Remove references to Cassandra 2 from Parsoid storage T179417 (duration: 04m 43s)

mobrovac closed this task as Resolved.Dec 26 2017, 9:22 AM
mobrovac claimed this task.
mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (doing).

The dumps for en, de, fr, es, it and he have been completed. Calling this one done \o/

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL