Page MenuHomePhabricator
Create Task
Maniphest T162539

Deploy schema change for adding term_full_entity_id column to wb_terms table
Closed, ResolvedPublic
Actions

Description

This new column needs to be added soon. We decided to do it for masters during the switchover.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclineddchenT118706 Conduct heuristic evaluation of image upload and insert flow in VisualEditor
 OpenNoneT115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
 OpenNoneT115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
 DuplicateNoneT115864 Figure out if the description of the image can be used as the caption on-wiki
 OpenFeatureNoneT53032 When inserting an image, set its caption by default to be the Commons image description
 OpenFeatureNoneT39534 Wikimedia Commons should support searching by color
 DuplicateNoneT39535 Wikimedia Commons should support filtering by color
 ResolvedNoneT19503 Provide metadata support on Wikimedia Commons
 ResolvedNoneT51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
 ResolvedNoneT68108 [Epic] Store media information for files on Wikimedia Commons as structured data
 DuplicateNoneT66288 basic support for structured data on mediawiki files
 InvalidLydia_PintscherT76012 make use of new entity type for multimedia / structured data of media files
 OpenNoneT109579 [Epic] Give more sister projects access to Wikidata
 OpenNoneT187900 There is no way to reference a specific quote on Wikiquote
 StalledNoneT71753 [Story] Wikibase / Wikidata support on Wikiquote
 OpenNoneT67626 [Epic] Support for queries on-wiki (automated list generation)
 ResolvedAddshoreT76019 [Story] Support new types of Entities in Wikibase Client
 ResolvedthiemowmdeT135650 [Task] Migrate PropertySuggester away from assuming all entities are numeric
 ResolvedAddshoreT75496 [Epic] Support new types of Entities in Wikibase Repository
 DeclinedNoneT58711 [Task] Update the wb_terms table so it does not have a numeric entity id
 OpenNoneT30599 Deadlock tracking bug (tracking)
 ResolvedhooT111535 Wikibase\Repo\Store\SQL\EntityPerPageTable::{closure} creating high number of deadlocks
 ResolvedLydia_PintscherT146637 Wikidata 2016 Q4 goals
 ResolvedNoneT86530 Replace wb_terms table with more specialized mechanisms for terms (tracking)
 DeclinedNoneT51982 Add missing wb_entity_per_page entries on LinksUpdate
 InvalidLydia_PintscherT70176 EntityPerPageTable class should be usable from the client
 ResolvedLadsgroupT67333 Wikibase\EntityPerPageTable::getItemsWithoutSitelinks slow query with large LIMIT offset
 ResolvedAddshoreT114902 Remove numeric entity IDs from database schema
 ResolvedLadsgroupT95685 Drop wb_entity_per_page table
 DeclinedNoneT114903 Migrate wb_terms to using prefixed entity IDs instead of numeric IDs
 Resolved• MarosteguiT162539 Deploy schema change for adding term_full_entity_id column to wb_terms table

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Apr 24 2017, 6:12 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-24T06:12:32Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1092, depoll db1087 - T162539 T163548 (duration: 02m 19s)

Stashbot added a comment.Apr 24 2017, 6:12 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-24T06:12:44Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms on db1087 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

• Marostegui added a comment.Apr 24 2017, 1:04 PM

db1087 is done:

root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb1087 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.Apr 24 2017, 1:08 PM

Change 349953 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1082

https://gerrit.wikimedia.org/r/349953

Stashbot added a comment.Apr 24 2017, 1:11 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-24T13:11:03Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms on db1063 - T162539 https://phabricator.wikimedia.org/T163548

Stashbot added a comment.Apr 24 2017, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-24T13:12:04Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms on db1082 - T162539 - T163548

gerritbot added a comment.Apr 24 2017, 1:15 PM

Change 349953 abandoned by Marostegui:
db-eqiad.php: Depool db1082

Reason:
not needed

https://gerrit.wikimedia.org/r/349953

• Marostegui added a comment.Apr 25 2017, 5:53 AM

db1082 is done:

root@neodymium:~# mysql --skip-ssl -hdb1082 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.Apr 25 2017, 5:58 AM

Change 350119 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1071 and db1026

https://gerrit.wikimedia.org/r/350119

gerritbot added a comment.Apr 25 2017, 6:04 AM

Change 350119 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1071 and db1026

https://gerrit.wikimedia.org/r/350119

Stashbot added a comment.Apr 25 2017, 6:06 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-25T06:06:12Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1071, depool db1026 - T162539 T163548 (duration: 01m 17s)

• Marostegui added a comment.Apr 26 2017, 5:54 AM

db1071 and db1063 are done:

root@neodymium:~# mysql --skip-ssl -hdb1071 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
root@neodymium:~# mysql --skip-ssl -hdb1063 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.Apr 26 2017, 6:50 AM

Change 350369 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1071

https://gerrit.wikimedia.org/r/350369

gerritbot added a comment.Apr 26 2017, 6:53 AM

Change 350369 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1071

https://gerrit.wikimedia.org/r/350369

Stashbot added a comment.Apr 26 2017, 6:56 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-26T06:56:08Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1071 - T162539 T163548 (duration: 02m 24s)

• Marostegui added a comment.Apr 26 2017, 9:21 AM

db1026 is done

root@neodymium:~# mysql --skip-ssl -hdb1026 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.Apr 26 2017, 1:16 PM

Change 350411 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1026

https://gerrit.wikimedia.org/r/350411

Stashbot added a comment.Apr 26 2017, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-26T13:23:23Z] <marostegui> Deploy alter table db1045 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

Stashbot added a comment.Apr 26 2017, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-26T13:27:01Z] <marostegui> Deploy alter table labsdb1001 https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

Stashbot added a comment.Apr 26 2017, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-26T13:29:44Z] <marostegui> Deploy alter table on db1069 (wikidatawiki) https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

gerritbot added a comment.Apr 26 2017, 1:59 PM

Change 350411 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1026, depool db1045

https://gerrit.wikimedia.org/r/350411

Stashbot added a comment.Apr 26 2017, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-26T14:00:30Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1026, depool db1045 - T162539 T163548 (duration: 00m 53s)

Reedy added a project: DBA.Apr 26 2017, 3:14 PM
Reedy removed a project: Patch-For-Review.
• Marostegui moved this task from Backlog to In progress on the Schema-change-in-production board.Apr 26 2017, 3:17 PM
Reedy moved this task from Triage to In progress on the DBA board.Apr 26 2017, 3:19 PM
• Marostegui added a comment.Apr 26 2017, 4:04 PM

For the record: the alter on labsdb1001 has been killed because we ran out of space there, same thing will happen on labsdb1003, so we need to find a solution for those two hosts before applying this change.
If we start using this column before it is applied there, it will break replication on labs.

• Marostegui added a comment.Apr 26 2017, 5:44 PM
In T162539#3214508, @Marostegui wrote:

For the record: the alter on labsdb1001 has been killed because we ran out of space there, same thing will happen on labsdb1003, so we need to find a solution for those two hosts before applying this change.
If we start using this column before it is applied there, it will break replication on labs.

We have hotfixed the issue by removing an unused partition and adding it to the main volume. So l am re-running the ALTER on labsdb1001 now.

• Marostegui added a comment.Apr 27 2017, 5:58 AM

labsdb1001 is done:

[root@labsdb1001 05:57 /root]
# mysql  wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
Stashbot added a comment.Apr 27 2017, 5:59 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-27T05:59:54Z] <marostegui> Deploy alter table labsdb1003 (wikidatawiki) https://phabricator.wikimedia.org/T162539%C2%A0https://phabricator.wikimedia.org/T163548

Stashbot mentioned this in T130067: Add wl_id to watchlist tables on production dbs.Apr 27 2017, 6:09 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-27T06:08:59Z] <marostegui> Deploy alter table on s5 (wikidatawiki) on db1049 - T130067 T162539

Stashbot added a comment.Apr 27 2017, 6:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-27T06:11:00Z] <marostegui> Deploy alter table on s5 (wikidatawiki) on db1070 (running locally instead of neodymium as this host will be affected by the network maintenance) - T130067 T162539

• Marostegui added a comment.Apr 27 2017, 6:14 AM

I am altering db1070 locally, instead of from neodymium as this host is going to be affected by: T162681
I ran it with set session sql_log_bin=0

• Marostegui added a comment.Apr 27 2017, 1:49 PM

db1069 is done:

root@db1069:~# mysql -S /tmp/mysql.s5.sock --skip-ssl wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
• Marostegui added a comment.Apr 27 2017, 3:59 PM

db1070 is done:

root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb1070 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
• Marostegui added a comment.Apr 28 2017, 6:24 AM

db1049 is done:

root@neodymium:~# mysql --skip-ssl -hdb1049 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

db1045 is done:

root@neodymium:~# mysql --skip-ssl -hdb1045 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

All the eqiad core hosts (dbstore servers pending) are now done.
This is not yet finished and this column cannot be used until this ticket gets fully resolved as it is not present on codfw yet

gerritbot added a comment.Apr 28 2017, 7:50 AM

Change 350798 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1045

https://gerrit.wikimedia.org/r/350798

gerritbot added a project: Patch-For-Review.Apr 28 2017, 7:50 AM
gerritbot added a comment.Apr 28 2017, 7:55 AM

Change 350798 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1045

https://gerrit.wikimedia.org/r/350798

Stashbot added a comment.Apr 28 2017, 7:58 AM

Mentioned in SAL (#wikimedia-operations) [2017-04-28T07:58:44Z] <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1045 - T162539 T163548 (duration: 02m 38s)

Stashbot added a comment.May 4 2017, 6:03 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-04T06:03:48Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - dbstore2002 - T162539 T163548

gerritbot added a comment.May 4 2017, 6:05 AM

Change 351766 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2066

https://gerrit.wikimedia.org/r/351766

gerritbot added a comment.May 4 2017, 6:08 AM

Change 351766 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2066

https://gerrit.wikimedia.org/r/351766

Stashbot added a comment.May 4 2017, 6:10 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-04T06:10:09Z] <marostegui@naos> Synchronized wmf-config/db-codfw.php: Depool db2066 - T162539 T163548 (duration: 01m 25s)

Stashbot added a comment.May 4 2017, 6:10 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-04T06:10:26Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - db2066 - T162539 T163548

• Marostegui added a comment.May 4 2017, 3:02 PM

db2066 is done:

root@neodymium:~# mysql --skip-ssl -hdb2066.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.May 4 2017, 3:06 PM

Change 351858 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2066, depool db2059

https://gerrit.wikimedia.org/r/351858

gerritbot added a comment.May 4 2017, 3:12 PM

Change 351858 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2066, depool db2059

https://gerrit.wikimedia.org/r/351858

Stashbot added a comment.May 4 2017, 3:14 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-04T15:14:28Z] <marostegui@naos> Synchronized wmf-config/db-codfw.php: Repool db2066, depool db2059 - T162539 T163548 (duration: 01m 06s)

Stashbot added a comment.May 4 2017, 3:16 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-04T15:16:17Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - db2059- https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

• Marostegui added a comment.May 5 2017, 5:44 AM

db2059 is done:

root@neodymium:~# mysql --skip-ssl -hdb2059.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

dbstore2002 is done:

root@neodymium:~# mysql --skip-ssl -hdbstore2002.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.May 5 2017, 5:48 AM

Change 352076 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2059, depool db2052

https://gerrit.wikimedia.org/r/352076

gerritbot added a comment.May 5 2017, 5:53 AM

Change 352076 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2059, depool db2052

https://gerrit.wikimedia.org/r/352076

Stashbot added a comment.May 5 2017, 5:55 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T05:55:01Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2059, depool db2052 - T162539 T163548 (duration: 00m 40s)

Stashbot added a comment.May 5 2017, 5:55 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T05:55:19Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - db2052 - T162539 T163548

Stashbot added a comment.May 5 2017, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T07:11:58Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - dbstore2001 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

Stashbot added a comment.May 5 2017, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T07:49:15Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - dbstore1002 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

Lydia_Pintscher moved this task from incoming to in progress on the Wikidata board.May 5 2017, 12:29 PM
Lydia_Pintscher mentioned this in T159718: Evaluate how to best add a column for full entity ID to wb_terms without affecting wikidata.org users.May 5 2017, 12:34 PM
• Marostegui added a comment.May 5 2017, 3:12 PM

db2052 is done:

root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb2052.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.May 5 2017, 3:21 PM

Change 352167 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2052, depool db2045

https://gerrit.wikimedia.org/r/352167

gerritbot added a comment.May 5 2017, 3:26 PM

Change 352167 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2052, depool db2045

https://gerrit.wikimedia.org/r/352167

Stashbot added a comment.May 5 2017, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T15:28:12Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2052, depool db2045 - T162539 T163548 (duration: 00m 41s)

Stashbot added a comment.May 5 2017, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2017-05-05T15:28:21Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms - db2045 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548

• Marostegui added a comment.May 8 2017, 6:08 AM

db2045 is done:

root@neodymium:~# mysql --skip-ssl -hdb2045.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

dbstore2001 is done:

root@neodymium:~# mysql --skip-ssl -hdbstore2001.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

dbstore1002 is done:

root@neodymium:~# mysql --skip-ssl -hdbstore1002 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
gerritbot added a comment.May 8 2017, 6:11 AM

Change 352543 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2045, depool db2038

https://gerrit.wikimedia.org/r/352543

gerritbot added a comment.May 8 2017, 6:14 AM

Change 352543 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2045, depool db2038

https://gerrit.wikimedia.org/r/352543

Stashbot added a comment.May 8 2017, 6:18 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-08T06:18:18Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2045, depool db2038 - T162539 T163548 (duration: 00m 40s)

• Marostegui added a comment.May 8 2017, 3:55 PM

db2038 is done:

root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdb2038.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))

Only pending host is the codfw master which will be done tomorrow morning.

gerritbot added a comment.May 8 2017, 3:57 PM

Change 352607 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2038

https://gerrit.wikimedia.org/r/352607

Stashbot added a comment.May 9 2017, 5:54 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-09T05:54:39Z] <marostegui> Deploy alter table on wikidatawiki.wb_terms on codfw master db2023 - https://phabricator.wikimedia.org/T162539 - https://phabricator.wikimedia.org/T163548

gerritbot added a comment.May 9 2017, 6:00 AM

Change 352607 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2038

https://gerrit.wikimedia.org/r/352607

Stashbot added a comment.May 9 2017, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-09T06:01:53Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2038 - T162539 T163548 (duration: 00m 41s)

Ladsgroup added a comment.May 9 2017, 4:05 PM

Hey @Marostegui, I can't see db2023 (master in codfw) gets deplooed and repooled (maybe you don't do that for masters). But I don't know if this schema change is done for db2023 or not. Please tell me once it's done, so we deploy the config change.

• jcrespo added a comment.May 9 2017, 4:17 PM

Please @aude @Ladsgroup don't stress Marostegui! :-) As you can see he is currently working (very hard, I would say) on this, but you need to be patient. Schema changes take time, and they require careful checking after the fact. Marostegui or me will resolve this ticket or move it to the "Done" column when we are confident it is finished. Until then, even if you see it mostly completed, any deployment assuming this is done would risk a site-wide outage due to replication getting broken, and you probably do not want that!

You will be the first ones to know when this is ok to proceed.

Ladsgroup added a comment.May 9 2017, 4:21 PM

My apologies. I was a little bit confused about the codfw master. Thanks for letting me know.

• Marostegui added a comment.May 9 2017, 6:02 PM
In T162539#3248673, @Ladsgroup wrote:

Hey @Marostegui, I can't see db2023 (master in codfw) gets deplooed and repooled (maybe you don't do that for masters). But I don't know if this schema change is done for db2023 or not. Please tell me once it's done, so we deploy the config change.

We do not pool/depool masters. That is why you don't see the change. The master is currently being altered. It will take a bit longer than the rest of hosts as the hardware is less powerful.

As Jaime said, you will clearly know when the whole shard is done. We will say so on the ticket and close it. Please be patient :-)

aude added a comment.May 10 2017, 1:09 AM

@jcrespo no hurry on our side. (was just more curious how this is going and seems good to me)

• Marostegui added a comment.May 10 2017, 6:09 AM

codfw master, db2023 is done:

root@neodymium:~# mysql --skip-ssl -hdb2023.codfw.wmnet wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
Stashbot added a comment.May 10 2017, 6:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-05-10T06:15:23Z] <marostegui> Deploy alter table wikidatawiki.wb_terms on dbstore1001 - T162539 T163190

Stashbot mentioned this in T163190: Checksum data on s7.May 10 2017, 6:15 AM
• Marostegui added a comment.May 11 2017, 12:04 PM

dbstore1001 is done:

root@neodymium:/home/marostegui/git/software/dbtools# mysql --skip-ssl -hdbstore1001 wikidatawiki -e "show create table wb_terms\G" | egrep "term_full_entity_id|term_search_full"
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
• Marostegui closed this task as Resolved.May 12 2017, 5:53 AM

All hosts are done:

root@neodymium:/home/marostegui/git/software/dbtools# cat s5.hosts | while read host port; do echo $host; mysql --skip-ssl -h $host -P $port -e "show create table wikidatawiki.wb_terms\G" |  egrep "term_full_entity_id|term_search_full"; done
dbstore2002.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
dbstore2001.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2038.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2045.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2052.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2059.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2066.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db2023.codfw.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
labsdb1001.eqiad.wmnet
ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1003.eqiad.wmnet
ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1069.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
dbstore1001.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
dbstore1002.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1026.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1045.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1049.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1070.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1071.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1082.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1087.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1092.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
db1063.eqiad.wmnet
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
root@neodymium:/home/marostegui/git/software/dbtools# cumin 'labsdb100[1-3]*' 'sudo mysql --skip-ssl -e "show create table wikidatawiki.wb_terms\G" |  egrep "term_full_entity_id|term_search_full"'
2 hosts will be targeted:
labsdb[1001,1003].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(2) labsdb[1001,1003].eqiad.wmnet
----- OUTPUT of 'sudo mysql --ski...erm_search_full"' -----
  `term_full_entity_id` varbinary(32) DEFAULT NULL,
  KEY `term_full_entity` (`term_full_entity_id`),
  KEY `term_search_full` (`term_language`,`term_full_entity_id`,`term_type`,`term_search_key`(16))
================
PASS |████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [00:00<00:00,  5.57hosts/s]
FAIL |                                                                                        |   0% (0/2) [00:00<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'sudo mysql --ski...erm_search_full"'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
aude added a comment.EditedMay 12 2017, 12:49 PM

i checked test wikidata (testwikidatawiki) and it needs the change also. the tables there tend to be small.

select count(*) from wb_terms;
+----------+
| count(*) |
+----------+
|   192947 |
+----------+

Is it safe to apply this schema change test wikidata on terbium? (like how we add new tables) me or someone else can do that, if it is ok.

• Marostegui added a comment.May 12 2017, 1:26 PM
In T162539#3258455, @aude wrote:

i checked test wikidata (testwikidatawiki) and it needs the change also. the tables there tend to be small.

select count(*) from wb_terms;
+----------+
| count(*) |
+----------+
|   192947 |
+----------+

Is it safe to apply this schema change test wikidata on terbium? (like how we add new tables) me or someone else can do that, if it is ok.

Hi!

I can do that, however I am a bit confused, is it terbium what you really mean?

root@terbium:~# ps aux | grep mysql
root     22730  0.0  0.0  12728  2216 pts/1    S+   13:24   0:00 grep mysql
aude added a comment.May 12 2017, 1:31 PM

@Marostegui for small schema changes (e.g. adding a table, or for small wikis like test wikidata), we normally apply schema changes run it via mediawiki maintenance script:

https://www.mediawiki.org/wiki/Manual:Sql.php

We should just be sure it's okay to do this one this way

• Marostegui added a comment.May 12 2017, 1:34 PM
In T162539#3258550, @aude wrote:

@Marostegui for small schema changes (e.g. adding a table, or for small wikis like test wikidata), we normally apply schema changes run it via mediawiki maintenance script:

https://www.mediawiki.org/wiki/Manual:Sql.php

We should just be sure it's okay to do this one this way

Sure, that is fine by me if you want to do it yourself, but on which host does that test wikidata database lives?

• jcrespo added a comment.May 14 2017, 11:50 AM

@aude: don't run update.php on s3 for altering a table- you will create lag
on 900 wikis unless connections are cleared and table is pre-warmed-up (and
appropiately tested).

Also, the only wiki mentioned here was wikidatawiki, you have to create a
separate request for other wikis.

• Marostegui added a comment.May 14 2017, 3:03 PM
In T162539#3260955, @jcrespo wrote:

@aude: don't run update.php on s3 for altering a table- you will create lag
on 900 wikis unless connections are cleared and table is pre-warmed-up (and
appropiately tested).

Also, the only wiki mentioned here was wikidatawiki, you have to create a
separate request for other wikis.

Ah, thanks for the good catch, I didn't know he meant s3 (I thought he was talking about another small host or something that wasn't on my radar, hence my two questions to see where that database he wanted to alter lives in, as I was planning to check the table size and if there were slaves or stuff).
I am glad you mentioned s3 now I have the full picture.

By the way:

root@db1075:/srv/sqldata/testwikidatawiki# ls -lh wb_terms.ibd
-rw-rw---- 1 mysql mysql 68M May 14 13:43 wb_terms.ibd
aude added a comment.May 14 2017, 4:56 PM

@jcrespo @Marostegui created T165246 for the task of adding the column on test wikidata.

• Marostegui added a comment.May 14 2017, 7:56 PM

Thanks @aude!

daniel mentioned this in T114903: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs.Jul 24 2017, 11:12 AM
daniel edited parent tasks, added: T114903: Migrate wb_terms to using prefixed entity IDs instead of numeric IDs; removed: T159851: Add a column for full entity ID to wb_terms table.Jul 24 2017, 11:15 AM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 24 2017, 11:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits