Page MenuHomePhabricator
Create Task
Maniphest T144010

Drop eu_touched in production
Closed, ResolvedPublic
Actions

Description

Schema change progress:

  • wikitech - doesn't apply, this table doesn't exist there.
    • labswiki
    •  labtestwiki

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui added a comment.EditedJul 19 2018, 8:33 AM

s2 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • db1125
  • db1122
  • db1105
  • db1103
  • db1090
  • db1076
  • db1074
  • db1066
gerritbot added a comment.Jul 19 2018, 1:38 PM

Change 446817 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1103:3312

https://gerrit.wikimedia.org/r/446817

gerritbot added a comment.Jul 19 2018, 1:41 PM

Change 446817 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1103:3312

https://gerrit.wikimedia.org/r/446817

Stashbot added a comment.Jul 19 2018, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-19T13:43:27Z] <marostegui> Deploy schema change on db1103:3312 T144010 T51190 T199368

gerritbot added a comment.Jul 19 2018, 2:24 PM

Change 446830 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3312

https://gerrit.wikimedia.org/r/446830

gerritbot added a comment.Jul 19 2018, 2:28 PM

Change 446830 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3312

https://gerrit.wikimedia.org/r/446830

Stashbot added a comment.Jul 19 2018, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-19T14:30:31Z] <marostegui> Deploy schema change on db1105:3312 T144010 T51190 T199368

gerritbot added a comment.Jul 20 2018, 5:35 AM

Change 447020 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1076

https://gerrit.wikimedia.org/r/447020

gerritbot added a comment.Jul 20 2018, 5:42 AM

Change 447020 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1076

https://gerrit.wikimedia.org/r/447020

Stashbot added a comment.Jul 20 2018, 5:49 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-20T05:49:30Z] <marostegui> Deploy schema change on db1076 T144010 T51190 T199368

gerritbot added a comment.Jul 20 2018, 6:08 AM

Change 447026 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1074

https://gerrit.wikimedia.org/r/447026

gerritbot added a comment.Jul 20 2018, 6:11 AM

Change 447026 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1074

https://gerrit.wikimedia.org/r/447026

Stashbot added a comment.Jul 20 2018, 6:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-20T06:13:08Z] <marostegui> Deploy schema change on db1074 with replication, this will generate lag on labsdb:s2 T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 20 2018, 8:00 AM
Marostegui updated the task description. (Show Details)Jul 22 2018, 6:21 AM
Stashbot added a comment.Jul 23 2018, 4:42 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-23T04:42:37Z] <marostegui> Deploy schema change on db1061 (s6 primary master) T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 23 2018, 5:08 AM
gerritbot added a comment.Jul 23 2018, 7:23 AM

Change 447363 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1122 for alter table

https://gerrit.wikimedia.org/r/447363

gerritbot added a comment.Jul 23 2018, 7:32 AM

Change 447363 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1122 for alter table

https://gerrit.wikimedia.org/r/447363

Stashbot added a comment.Jul 23 2018, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-23T07:33:52Z] <marostegui> Deploy schema change on db1122 T144010 T51190 T199368

Stashbot added a comment.Jul 23 2018, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-23T13:23:25Z] <marostegui> Deploy schema change on labswiki and labstestwiki T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 23 2018, 1:29 PM
Stashbot added a comment.Jul 23 2018, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-23T13:36:43Z] <marostegui> Deploy schema change on s4 codfw master (db2051), this will generate lag on s4 codfw T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 24 2018, 4:43 AM
Stashbot added a comment.Jul 24 2018, 4:44 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T04:44:49Z] <marostegui> Deploy schema change on db1066 (s2 primary master) T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 24 2018, 5:05 AM
Stashbot added a comment.Jul 24 2018, 5:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T05:05:55Z] <marostegui> Deploy schema change on db1081 T144010 T51190 T199368

Marostegui added a comment.EditedJul 24 2018, 6:10 AM

s4 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • db1125
  • db1121
  • db1103
  • db1097
  • db1091
  • db1084
  • db1081
  • db1068
Marostegui updated the task description. (Show Details)Jul 24 2018, 6:11 AM
Stashbot added a comment.Jul 24 2018, 6:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T06:32:07Z] <marostegui> Deploy schema change on dbstore1002:s4 T144010 T51190 T199368

Stashbot added a comment.Jul 24 2018, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T09:17:56Z] <marostegui> Deploy schema change on db1097:3314 T144010 T51190 T199368

Stashbot added a comment.Jul 24 2018, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T13:40:34Z] <marostegui> Deploy schema change on db1081 T144010 T51190 T199368

Stashbot added a comment.Jul 24 2018, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T13:43:34Z] <marostegui> Deploy schema change on db1084 T144010 T51190 T199368

Stashbot added a comment.Jul 24 2018, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-24T14:09:29Z] <marostegui> Deploy schema change on db1103:3314 T144010 T51190 T199368

Stashbot added a comment.Jul 25 2018, 7:12 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-25T07:12:08Z] <marostegui> Deploy schema change on db1091 T144010 T51190 T199368

Stashbot added a comment.Jul 25 2018, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-25T08:14:31Z] <marostegui> Deploy schema change on db1121 with replication, this will generate lag on labsdb hosts for s4 T144010 T51190 T199368

Marostegui added a comment.EditedJul 25 2018, 10:00 AM

This table fails to get altered when there is high concurrency (I have had to stop replication on s4 slaves to be able to alter it), so I am not going to even attempt to alter it on the master. I will do that once eqiad is passive.

They were failing with duplicate key (which didn't exist really) so I think it was a matter of high writes coming in and maybe race condition when the alter was finishing and doing the final r

That is my theory seeing that: it was failing with different keys every time and once replication was stopped, it would work straightaway

Marostegui updated the task description. (Show Details)Jul 25 2018, 10:01 AM
Marostegui added a parent task: T189107: DB meta task for next DC failover issues.
Stashbot added a comment.Jul 25 2018, 10:07 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-25T10:07:08Z] <marostegui> Deploy schema change on db2040 (s7 codfw master) with replication, this will generate lag on s7 codfw T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 25 2018, 10:59 AM
Marostegui added a comment.EditedJul 25 2018, 1:19 PM

s7 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • db1125
  • db1101
  • db1098
  • db1094
  • db1090
  • db1086
  • db1079
  • db1062
GoranSMilovanovic mentioned this in T200330: Cannot SSH to stat1004.Jul 25 2018, 1:24 PM
Stashbot added a comment.Jul 25 2018, 2:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-25T14:45:09Z] <marostegui> Deploy schema change on db1098:3317 T144010 T51190 T199368

GoranSMilovanovic subscribed.Jul 25 2018, 3:03 PM

@Marostegui Hi, could you please confirm the following: no other changes in the wbc_entity_usage schema will take place except for dropping the eu_touched field?

The WDCM operations rely on weekly Apache Sqoop runs across the wbc_entity_usage tables from stat1004. These runs produce the necessary Hive tables from which we perform ETL for Wikidata usage statistics. It is necessary for the WDCM's consistent operation to let me know immediately if and when any changes in the wbc_entity_usage will take place, simply because I will need to adjust the related Sqoop/R scripts accordingly.

Thanks a lot!

Marostegui added a comment.Jul 25 2018, 3:05 PM

There is no other schema change related to wbc_entity_usage on-going apart from this, yes :-)

GoranSMilovanovic added a comment.Jul 25 2018, 3:07 PM

@Marostegui Thank you very much! Can you estimate when would the new schema become operational?

GoranSMilovanovic mentioned this in T200345: Adjust WDCM Sqoop step for changes in the wbc_entity_usage schema.Jul 25 2018, 3:08 PM
Marostegui added a comment.Jul 25 2018, 3:08 PM
In T144010#4450637, @GoranSMilovanovic wrote:

@Marostegui Thank you very much! Can you estimate when would the new schema become operational?

That's hard to say, you can track the progress on the task description, but it will take a few more weeks at least.

Stashbot added a comment.Jul 26 2018, 5:02 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T05:02:40Z] <marostegui> Deploy schema change on db1101:3317 T144010 T51190 T199368

Stashbot added a comment.Jul 26 2018, 6:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T06:47:27Z] <marostegui> Deploy schema change on db1086 T144010 T51190 T199368

Stashbot added a comment.Jul 26 2018, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T08:20:00Z] <marostegui> Deploy schema change on db1090:3317 T144010 T51190 T199368

Stashbot added a comment.Jul 26 2018, 9:22 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T09:22:53Z] <marostegui> Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368

Stashbot added a comment.Jul 27 2018, 4:59 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-27T04:59:17Z] <marostegui> Deploy schema change on db1094 T144010 T51190 T199368

Stashbot added a comment.Jul 30 2018, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-30T05:00:28Z] <marostegui> Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 30 2018, 5:26 AM
Stashbot added a comment.Jul 30 2018, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-30T07:32:15Z] <marostegui> Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Jul 30 2018, 8:46 AM
Stashbot added a comment.Jul 30 2018, 8:49 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-30T08:49:36Z] <marostegui> Deploy schema change on dbstore1002:s8 T144010 T51190 T199368

Marostegui added a comment.EditedJul 30 2018, 1:28 PM

s8 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • db1124
  • db1109
  • db1104
  • db1101
  • db1099
  • db1092
  • db1087
  • db1071
Marostegui updated the task description. (Show Details)Jul 30 2018, 1:28 PM
Stashbot added a comment.Jul 31 2018, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-31T05:00:04Z] <marostegui> Deploy schema change on dbstore1002:s8 T144010 T51190 T199368/script unload irssinotifier

Stashbot added a comment.Jul 31 2018, 5:01 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-31T05:01:05Z] <marostegui> Deploy schema change on db1099:3318 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 4:46 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T04:46:05Z] <marostegui> Deploy schema change on db1101:3318 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 6:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T06:47:53Z] <marostegui> Deploy schema change on db1087 with replication, this will generate lag on labs:s8 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 8:37 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T08:37:24Z] <marostegui> Deploy schema change on db1109:3318 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T09:19:03Z] <marostegui> Deploy schema change on db1092 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T09:50:15Z] <marostegui> Deploy schema change on db2048 (s1 codfw master) with replication, this will generate lag on codfw s1 T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Aug 1 2018, 10:31 AM
Marostegui added a comment.EditedAug 1 2018, 11:12 AM

s1 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • dbstore1001
  • db1124
  • db1119
  • db1118 (test host)
  • db1114
  • db1106
  • db1105
  • db1102 (test host)
  • db1099
  • db1095 (test host)
  • db1089
  • db1083
  • db1080
  • db1067
Stashbot added a comment.Aug 1 2018, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T12:16:01Z] <marostegui> Deploy schema change on db1099:3311 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T12:35:20Z] <marostegui> Deploy schema change on db1105:3311 T144010 T51190 T199368

Stashbot added a comment.Aug 1 2018, 1:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T13:28:57Z] <marostegui> Deploy schema change on db1083 T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Aug 2 2018, 4:53 AM
Stashbot added a comment.Aug 2 2018, 6:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-02T06:18:11Z] <marostegui> Deploy schema change on db1106 with replication, this will generate lag on labsdb:s1 T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Aug 2 2018, 8:04 AM
Stashbot added a comment.Aug 2 2018, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-02T08:14:50Z] <marostegui> Deploy schema change on db2043 (s3 codfw master) this will generate lag on codfw:s3 T144010 T51190 T199368

Marostegui updated the task description. (Show Details)Aug 2 2018, 3:51 PM
Marostegui added a comment.EditedAug 3 2018, 8:19 AM

s3 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1002
  • db1124
  • db1123
  • db1078
  • db1077
  • db1075
Marostegui updated the task description. (Show Details)Aug 3 2018, 11:24 AM
Stashbot added a comment.Aug 6 2018, 6:49 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-06T06:49:22Z] <marostegui> Deploy schema change on db1077 with replication, this will generate lag on labsdb:s3 T144010 T51190 T199368

Stashbot added a comment.Aug 7 2018, 5:21 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-07T05:21:26Z] <marostegui> Deploy schema change on db1123 T144010 T51190 T199368

Stashbot added a comment.Aug 7 2018, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-07T06:58:11Z] <marostegui> Deploy schema change on db1075 (s3 master) T144010 T51190 T199368

Marostegui changed the task status from Open to Stalled.Aug 7 2018, 7:36 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from In progress to Blocked external/Not db team on the DBA board.

Blocking this until we have done DC failover so we can do s4 primary master (T144010#4449908)

eranroz mentioned this in T201563: issue with accessing wbc_entity_usage from tools-db.Aug 8 2018, 9:35 PM
GoranSMilovanovic added a comment.Sep 3 2018, 1:38 PM

@Marostegui I see that in enwiki there is no more eu_touched in the wbc_entity_usage table.

This made one of my crucial scripts running from stat1004 statbox crash, and the whole WDCM system is thus not being updated.

Could you please confirm that these changes are in effect across all project databases that maintain the wbc_entity_usage tables?

Thanks a lot!

Marostegui added a comment.Sep 3 2018, 1:40 PM
In T144010#4553520, @GoranSMilovanovic wrote:

@Marostegui I see that in enwiki there is no more eu_touched in the wbc_entity_usage table.

This made one of my crucial scripts running from stat1004 statbox crash, and the whole WDCM system is thus not being updated.

Could you please confirm that these changes are in effect across all project databases that maintain the wbc_entity_usage tables?

Thanks a lot!

Hello,

Everything is done except s4 (commons) master as specified on the description of the task.

GoranSMilovanovic mentioned this in T203389: WDCM did not update in September.Sep 3 2018, 1:40 PM

@Marostegui Thank you!

Stashbot added a comment.Sep 13 2018, 8:16 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-13T08:16:31Z] <marostegui> Stop replication on s4 eqiad master (db1068) and deploy a schema change - this will generate lag on s4 eqiad - T144010

Marostegui changed the task status from Stalled to Open.Sep 13 2018, 8:17 AM
Marostegui raised the priority of this task from Low to Medium.
Marostegui moved this task from Blocked external/Not db team to In progress on the DBA board.Sep 13 2018, 8:37 AM
Marostegui closed this task as Resolved.Sep 13 2018, 9:42 AM
Marostegui updated the task description. (Show Details)

All done

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL