Page MenuHomePhabricator
Create Task
Maniphest T171629

Stop collecting Data for outdated schemas PageCreation, PageDeletion, PageMove, PageRestoration. Archive tables on hdfs
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Stop collecting Data for PageCreation, PageDeletion, PageMove, PageRestoration schemas, archive table on hdfs

This data is now available on mediawiki edit reconstruction and in events coming from EventBus

Lets:

  • stop collecting data for these schemas
  • back up schema data to known location on hdfs (and document that on eventlogging docs/schema talk page)
  • Remove instrumentation that is sending this data

Details

SubjectRepoBranchLines +/-
PageCreate events are no longer flowingoperations/puppetproduction+1 -1
Removing from whitelist tables that no longer existoperations/puppetproduction+0 -53
Removing schema that no longer existsoperations/puppetproduction+1 -2
Removing instrumentation for outdated schemasmediawiki/extensions/WikimediaEventsmaster+0 -146
Stopping event collection for Page eventsoperations/puppetproduction+2 -1
Customize query in gerrit

Event Timeline

• Nuria created this task.Jul 25 2017, 6:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 25 2017, 6:06 PM
• Nuria renamed this task from Stop collecting Data for pageCreate schema, archive table on hdfs to Stop collecting Data for PageCreation schema, archive table on hdfs .Jul 25 2017, 6:10 PM
• fdans triaged this task as Medium priority.Jul 27 2017, 3:36 PM
• fdans moved this task from Incoming to Operational Excellence Future on the Analytics board.
• fdans moved this task from Operational Excellence Future to Dashiki on the Analytics board.
• fdans set the point value for this task to 8.
• fdans edited subscribers, added: • fdans, elukey; removed: Aklapper.
• Nuria claimed this task.Sep 12 2017, 11:13 PM
• Nuria edited projects, added Analytics-Kanban; removed Analytics.
• Nuria updated the task description. (Show Details)
• Nuria changed the point value for this task from 8 to 5.
• Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Sep 12 2017, 11:20 PM

Change 377667 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Stopping event collection for PageCreation events

https://gerrit.wikimedia.org/r/377667

gerritbot added a project: Patch-For-Review.Sep 12 2017, 11:20 PM
• Nuria added a comment.Sep 12 2017, 11:29 PM

We can probably do the same for these three schemas:

https://meta.wikimedia.org/wiki/Schema:PageDeletion
https://meta.wikimedia.org/wiki/Schema:PageMove
https://meta.wikimedia.org/wiki/Schema:PageRestoration

gerritbot added a comment.Sep 14 2017, 5:30 PM

Change 377667 merged by Ottomata:
[operations/puppet@production] Stopping event collection for Page events

https://gerrit.wikimedia.org/r/377667

• Nuria renamed this task from Stop collecting Data for PageCreation schema, archive table on hdfs to Stop collecting Data for PageCreation and other Page* schemas, archive tables on hdfs .Sep 19 2017, 8:30 PM
• Nuria renamed this task from Stop collecting Data for PageCreation and other Page* schemas, archive tables on hdfs to Stop collecting Data for PageCreation archive tables on hdfs .Sep 19 2017, 9:11 PM
• Nuria renamed this task from Stop collecting Data for PageCreation archive tables on hdfs to Stop collecting Data for PageCreation archive table on hdfs .Sep 19 2017, 9:33 PM

Ok, we are ready to drop PageCreation_7481635_1542324 and PageCreation_7481635 from MySQL ping @elukey
Working now on removing events from mediawiki

• Nuria renamed this task from Stop collecting Data for PageCreation archive table on hdfs to Stop collecting Data for outdated schems PageCreation, PageDeletion, PageMove, archive tables on hdfs .Sep 19 2017, 10:32 PM
• Nuria renamed this task from Stop collecting Data for outdated schems PageCreation, PageDeletion, PageMove, archive tables on hdfs to Stop collecting Data for outdated schemas PageCreation, PageDeletion, PageMove, PageRestoration. Archive tables on hdfs .
• Nuria changed the point value for this task from 5 to 8.
• Nuria updated the task description. (Show Details)
• Nuria updated the task description. (Show Details)
gerritbot added a comment.Sep 19 2017, 10:41 PM

Change 379137 had a related patch set uploaded (by Nuria; owner: Nuria):
[mediawiki/extensions/WikimediaEvents@master] [WIP] Removing instrumentation for outdated schemas

https://gerrit.wikimedia.org/r/379137

gerritbot added a comment.Sep 22 2017, 6:38 PM

Change 379137 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Removing instrumentation for outdated schemas

https://gerrit.wikimedia.org/r/379137

• Nuria moved this task from In Progress to In Code Review on the Analytics-Kanban board.Sep 22 2017, 6:45 PM
ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)).Sep 22 2017, 7:01 PM
• Nuria moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Sep 28 2017, 3:03 PM
gerritbot added a comment.Sep 29 2017, 7:47 PM

Change 381493 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Removing schema that no longer exists

https://gerrit.wikimedia.org/r/381493

gerritbot added a comment.Oct 2 2017, 1:26 PM

Change 381493 merged by Ottomata:
[operations/puppet@production] Removing schema that no longer exists

https://gerrit.wikimedia.org/r/381493

gerritbot added a comment.Oct 9 2017, 7:58 PM

Change 383185 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] [WIP] Removing from whitelist tables that no longer exist

https://gerrit.wikimedia.org/r/383185

• Nuria added a comment.Oct 9 2017, 8:40 PM

Ping @elukey pagedeletion_7481655_15423246 and pagedeletion_7481655 can now be deleted from MySQL. Onto the next ones.

gerritbot added a comment.Oct 10 2017, 11:32 PM

Change 383485 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] PageCreate events are no longer flowing

https://gerrit.wikimedia.org/r/383485

• Nuria added a comment.Oct 10 2017, 11:33 PM

Ping @elukey more tables that can be deleted from MYSQL:

PageMove_7495717
PageMove_7495717_15423246
PageRestoration_7758372
PageRestoration_7758372_15423246

• Nuria added a comment.Oct 10 2017, 11:36 PM

Also , @elukey we probably need to merge this change https://gerrit.wikimedia.org/r/#/c/383185/ and stopeventlogging script before we remove the tables.

Stashbot subscribed.Oct 11 2017, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-11T15:39:25Z] <elukey> drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002

gerritbot added a comment.Oct 11 2017, 3:56 PM

Change 383185 merged by Elukey:
[operations/puppet@production] Removing from whitelist tables that no longer exist

https://gerrit.wikimedia.org/r/383185

elukey moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Oct 12 2017, 1:47 PM
• Nuria closed this task as Resolved.Oct 12 2017, 6:35 PM
gerritbot added a comment.Feb 6 2018, 11:19 PM

Change 383485 abandoned by Nuria:
PageCreate events are no longer flowing

https://gerrit.wikimedia.org/r/383485

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits