Page MenuHomePhabricator

Stop collecting Data for outdated schemas PageCreation, PageDeletion, PageMove, PageRestoration. Archive tables on hdfs
Closed, ResolvedPublic8 Story Points

Description

Stop collecting Data for PageCreation, PageDeletion, PageMove, PageRestoration schemas, archive table on hdfs

This data is now available on mediawiki edit reconstruction and in events coming from EventBus

Lets:

  • stop collecting data for these schemas
  • back up schema data to known location on hdfs (and document that on eventlogging docs/schema talk page)
  • Remove instrumentation that is sending this data

Event Timeline

Nuria created this task.Jul 25 2017, 6:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 25 2017, 6:06 PM
Nuria renamed this task from Stop collecting Data for pageCreate schema, archive table on hdfs to Stop collecting Data for PageCreation schema, archive table on hdfs .Jul 25 2017, 6:10 PM
fdans triaged this task as Normal priority.Jul 27 2017, 3:36 PM
fdans moved this task from Incoming to Operational Excellence Future on the Analytics board.
fdans moved this task from Operational Excellence Future to Dashiki on the Analytics board.
fdans set the point value for this task to 8.
fdans edited subscribers, added: fdans, elukey; removed: Aklapper.
Nuria claimed this task.Sep 12 2017, 11:13 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
Nuria updated the task description. (Show Details)
Nuria changed the point value for this task from 8 to 5.
Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 377667 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Stopping event collection for PageCreation events

https://gerrit.wikimedia.org/r/377667

Change 377667 merged by Ottomata:
[operations/puppet@production] Stopping event collection for Page events

https://gerrit.wikimedia.org/r/377667

Nuria renamed this task from Stop collecting Data for PageCreation schema, archive table on hdfs to Stop collecting Data for PageCreation and other Page* schemas, archive tables on hdfs .Sep 19 2017, 8:30 PM
Nuria renamed this task from Stop collecting Data for PageCreation and other Page* schemas, archive tables on hdfs to Stop collecting Data for PageCreation archive tables on hdfs .Sep 19 2017, 9:11 PM
Nuria renamed this task from Stop collecting Data for PageCreation archive tables on hdfs to Stop collecting Data for PageCreation archive table on hdfs .Sep 19 2017, 9:33 PM

Ok, we are ready to drop PageCreation_7481635_1542324 and PageCreation_7481635 from MySQL ping @elukey
Working now on removing events from mediawiki

Nuria renamed this task from Stop collecting Data for PageCreation archive table on hdfs to Stop collecting Data for outdated schems PageCreation, PageDeletion, PageMove, archive tables on hdfs .Sep 19 2017, 10:32 PM
Nuria renamed this task from Stop collecting Data for outdated schems PageCreation, PageDeletion, PageMove, archive tables on hdfs to Stop collecting Data for outdated schemas PageCreation, PageDeletion, PageMove, PageRestoration. Archive tables on hdfs .
Nuria changed the point value for this task from 5 to 8.
Nuria updated the task description. (Show Details)
Nuria updated the task description. (Show Details)

Change 379137 had a related patch set uploaded (by Nuria; owner: Nuria):
[mediawiki/extensions/WikimediaEvents@master] [WIP] Removing instrumentation for outdated schemas

https://gerrit.wikimedia.org/r/379137

Change 379137 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Removing instrumentation for outdated schemas

https://gerrit.wikimedia.org/r/379137

Change 381493 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Removing schema that no longer exists

https://gerrit.wikimedia.org/r/381493

Change 381493 merged by Ottomata:
[operations/puppet@production] Removing schema that no longer exists

https://gerrit.wikimedia.org/r/381493

Change 383185 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] [WIP] Removing from whitelist tables that no longer exist

https://gerrit.wikimedia.org/r/383185

Nuria added a comment.Oct 9 2017, 8:40 PM

Ping @elukey pagedeletion_7481655_15423246 and pagedeletion_7481655 can now be deleted from MySQL. Onto the next ones.

Change 383485 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] PageCreate events are no longer flowing

https://gerrit.wikimedia.org/r/383485

Ping @elukey more tables that can be deleted from MYSQL:

PageMove_7495717
PageMove_7495717_15423246
PageRestoration_7758372
PageRestoration_7758372_15423246

Also , @elukey we probably need to merge this change https://gerrit.wikimedia.org/r/#/c/383185/ and stopeventlogging script before we remove the tables.

Mentioned in SAL (#wikimedia-operations) [2017-10-11T15:39:25Z] <elukey> drop tables listed in https://phabricator.wikimedia.org/T171629#3674250 from db1046, db1047, dbstore1002

Change 383185 merged by Elukey:
[operations/puppet@production] Removing from whitelist tables that no longer exist

https://gerrit.wikimedia.org/r/383185

elukey moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Oct 12 2017, 1:47 PM
Nuria closed this task as Resolved.Oct 12 2017, 6:35 PM

Change 383485 abandoned by Nuria:
PageCreate events are no longer flowing

https://gerrit.wikimedia.org/r/383485