Page MenuHomePhabricator
Create Task
Maniphest T214586

Remove goransm.wdcm_maintable from the Data Lake
Closed, ResolvedPublic
Actions

Description

  • The new WDCM ETL procedures in Pyspark work directly with the goransm.wdcm_clients_wb_entity_usage table, so
  • The wdcm_maintable in goransm, HDFS, WMF Data Lake, is not needed for WDCM or any other related system anymore.

Remove the table completely.

Related Objects

Event Timeline

GoranSMilovanovic created this task.Jan 24 2019, 2:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2019, 2:34 PM
GoranSMilovanovic added a comment.May 30 2019, 8:28 AM
  • WDCM Geo Dashboard does not depend upon this table anymore.
Addshore unsubscribed.May 30 2019, 1:42 PM
GoranSMilovanovic mentioned this in T217994: WDCM Dashboards Maintenance.May 31 2019, 11:12 AM
GoranSMilovanovic moved this task from Current/Deprioritized to Prioritized on the User-GoranSMilovanovic board.Jun 5 2019, 10:59 AM
  • The WDCM Biases Dashboard is the only remaining dashboard whose back-end relies on this Hive table.
  • As soon as the changes are implemented there, the table will be removed from hdfs.
GoranSMilovanovic closed this task as Resolved.Jun 17 2019, 9:12 AM
  • wdcm_maintable removed from hdfs;
  • all WDCM dashboards now running Apache Spark supported update engines;
  • resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL