Page MenuHomePhabricator

Remove goransm.wdcm_maintable from the Data Lake
Closed, ResolvedPublic

Description

  • The new WDCM ETL procedures in Pyspark work directly with the goransm.wdcm_clients_wb_entity_usage table, so
  • The wdcm_maintable in goransm, HDFS, WMF Data Lake, is not needed for WDCM or any other related system anymore.

Remove the table completely.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2019, 2:34 PM
  • WDCM Geo Dashboard does not depend upon this table anymore.
Addshore removed a subscriber: Addshore.May 30 2019, 1:42 PM
  • The WDCM Biases Dashboard is the only remaining dashboard whose back-end relies on this Hive table.
  • As soon as the changes are implemented there, the table will be removed from hdfs.
GoranSMilovanovic closed this task as Resolved.Jun 17 2019, 9:12 AM
  • wdcm_maintable removed from hdfs;
  • all WDCM dashboards now running Apache Spark supported update engines;
  • resolved.