- Depending on: https://phabricator.wikimedia.org/T171258
- Following the puppetization of the WDCM: run the whole system on regular monthly updates
- Add a timestamp info somewhere in the WDCM dashboards
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | GoranSMilovanovic | T210147 Optimize WDCM update engines | |||
| Resolved | GoranSMilovanovic | T179286 WDCM Regular Updates |
Event Timeline
Comment Actions
Status:
- during the weekend, another (final) manual WDCM update before officially putting the WDCM Engine in production; puppetization in itself is almost done (see T171258#3761115)
- standardizing the update reporting code;
- putting the whole thing on cron stat1005 in production should not be problematic.
Comment Actions
- Following the developments on T210147:
- WDCM main update engine will run on weekly basis,
- synced to start 10 hours after the onset of the Sqoop procedure (i.e. transfer from MariaDB to HDFS),
- on 1st, 7th, 14th, 20th, and 27th each month - so we will have five monthly updates.
With the new procedures and following its scaling to Apache Spark we could run WDCM on daily basis with no trouble at all, except that we cannot do that because the Sqoop transfer from MariaDB (client wbc_entity_usage tables) to HDFS takes hours to complete. I will see if there is anything that I can do to speed it up, but I think the chances are slim.