Page MenuHomePhabricator

Add logging and lockfiles to fr analytics scripts where needed
Closed, ResolvedPublic

Description

To keep scripts from stepping on each other or causing performance issues, we need to add a few things to the analytics scripts:

  • logging framwork
  • capture timings for runs
  • use lock files where appropriate to keep script from stepping on each other

Implement these as needed for the fr analytics scripts.

Event Timeline

Change 608967 had a related patch set uploaded (by Dwisehaupt; owner: Dwisehaupt):
[wikimedia/fundraising/analytics@master] Remove print statement that causes cron mails when run

https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/analytics/ /608967

Change 608967 merged by Erin Yener:
[wikimedia/fundraising/analytics@master] Remove print statement that causes cron mails when run

https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/analytics/ /608967

Jgreen moved this task from Triage to Up Next on the fundraising-tech-ops board.Mon, Jul 6, 5:54 PM

@Dwisehaupt logging has been added to:
/home/eyener/analytics_ad_hoc/endowment_recon_update.py
/home/eyener/analytics/email_cube_update.py
/home/eyener/analytics/banner_cube_raw_update.py
/home/eyener/analytics/email_stats_update.py
/home/eyener/analytics/email_components/email_components_update.py

I've also indicated these on the Google sheet re: fran jobs

EYener added a comment.Thu, Jul 9, 6:01 PM

Added new cube (impressions_hourly) that runs once per week on Thursdays

Change 611408 had a related patch set uploaded (by Dwisehaupt; owner: Dwisehaupt):
[wikimedia/fundraising/analytics@master] Add logging and timing to banner_cube_raw_insert

https://gerrit.wikimedia.org/r/611408

Change 611408 merged by Erin Yener:
[wikimedia/fundraising/analytics@master] Add logging and timing to banner_cube_raw_insert

https://gerrit.wikimedia.org/r/611408

@EYener Is there more to do under this umbrella task or should we close it out and address new issues on new tasks?

Hi @Dwisehaupt I just added the locking scripts to _insert.py files, tested, and they ran just fine so I merged your commit. I think we're good to close this for now and open a new tasks as needed. Thanks again!

EYener closed this task as Resolved.Fri, Jul 24, 2:43 PM