Page MenuHomePhabricator
Feed Advanced Search

Jan 19 2021

EYener awarded T272390: Upgrade to Superset 1.0 a Party Time token.
Jan 19 2021, 3:42 PM · Analytics-Kanban, Analytics-Clusters

Jan 16 2021

EYener added a comment to T272080: Superset user - kill query after 60-65 seconds.

That would be great @Dwisehaupt, I think MAX_STATEMENT_TIME would be great for the research user (maybe 65 seconds just to be safe?). I just want to be sure - is there a way to check that this user is isolated to Superset queries? I'm almost sure it is, but want to double check. If 'yes,' then this would be a great solution.

Jan 16 2021, 8:59 PM · fundraising-tech-ops, FR-Tech-Analytics, Fundraising-Backlog

Jan 14 2021

EYener created T272080: Superset user - kill query after 60-65 seconds.
Jan 14 2021, 8:22 PM · fundraising-tech-ops, FR-Tech-Analytics, Fundraising-Backlog

Jan 12 2021

EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

Really cool @Eileenmcnaughton & @DStrine . Sorry for belaboring the point....is this 450 day limit related to the created date? Or the date of the action taken? I can see it going either way and just want to make sure I understand. With the example that "if a mailing happened 460 days ago, but the open action happened yesterday" example, it seems like Acoustic would not store that because the storage limit is tied to the create date of 460 days ago - but I want to check.

Jan 12 2021, 10:33 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog
EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

Awesome, thank you @Eileenmcnaughton !

Jan 12 2021, 9:25 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog

Jan 11 2021

EYener created T271714: banner_cube_raw_update.py reconfig sustainability.
Jan 11 2021, 1:38 PM · fundraising-tech-ops, FR-Tech-Analytics, Fundraising-Backlog

Jan 7 2021

EYener added a comment to T271168: CentralNoticeBannerHistory and CentralNoticeImpression Event Platform Migration.

Hi all! CCing @AndyRussG - do we in Fundraising need IP data for either CentralNoticeImpressions or CentralNoticeBannerHistory? Having the geodata at the state/territory level is important for us for both reporting and diagnostic reasons, but the IP level is perhaps more micro than we need.

Jan 7 2021, 8:34 PM · Patch-For-Review, Fundraising-Backlog, Analytics, Event-Platform

Jan 4 2021

EYener added a comment to T271111: adjust gerrit wikimedia/fundraising/analytics project for fundraising deploy use.

@Jgreen would we be able to do this with a private repo?

Jan 4 2021, 3:21 PM · fundraising-tech-ops

Dec 21 2020

EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

@KHaggard No issues here! Whatever @Eileenmcnaughton decides is best for ingesting this data, we can manage on our side. Thanks for checking!

Dec 21 2020, 5:42 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog

Dec 18 2020

EYener created T270503: Presto error in Superest - only when grouping.
Dec 18 2020, 3:57 PM · Analytics-Radar

Dec 11 2020

EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

Sounds good, @KHaggard. What segment was it? We can put a note on any of our reports that this is still pending.

Dec 11 2020, 7:02 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog
EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

That would be a better question for @KHaggard and team to prioritize in terms of email tasks. I'm not sure how large this send is or when the Email team would want this data back for analysis.

Dec 11 2020, 5:56 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog

Dec 9 2020

EYener added a comment to T269811: As an email stats updater, I need to find results for a send that is missing from Superset.

Hey FR-Tech team! I'm not seeing this in the database when I search for it, so I think it might be an import issue from Acoustic.

Dec 9 2020, 11:23 PM · Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, Patch-For-Review, FR-Tech-Analytics, FR-Email, Fundraising-Backlog

Dec 4 2020

EYener added a comment to T269455: Review and close loophole allowing recurring Endowment gifts.

@DStrine Here is most of what you'd want to know about these recurring endow donors:
select * from analytics.all_donations_cube where contribution_recur_id is not null and financial_type = 'Endowment Gift';

Dec 4 2020, 5:34 PM · Fundraising-Backlog, Fr-endowment, Recurring-Donations

Dec 2 2020

EYener updated subscribers of T269303: Error when trying to complete a recurring paypal transaction.
Dec 2 2020, 11:24 PM · Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-Banners
EYener added a comment to T269185: Monitor the return of RML import in December.

@DStrine Any chance we can chat about this during Civi Fortnightly? I'd also like to learn more / hear some details.

Dec 2 2020, 5:58 PM · Fundraising Sprint Zeitgeistbusters, Fundraising Sprint Yellow hornets of kindness and healing, fr-donorservices, Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM

Oct 28 2020

EYener added a comment to T264335: Fundraising access request for Melanie Demos.

After speaking with Melanie and @LeanneS I will be setting Melanie up for Superset access as well

Oct 28 2020, 7:58 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T266599: As an email reporter, I need to know how to pull domain reports in Civi for soft bounces.

Thanks @MNoorWMF ! So it sounds like we should be looking at the enUS segments only. Has this been for Email2 only? Or should we go back to email1?

Oct 28 2020, 1:04 PM · FR-Email, Fundraising-Backlog

Oct 27 2020

EYener updated subscribers of T266599: As an email reporter, I need to know how to pull domain reports in Civi for soft bounces.

Tagging @RMurthy to assist

Oct 27 2020, 9:35 PM · FR-Email, Fundraising-Backlog
EYener added a comment to T266599: As an email reporter, I need to know how to pull domain reports in Civi for soft bounces.

Sure thing we can help with this. Thanks for the ping @Eileenmcnaughton .

Oct 27 2020, 9:24 PM · FR-Email, Fundraising-Backlog

Oct 15 2020

EYener added a comment to T264051: Fundraising access request for Sheyna Daniels Major Gifts.

@RLewis I've added a Superset account for Sheyna as well.

Oct 15 2020, 4:44 PM · fundraising-tech-ops

Sep 9 2020

EYener created T262439: Upgrade Fundraising Superset to 0.37.x.
Sep 9 2020, 4:08 PM · FR-Tech-Analytics, fundraising-tech-ops, Fundraising-Backlog

Sep 4 2020

EYener updated subscribers of T259189: High Unique Clicks for nlNL RML program.

@KHaggard and I discussed this last week (or so) and I wanted to post some highlights here.

Sep 4 2020, 10:11 PM · FR-Netherlands, Fundraising-Backlog
EYener updated subscribers of T256315: Automated Email Import to Civi.

subscribing @spatton for awareness as there was discussion of RML reporting on a call yesterday

Sep 4 2020, 12:22 PM · FR-Email, FR-Tech-Analytics, Fundraising-Backlog

Sep 3 2020

EYener added a comment to T261960: Donors are being removed from Parent Groups by Admin.

To add context, the parent group I am looking at is group_id 701 from civicrm.civicrm_group_contact. It has child groups 685,704,723,724. All members of group 701 seem to be 'deleted'.

Sep 3 2020, 3:44 PM · Fundraising-Backlog

Aug 26 2020

EYener added a comment to T261257: Restrict MailingProvider data table to one row per action.

Cool! The uniqueness check suggested sounds good and beneficial to me. I would like to double check and make sure that it captures the first event occurrence.

Aug 26 2020, 2:12 PM · Fundraising-Backlog

Aug 25 2020

EYener closed T261203: Location of prior analyst folder as Resolved.

Thank you @elukey, I can!

Aug 25 2020, 2:05 PM · Analytics
EYener reopened T261203: Location of prior analyst folder as "Open".
Aug 25 2020, 1:10 PM · Analytics
EYener added a comment to T261203: Location of prior analyst folder.

Thank you @elukey ! Unfortunately, I can't view that task. Would you be able to grant me access to https://phabricator.wikimedia.org/T252364 so that I can potentially plan on future use cases?

Aug 25 2020, 12:51 PM · Analytics
EYener created T261203: Location of prior analyst folder.
Aug 25 2020, 12:40 PM · Analytics

Aug 14 2020

EYener added a comment to T98643: contribution_source triggers failed for 3% of a sample of donations.

Hi all! I found this task because I am experiencing this same problem for the Netherlands banners. As @Pcoombe indicated in the last comment (back in '17!) this seems to be happening with long modifiers. For example, the B2021_0708_nlNL_dsk_p1_lg_txt_twin1_optIn1.no-LP.rtbt.rtbt_ideal utm_source does not join to drupal.contribution_tracking, while the B2021_0708_enNL_m_p1_lg_txt_twin1_optIn1.no-LP.paypal does.

Aug 14 2020, 7:34 PM · Fundraising Sprint UB40, Fundraising-Backlog

Aug 11 2020

EYener added a comment to T259731: Efficiency of querying views.

This query takes 42.7 seconds on last run to execute in Superset:

Aug 11 2020, 11:49 PM · fundraising-tech-ops, FR-Tech-Analytics, Fundraising-Backlog

Aug 5 2020

EYener created T259731: Efficiency of querying views.
Aug 5 2020, 5:18 PM · fundraising-tech-ops, FR-Tech-Analytics, Fundraising-Backlog

Aug 3 2020

EYener added a comment to T256184: Add utm_medium to failed recur email.

Ah cool! Thank you for explaining @DStrine. @CCogdill_WMF your suggestion makes sense to me! One more question, what were the utm_ params before / currently?

Aug 3 2020, 9:24 PM · Fundraising Sprint Pseudopretzels, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener added a comment to T256184: Add utm_medium to failed recur email.

I might need more context here to give input. When does a failed recur email get sent out? What information might we want to report on and analyze? IE, would this affect multiple mediums / campaigns and would we want that level of granularity?

Aug 3 2020, 9:16 PM · Fundraising Sprint Pseudopretzels, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jul 24 2020

EYener closed T256924: Add logging and lockfiles to fr analytics scripts where needed, a subtask of T256420: Determine reason for daily increasing proc count on fran1001, as Resolved.
Jul 24 2020, 2:43 PM · fundraising-tech-ops
EYener closed T256924: Add logging and lockfiles to fr analytics scripts where needed as Resolved.
Jul 24 2020, 2:43 PM · fundraising-tech-ops
EYener added a comment to T256924: Add logging and lockfiles to fr analytics scripts where needed.

Hi @Dwisehaupt I just added the locking scripts to _insert.py files, tested, and they ran just fine so I merged your commit. I think we're good to close this for now and open a new tasks as needed. Thanks again!

Jul 24 2020, 2:43 PM · fundraising-tech-ops

Jul 20 2020

EYener created T258454: Bugfix for get_lock() function on fran1001 inserts.
Jul 20 2020, 10:56 PM · fundraising-tech-ops

Jul 13 2020

EYener added a comment to T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.

@Jgreen I'd like to set up more alerting on frdb1003 for before we make this change so that my data cube testing and runs don't cause the server to lag behind master. Any ideas you have on this - or ideas on how I can leverage existing alerting - would be great!

Jul 13 2020, 12:04 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops

Jul 9 2020

EYener added a project to T255810: Banner history logger records incorrect status code following campaign fallback: FR-Tech-Analytics.
Jul 9 2020, 7:34 PM · Fundraising Sprint 🐍 is not a valid zipcode, MW-1.36-notes (1.36.0-wmf.9; 2020-09-15), Fundraising Sprint Raw data never hurt anyone, Patch-For-Review, fundraising Sprint Quackery limited to ducks, Fundraising Sprint Pseudopretzels, Fundraising Sprint Octopus hugs, FR-Tech-Analytics, MediaWiki-extensions-CentralNotice, Fundraising-Backlog
EYener added a comment to T256924: Add logging and lockfiles to fr analytics scripts where needed.

Added new cube (impressions_hourly) that runs once per week on Thursdays

Jul 9 2020, 6:01 PM · fundraising-tech-ops

Jul 8 2020

EYener added a comment to T256924: Add logging and lockfiles to fr analytics scripts where needed.

@Dwisehaupt logging has been added to:
/home/eyener/analytics_ad_hoc/endowment_recon_update.py
/home/eyener/analytics/email_cube_update.py
/home/eyener/analytics/banner_cube_raw_update.py
/home/eyener/analytics/email_stats_update.py
/home/eyener/analytics/email_components/email_components_update.py

Jul 8 2020, 10:59 PM · fundraising-tech-ops

Jun 30 2020

EYener added a comment to T253152: Q4 FY2019/20 investigate export and upload issues with the silverpop export .

Hi @Eileenmcnaughton I just want to confirm that after the new silverpop job runs, the tables in the silverpop DB will be overwritten with the new fields, correct? Or will the new job write to a separate db / table structure?

Jun 30 2020, 7:46 PM · Fundraising Sprint Pseudopretzels, Fundraising Sprint Octopus hugs, Fundraising Sprint Nyan cats for everyone, Patch-For-Review, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Wikimedia-Fundraising-CiviCRM, FR-Email, Fundraising-Backlog

Jun 24 2020

EYener created T256315: Automated Email Import to Civi.
Jun 24 2020, 8:18 PM · FR-Email, FR-Tech-Analytics, Fundraising-Backlog

Jun 18 2020

EYener added a comment to T255066: Upgrade mariaDB on frdb1003 to >= 10.2.

Hi @Dwisehaupt how about 9PM UTC / 5PM Eastern / 2 PM Pactific on Monday? Alternately, the same time on Tuesday or Thursday would be preferable.

Jun 18 2020, 9:15 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T255066: Upgrade mariaDB on frdb1003 to >= 10.2.

Good question - let me check and get back to you. Thanks @Dwisehaupt

Jun 18 2020, 4:34 PM · fundraising-tech-ops, Fundraising-Backlog
EYener closed T254517: New list pull from Civi as Resolved.

Great, thanks! I will mark this as resolved (I think I can do that...)

Jun 18 2020, 4:05 PM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener updated subscribers of T254517: New list pull from Civi.

The table above is in frdb1003: analytics_ad_hoc.targetsmart_export_june_2020. However, I ended up running a tunnel through R, creating a query, and writing that query output to a CSV on the file server via RStudio. Thanks to @Dwisehaupt for the help figuring out how to connect to a volume from a local application!

Jun 18 2020, 1:26 PM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener added a comment to T252245: Add and delete fields from the _all_Wikimedia database (civi export to ESP).

Hi @Eileenmcnaughton - I'm curious which of these fields you're referring to? What complication do you mean? Asking because I'm also working with similar metrics and would like to brainstorm!

Jun 18 2020, 1:21 PM · Fundraising Sprint Nyan cats for everyone, Patch-For-Review, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Wikimedia-Fundraising-CiviCRM, FR-Email, Fundraising-Backlog

Jun 16 2020

EYener added a comment to T255559: New DB for Superset.

Thank you! It's working great now.

Jun 16 2020, 6:25 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T255559: New DB for Superset.

Hmm. @Jgreen , I'm not seeing it there. I added a table, so the DB is not empty, if that helps.

Jun 16 2020, 3:35 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T255559: New DB for Superset.

Thanks, Jeff!

Jun 16 2020, 3:26 PM · fundraising-tech-ops, Fundraising-Backlog
EYener created T255559: New DB for Superset.
Jun 16 2020, 1:11 PM · fundraising-tech-ops, Fundraising-Backlog

Jun 15 2020

EYener added a comment to T254517: New list pull from Civi.

@LeanneS is able to access the list through Superset. I did an update to make the export fields more human-readable (see below). I'll do another glance through tomorrow to make sure I'm not missing anything.

Jun 15 2020, 10:43 PM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener added a comment to T255066: Upgrade mariaDB on frdb1003 to >= 10.2.

Awesome! Two in one. Thanks!

Jun 15 2020, 8:16 PM · fundraising-tech-ops, Fundraising-Backlog

Jun 10 2020

EYener created T255066: Upgrade mariaDB on frdb1003 to >= 10.2.
Jun 10 2020, 7:30 PM · fundraising-tech-ops, Fundraising-Backlog

Jun 5 2020

EYener added a comment to T254517: New list pull from Civi.

I chatted with @LeanneS about it and it seems that keeping all communication preferences is okay for now (please correct me if this changes!)

Jun 5 2020, 12:35 AM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jun 4 2020

EYener added a comment to T254517: New list pull from Civi.

Adding query for transparency and any future needs:

Jun 4 2020, 10:42 PM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jun 1 2020

EYener reopened T247981: python3 modules for frdb1003, a subtask of T238395: Fundraising Analytics Infrastructure and Setup, as Open.
Jun 1 2020, 12:30 PM · fundraising-tech-ops, Fundraising-Backlog
EYener reopened T247981: python3 modules for frdb1003 as "Open".

@Jgreen could you also add the time module, please?

Jun 1 2020, 12:30 PM · fundraising-tech-ops, Fundraising-Backlog

May 28 2020

EYener updated subscribers of T253062: Deleted contacts and silverpop export questions.

Hi @Eileenmcnaughton - I ran the CIDs (both deleted and kept) above through activities yesterday as a test, and I'm not getting the expected match.

May 28 2020, 4:22 PM · FR-Email, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

May 27 2020

EYener added a comment to T252114: 'Current Day' reporting in Superset.

Hi @Jgreen I actually found a hack for current day (within the UI) by searching Stack Overflow. The time grain supports selecting "today:tomorrow" for current day reporting and I believe there are other work-arounds within the UI for month to date and year to date. It's not intuitive based on their selections but I don't believe this requires a code-level change - just more self-education!

May 27 2020, 8:10 PM · fundraising-tech-ops, Fundraising-Backlog

May 21 2020

EYener added a comment to T252049: Investigate pulling in page view data to the fr-tech version of superset.

To add to the context and background on this, it is often important to in Fundraising to report on impression rate (impressions / pageviews) for a particular campaign to ensure that there are not technical issues during campaigns. Viewing this metric for a live campaign alongside other health metrics in a Fundraising dashboard would be of great value to our Creative team. @AndyRussG had a great example of monitoring impression rates by country during our last Big English - and we were just discussing this use case today - so I will let him discuss in more detail.

May 21 2020, 8:01 PM · fundraising-tech-ops, Fundraising-Backlog

May 18 2020

EYener updated subscribers of T253062: Deleted contacts and silverpop export questions.

Thanks for adding this task @DStrine! Adding @jrobell as as subscriber.

May 18 2020, 8:14 PM · FR-Email, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener created T253050: Bring Banner History data into Fundraising infrastructure.
May 18 2020, 6:19 PM · Analytics-Radar, fundraising-tech-ops, Fundraising-Backlog

May 13 2020

EYener added a comment to T249752: Decomission notebook hosts .

I've deleted my notebooks as well. Thank you!

May 13 2020, 1:59 PM · Analytics-Kanban, Analytics-Clusters, Patch-For-Review

May 8 2020

EYener awarded T252200: CentralNotice banners shouldn't be served to bots a 100 token.
May 8 2020, 3:46 PM · Analytics-Radar, Performance-Team (Radar), Fundraising-Backlog, MediaWiki-extensions-CentralNotice

May 7 2020

spatton awarded T252114: 'Current Day' reporting in Superset a 100 token.
May 7 2020, 1:12 PM · fundraising-tech-ops, Fundraising-Backlog
EYener created T252114: 'Current Day' reporting in Superset.
May 7 2020, 12:15 PM · fundraising-tech-ops, Fundraising-Backlog

May 6 2020

EYener added a comment to T252049: Investigate pulling in page view data to the fr-tech version of superset.

@Jgreen pointed out that we should be able to connect fr-superset to Hive without pulling data into Fundraising from the data lake itself - this might be a great option

May 6 2020, 8:21 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T252034: Superset user bug: password.

@Jgreen what do you think of adding the new ChangeOwnPassword role to Beta and Gamma as well? I can see all new users needing to change their own passwords - it might be good to do this ahead of needing it.

May 6 2020, 8:04 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T251833: Figure out how to check regularly on screen scrape data coming in.

That cadence works for me - thanks!

May 6 2020, 8:02 PM · Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
EYener created T252034: Superset user bug: password.
May 6 2020, 3:57 PM · fundraising-tech-ops, Fundraising-Backlog

Mar 23 2020

EYener added a comment to T248314: SparkR Kernel not starting in Jupyter & JupyterLab.

Hi @elukey! Apologies - yes, notebook1003. This seems to have fixed the problem!

Mar 23 2020, 2:39 PM · Analytics
EYener created T248314: SparkR Kernel not starting in Jupyter & JupyterLab.
Mar 23 2020, 2:29 PM · Analytics

Mar 18 2020

EYener added a comment to T238394: Automation / optimization of data cubes.

@Jgreen we can close this as resolved. At least one data cube is running both inserts and updates on a schedule in the new ecosystem now.

Mar 18 2020, 3:22 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T247980: R packages for frdb1003.

My mistake - removed from the list. Thank you!

Mar 18 2020, 3:21 PM · fundraising-tech-ops, Fundraising-Backlog
EYener updated the task description for T247980: R packages for frdb1003.
Mar 18 2020, 3:21 PM · fundraising-tech-ops, Fundraising-Backlog
EYener created T247981: python3 modules for frdb1003.
Mar 18 2020, 2:40 PM · fundraising-tech-ops, Fundraising-Backlog
EYener added a comment to T247980: R packages for frdb1003.

Very similar, with a few additions, to T236750.

Mar 18 2020, 2:34 PM · fundraising-tech-ops, Fundraising-Backlog
EYener created T247980: R packages for frdb1003.
Mar 18 2020, 2:33 PM · fundraising-tech-ops, Fundraising-Backlog

Mar 17 2020

EYener updated the task description for T238395: Fundraising Analytics Infrastructure and Setup.
Mar 17 2020, 12:36 PM · fundraising-tech-ops, Fundraising-Backlog

Mar 13 2020

EYener added a comment to T245679: Support CSV uploads in Superset.

Thank you, @Nuria ! It works seamlessly.

Mar 13 2020, 10:57 PM · Analytics-Kanban, Analytics
EYener added a comment to T245679: Support CSV uploads in Superset.

Hi @Nuria and all, we're ready to try a 'mock' data set as well. Can someone point me toward instructions on accessing and utilizing the staging environment so that I can get started with the upload? Thank you!

Mar 13 2020, 10:52 PM · Analytics-Kanban, Analytics

Mar 12 2020

EYener created T247582: Timing of active jobs.
Mar 12 2020, 11:45 PM · fundraising-tech-ops, Fundraising-Backlog
EYener created T247581: Silverpop replication in frdb1003.
Mar 12 2020, 11:42 PM · Fundraising-Backlog
EYener updated subscribers of T238394: Automation / optimization of data cubes.

@Jgreen can we merge this task with the ongoing umbrella task for infrastructure setup? https://phabricator.wikimedia.org/T238395

Mar 12 2020, 3:28 PM · fundraising-tech-ops, Fundraising-Backlog

Mar 5 2020

EYener added a comment to T238394: Automation / optimization of data cubes.

Removed test cube from cron and set up a v1 production cube on a 10 minute schedule for inserts only. Next, I'll be working on updates on the whole cube, which would be more resource-intensive.

Mar 5 2020, 12:14 AM · fundraising-tech-ops, Fundraising-Backlog

Feb 25 2020

EYener added a comment to T245755: Install superset on front end server for analytics.

It would be good to touch base, @Milimetric - I'll find time on your calendar for later next week.

Feb 25 2020, 4:25 PM · Analytics-Radar, Patch-For-Review, fundraising-tech-ops, Fundraising-Backlog

Feb 24 2020

EYener added a comment to T245755: Install superset on front end server for analytics.

@Milimetric Likewise excited for collaboration! I agree that visualization is the final piece of this puzzle. In parallel to discussing a front end tool, I have been testing/working on creating and automating (via cron job) OLAP-style data cubes, based on the business units found within Fundraising, that would be able to interact with many visual front ends. I would be happy to discuss the full approach and hear more about best practices at any time!

Feb 24 2020, 7:16 PM · Analytics-Radar, Patch-For-Review, fundraising-tech-ops, Fundraising-Backlog
EYener updated subscribers of T245755: Install superset on front end server for analytics.
Feb 24 2020, 7:14 PM · Analytics-Radar, Patch-For-Review, fundraising-tech-ops, Fundraising-Backlog
EYener updated subscribers of T245755: Install superset on front end server for analytics.

@Jgreen do you have an estimated level of effort on installing Superset on the application server? We are trying to determine if this is something we should scope out and commit to being our visualization tool of choice before requesting. If the level of effort is relatively low, it would be great to have.

Feb 24 2020, 5:00 PM · Analytics-Radar, Patch-For-Review, fundraising-tech-ops, Fundraising-Backlog

Feb 21 2020

EYener added a comment to T238394: Automation / optimization of data cubes.

I have a small test cube on dev_analytics pulling down new donation IDs, contact IDs, and utm_medium on a 10 minute interval via a python3 script in my home directory. I will alter the time table to a 1 hour interval to run overnight (through Saturday, possibly the weekend) for additional timing tests, and pick this up again Monday for more full-scale trial of larger data sets.

Feb 21 2020, 8:14 PM · fundraising-tech-ops, Fundraising-Backlog

Feb 10 2020

EYener updated subscribers of T238395: Fundraising Analytics Infrastructure and Setup.
Feb 10 2020, 3:12 PM · fundraising-tech-ops, Fundraising-Backlog

Feb 6 2020

EYener added a comment to T244484: Issues querying table in Hive.

I'm curious, @Ottomata and @JAllemandou, if there is an elegant solution to dynamically filling partitions. IE, once a table is created with the partition types declared and a main location established, is the best way to fill the table to declare individual ALTER statements by day, hour, etc? Or is there a better way to accomplish this?

Feb 6 2020, 9:00 PM · Analytics
EYener added a comment to T244484: Issues querying table in Hive.

Thanks for the suggestion, @Ottomata - I'll take it back to the team and see what makes sense. Since we've been using the json_string format since 2016, it might make sense to have the 2019 data in the same format as well.

Feb 6 2020, 7:56 PM · Analytics
EYener added a comment to T244292: Database creation in Hive.

Thank you, @JAllemandou! I am new to Hive, and was not aware I could do this myself. It worked well, though, and I appreciate the resources. You can close this ticket; much appreciated.

Feb 6 2020, 6:18 PM · Analytics
EYener added a comment to T244484: Issues querying table in Hive.

Thank you, @JAllemandou! I've learned several new Hive features and commands.

Feb 6 2020, 3:56 PM · Analytics
EYener created T244484: Issues querying table in Hive.
Feb 6 2020, 2:23 PM · Analytics