May 26 2022
May 24 2022
Apr 8 2022
Adding @JMando : Could you please give us a description of how the landing pages should be parameterized so that the data can be easily assessed for this project? Note that there is a seconardy request to determine the split of mobile vs desktop devices on that land on the TY page, though that is not necessary if it adds to the lift.
Apr 5 2022
Thanks @KHaggard : I'd suggest email_greeting. In the UI, it is listed as Email Greeting and in the database it's listed as email_greeting_display so for the export, email_greeting seems like a human-readable term that wouldn't get confused between UI and backend.
Thank you, and thanks to Thea! We do want to keep in coordination with data and import / export requests, but since this one involves obvious PII, we (Analytics) won't have an active role. We don't capture this information on our side. I'll keep an eye on the ticket for progress, and much appreciate the info and being looped in on the request.
Mar 23 2022
Mar 22 2022
Great @Ottomata yes it is in that yaml file!
Thank you @Jdrewniak and @Ottomata! This is a schema that I've worked with before to have the data whitelisted for permanent data capture. Is there anything I'll need to do on my end to ensure that this process continues after the migration?
Mar 17 2022
Hi @Mayakp.wiki, I can add some context here. Below are the campaign dates for the English fundraiser for the last 3 years, though we also want to highlight that it's not our highest priority going into next week to provide exact breakdowns for those days. As we've seen declines in all KPIs for all days of the campaign YoY(oY), the day of year / day of month / day of week between years would be interesting to see but not of urgent need.
Mar 2 2022
Feb 11 2022
Hi all! Pinging @mforns who was helpful with the original whitelist request from phab ticket T273246. I recall that we had some discussion over whether we could keep the referrer data in the sanitized version of this data, which is why we created this new task - to scrub that data.
Feb 3 2022
Group ID = 310 - noting for future
Jan 26 2022
@DStrine to confirm, is this a static Civi group?
Jan 24 2022
Hey @Jgreen yes this will need to be on fran1001 - correct. Thanks for the note on python3 - we will keep using that path!
Jan 23 2022
Jan 10 2022
@DStrine Let's plan to talk about it during out next FR-Tech / Analytics sync up - I think there are nuances here that will be better to talk through. I'll put it on the agenda. Thanks!
@DStrine this task is to look at these elements from the database side - where is this data physically collected on the database and how often do we get it? I don't think it should involve the email team. We know the workflows exist - this task is just to get aligned between FR-Tech and Analytics about the data capture.
Jan 6 2022
A temporary inconvenience @BTullis , thank you for the quick fix! I am back up and running and I'll let @JMando confirm when he gets some time. I can now see the SQL Lab drop down menu as well as run queries.
Dec 28 2021
Thank you, @Ejegg - that is interesting! I'm adding a comment here from a donor ticket, just in case this is helpful and related to this same issue. This was ticket #1071828 in case you have access to / can reference ZD, and per a Slack thread I had about this with others, the issue is "once I select my donation amount and use Apple Pay to check out, nothing happens. There is no further checkout process or submission." Sounds like it might be related, so I wanted to relay this here; as others have noted, it could be completely unrelated or a one-off issue as well.
Dec 23 2021
Here is a query to show the donations which have come in through utm_mediums of 'endowment' but have a financial type of 9 (cash):
Dec 9 2021
Amazing! @razzi Thank you so much! I was trying at this all day and I think what the difference was that it would not respond to my email (eyener@) for recovery, but only to my old contractor email (eyenever-ctr@) Well, something in the docs triggered it so thank you very much. I've logged into both Superset and Gerrit. Closing this out and thank you again!!
Dec 7 2021
Hi all, to circle around from the conversation via email, @JMando and I can take this on. Thanks!
Nov 22 2021
Marking resolved as @BTullis was able to help us get this working. Thank you!
Thank you @BTullis ! Apologies for missing the ownership of event_sanitized. - I tried to edit this just now and it is working. I also tried to add @JMando via his alias (which ended up being just jm ) and was able to do so via the legacy editor. Thank you - we're all set now. I'll mark this resolved as well as the other open task we have.
Nov 19 2021
Nov 9 2021
Thanks @BTullis - we do use Superset's SQL Lab, which is okay at showing a schema once you know the dataset you want to query, but not all we would like a SQL UI to be in terms of showing the whole of available databases and their tables. We can use manual commands in beeline to accomplish this, but a UI is very helpful.
Nov 8 2021
So interesting! Thank you @BTullis, adding @Dwisehaupt to this thread because we were having this same issue - not seeing full user list in dropdown to add them to the dataset owner list - in the Fundraising instance of Superset. @Dwisehaupt looks like I'll just have to use the legacy version until 1.4.0+ is released. Thanks!
Nov 4 2021
That's great to know @odimitrijevic ! We can work with this data in Superset, yes.
Nov 3 2021
Nov 2 2021
Ah sorry, @Eileenmcnaughton can I help clarify more? In broad terms, it would be great if we could create a searchkit and then see a big aggregated total of how many contacts / activities / etc the searchkit contains : There are 14,000 donors in this searchkit (and maybe drill down into that 14,000 to bring up a list of donors).
Oct 28 2021
Oct 27 2021
Oct 26 2021
Declining. Previously, the silverpop tables were used in some of our scripts for an extra layer of dedupe, but (1) dedupe has evolved since this ticket creation and (2) the effort involved by all parties is much higher than the reward to add that added layer of dedupe on raw email addresses.
Oct 25 2021
Oct 22 2021
Oct 21 2021
Oct 11 2021
Interestingly, I don't seem to get this error using the pivot table chart type - this could be isolated to the table chart type
Oct 7 2021
Thank you! I'm adding @JMando here as a subscriber. We have a few metrics we'd like to add to the event_sanitized.centralnoticebannerhistory table in Superset and it would be great if we both had access to modify this data source.
Oct 6 2021
Added contact info for @JLaytonWMF to the Collab page.
Oct 5 2021
Of course, @cchen!
Sep 30 2021
We should collaborate after code freeze in Q3 with @Cstone to turn logfiles into more usable raw files. Can we follow up by creating a new task on that as a first step? Thank you!
Sep 29 2021
Thanks, @Eileenmcnaughton this is great! I'm seeing them in wmf_donor now.
Sep 27 2021
Great! Thank you. I'll be there for the second half of it (I have a conflict for the first 30 minutes).
Hey XenoRyet checking in - would Tuesday or Wednesday this week be a good time to bring this to Tech Talks?
Sep 21 2021
Thanks, @Ejegg, I haven't read through the comment thread thoroughly, but due to this being authored in 2016 and now having much better (entirely new?) systems of tracking this, I'd say we can close this and start afresh. We have most of this through other systems, and I'd be happy to talk to any stakeholders on the ticket + FR-Tech to see if / what is outstanding.
Sep 20 2021
Hi @Ottomata! Actually that would be super helpful. Would you mind picking anything on my calendar that is open and works for you? I'll remove unnecessary events, and I have my working hours blocked off.
Sep 7 2021
Hi @Ottomata thanks for the ping - this is on my list of quarterly projects, and I've scheduled time out of this week to focus on it in earnest. I've read through the docs and I'll start on step 3 as outlined here this week. I might be pinging back here or on IRC if I run into any questions!
Hi all, just piping in to say thank you for adding this to your queue! There is no urgent need to add this to this quarter's work (and wow, this quarter is nearly done with) but I look forward to using this data when it becomes available.
Sep 3 2021
Ah, nice. I'll go get my vision checked and close out this task. That you for correcting my numerous typos during this setup process.
Hi again! One further question for you all; does @JMando have access to jupyter? The command ssh -N stat1005.eqiad.wmnet -L 8880:127.0.0.1:8880 seems to open a tunnel successfully but he was unable to reach Jupyter at localhost:8000 ("This site can't be reached" error).
Oh yes. That will help. :) Connected, thank you!
Thanks! Error for me when I ssh -v stat1007.eqiad.wmnet:
Sep 2 2021
Hi all! Thank you for working on this and granting access for @JMando! We have been working from the level of access authorized, which you've all granted, to trying to confirm that we can login as expected, and hitting some roadblocks - probably user error on my part, as I'm guessing the instructions I've written up have changed since I last checked.
Yeah I agree with @DStrine. We have a request in for all_funds_last_donated and similar all_funds categories to be implemented that this would fall under.
Aug 30 2021
Aug 25 2021
Thank you for the notes, and absolutely approved! @JMando is a 40-hour / week. long-term contractor in a senior level position with a full confidentiality agreement who is a core member of our team. He will need access to production data on stat100x machines in order to query, model, and analyze data that impacts Fundraising Analytics outcomes, such as pageviews and banner history event collection during Fundraising campaigns.
Aug 16 2021
Adding @JMando to keep track of changes. @Eileenmcnaughton just so I understand it: the current state is that the FY2122 fields are added and in production in the current table (we can see them, and thank you!) but the new fields and new indexing are going to be covered by this ticket?
Aug 11 2021
Chiming in here to clarify a few things on my side - are the 'both_' fields specific to Acoustic? Are they already stored somewhere under this name? If so, what type of effort would be involved in renaming?
Aug 6 2021
Pinging this thread again, as this has come up as an Analytics request to report on RML data:
Jul 19 2021
@Eileenmcnaughton and I talked about this today. In addition to the FY2122 rollup field, I would also be looking for:
- change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - AF only)
- endowment_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - Endowment only)
- foundation_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - combined funds)
- foundation_change_fy1920_fy2021 (or the equivalent change between FY1920 and FY2021 metric - combined funds)
- foundation_change_fy1819_fy1920 (or the equivalent change between FY1819 and FY1920 metric - combined funds)
Update on the immediate use case for this that I was able to pretty easily provide an update for the Email team on who was / was not on the MSL on their necessary list.
@Dwisehaupt Just noting that I'll send Joseph your way today once he's in his Gmail and IRC, etc, to get setup configured. I forgot to mention that he won't have the 'usual' superset permissions but he'll need an Admin account instead of a Data Consumer account. Thanks!
Jul 14 2021
Adding to the list that Joseph will also need a Jupyter instance. I could create a separate task for that if this isn't onboarding-appropriate yet.
Jul 13 2021
Oh, Joseph also has an ITS-provided yubikey and macbook pro laptop.
Joseph's start date: July 19, 2021. Anything we can get access to (like sql / server / Superset / SSH access setup started) would be awesome! Thank you!
Ah, got it. Thanks @Eileenmcnaughton
Jul 12 2021
Makes perfect sense, there are a lot of systems that these would have to go live in. I would just be interested in involvement up to the point of seeing new fields visible in civicrm.wmf_donor - we would use these as soon as they're available (my fault I didn't double check that FY2122 was available already).
Just guessing - probably for the Civi UI update to ensure these fields are visible in the Civi UI, but for use in analytics and reporting, the update to the civicrm.wmf_donor table will get us where we need to be for most use cases, where we can work with this for the Japan campaign and ongoing for 99% of reporting
Hi @Eileenmcnaughton - I was just asking about this today! I'll offer to write a commit for wmf_donor for this and submit it to you as a reviewer, if @DStrine is okay with that, as we use these fields all the time in Analytics. I would also like to propose adding a few additional fields that combine Endowment and Annual Fund totals, as this would cut down on the post-processing I do in our data cubes.
Sorry @Eileenmcnaughton I'm coming back to this now; is the query in prod in the correct format, or was the patch to make the UI more SQL-like, or something different...? Thanks for the help! Either way, it's working now.
Jun 29 2021
@Eileenmcnaughton thanks! This is working in dev but not in prod. Here is the same searchkit reproduced in prod. I'm getting a spinning wheel when I click on 'search' and the results are unexpected: CIDs as old as 1 are showing up in the initial results, which never fully return.
Just a note as I saw this pop up - can we change the task title to New FY2021 Donors to avoid confusion? I think I understand from context that this is to be used in FY2122, but my two cents is that it will be hard to tell in the future what links these donors to FY2122. Thanks!
Jun 22 2021
I'm trying to test this under the Mailing Datas test searchkit and not seeing any results come up when query for 3 mailing_identifiers that were sent out around October - November
Jun 13 2021
The below query pulls a total of 832,108 donors. @LeanneS I realize this is lower than expected: there are a number of US individuals who are first time donors in FY2021 who are opted in (criteria mentioned above) who nonetheless do not have a listed street address, mostly due to matching gifts (at first glance).
Jun 11 2021
Thank you! Yes we can get this done in Q1.
@LeanneS where should the data pull live on the file server once I gather the data? I'll be working on this tomorrow.
May 6 2021
Fair enough, @Ottomata! Is Q1 too late for this self-imposed deadline?
@Ottomata when would this migration need to happen? If we have a deadline, we might be able to prioritize it and help find an owner.