Tue, Oct 12
Better formatting on that query:
Thanks @Pcoombe - it's interesting though that we're not seeing anything sampled at 100% at that time. Here is a hive query to show the issue beginning after the 17th:
Mon, Oct 11
Interestingly, I don't seem to get this error using the pivot table chart type - this could be isolated to the table chart type
Sun, Oct 10
Thu, Oct 7
Thank you! I'm adding @JMando here as a subscriber. We have a few metrics we'd like to add to the event_sanitized.centralnoticebannerhistory table in Superset and it would be great if we both had access to modify this data source.
Wed, Oct 6
Added contact info for @JLaytonWMF to the Collab page.
Tue, Oct 5
Of course, @cchen!
Thu, Sep 30
We should collaborate after code freeze in Q3 with @Cstone to turn logfiles into more usable raw files. Can we follow up by creating a new task on that as a first step? Thank you!
Wed, Sep 29
Thanks, @Eileenmcnaughton this is great! I'm seeing them in wmf_donor now.
Mon, Sep 27
Great! Thank you. I'll be there for the second half of it (I have a conflict for the first 30 minutes).
Hey XenoRyet checking in - would Tuesday or Wednesday this week be a good time to bring this to Tech Talks?
Tue, Sep 21
Thanks, all! This is great information to have. Realizing, of course, that we have no lead in this case, I think it's still informative to add some notes about this experience:
Thanks, @Ejegg, I haven't read through the comment thread thoroughly, but due to this being authored in 2016 and now having much better (entirely new?) systems of tracking this, I'd say we can close this and start afresh. We have most of this through other systems, and I'd be happy to talk to any stakeholders on the ticket + FR-Tech to see if / what is outstanding.
Mon, Sep 20
Hi @Ottomata! Actually that would be super helpful. Would you mind picking anything on my calendar that is open and works for you? I'll remove unnecessary events, and I have my working hours blocked off.
Sep 7 2021
Hi @Ottomata thanks for the ping - this is on my list of quarterly projects, and I've scheduled time out of this week to focus on it in earnest. I've read through the docs and I'll start on step 3 as outlined here this week. I might be pinging back here or on IRC if I run into any questions!
Hi all, just piping in to say thank you for adding this to your queue! There is no urgent need to add this to this quarter's work (and wow, this quarter is nearly done with) but I look forward to using this data when it becomes available.
Sep 3 2021
Ah, nice. I'll go get my vision checked and close out this task. That you for correcting my numerous typos during this setup process.
Hi again! One further question for you all; does @JMando have access to jupyter? The command ssh -N stat1005.eqiad.wmnet -L 8880:127.0.0.1:8880 seems to open a tunnel successfully but he was unable to reach Jupyter at localhost:8000 ("This site can't be reached" error).
Oh yes. That will help. :) Connected, thank you!
Thanks! Error for me when I ssh -v stat1007.eqiad.wmnet:
Sep 2 2021
Hi all! Thank you for working on this and granting access for @JMando! We have been working from the level of access authorized, which you've all granted, to trying to confirm that we can login as expected, and hitting some roadblocks - probably user error on my part, as I'm guessing the instructions I've written up have changed since I last checked.
Yeah I agree with @DStrine. We have a request in for all_funds_last_donated and similar all_funds categories to be implemented that this would fall under.
Aug 30 2021
Aug 25 2021
Thank you for the notes, and absolutely approved! @JMando is a 40-hour / week. long-term contractor in a senior level position with a full confidentiality agreement who is a core member of our team. He will need access to production data on stat100x machines in order to query, model, and analyze data that impacts Fundraising Analytics outcomes, such as pageviews and banner history event collection during Fundraising campaigns.
Aug 16 2021
Adding @JMando to keep track of changes. @Eileenmcnaughton just so I understand it: the current state is that the FY2122 fields are added and in production in the current table (we can see them, and thank you!) but the new fields and new indexing are going to be covered by this ticket?
Aug 11 2021
Chiming in here to clarify a few things on my side - are the 'both_' fields specific to Acoustic? Are they already stored somewhere under this name? If so, what type of effort would be involved in renaming?
Aug 6 2021
Pinging this thread again, as this has come up as an Analytics request to report on RML data:
Jul 19 2021
@Eileenmcnaughton and I talked about this today. In addition to the FY2122 rollup field, I would also be looking for:
- change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - AF only)
- endowment_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - Endowment only)
- foundation_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - combined funds)
- foundation_change_fy1920_fy2021 (or the equivalent change between FY1920 and FY2021 metric - combined funds)
- foundation_change_fy1819_fy1920 (or the equivalent change between FY1819 and FY1920 metric - combined funds)
Update on the immediate use case for this that I was able to pretty easily provide an update for the Email team on who was / was not on the MSL on their necessary list.
@Dwisehaupt Just noting that I'll send Joseph your way today once he's in his Gmail and IRC, etc, to get setup configured. I forgot to mention that he won't have the 'usual' superset permissions but he'll need an Admin account instead of a Data Consumer account. Thanks!
Jul 14 2021
Adding to the list that Joseph will also need a Jupyter instance. I could create a separate task for that if this isn't onboarding-appropriate yet.
Jul 13 2021
Oh, Joseph also has an ITS-provided yubikey and macbook pro laptop.
Joseph's start date: July 19, 2021. Anything we can get access to (like sql / server / Superset / SSH access setup started) would be awesome! Thank you!
Ah, got it. Thanks @Eileenmcnaughton
Jul 12 2021
Makes perfect sense, there are a lot of systems that these would have to go live in. I would just be interested in involvement up to the point of seeing new fields visible in civicrm.wmf_donor - we would use these as soon as they're available (my fault I didn't double check that FY2122 was available already).
Just guessing - probably for the Civi UI update to ensure these fields are visible in the Civi UI, but for use in analytics and reporting, the update to the civicrm.wmf_donor table will get us where we need to be for most use cases, where we can work with this for the Japan campaign and ongoing for 99% of reporting
Hi @Eileenmcnaughton - I was just asking about this today! I'll offer to write a commit for wmf_donor for this and submit it to you as a reviewer, if @DStrine is okay with that, as we use these fields all the time in Analytics. I would also like to propose adding a few additional fields that combine Endowment and Annual Fund totals, as this would cut down on the post-processing I do in our data cubes.
Sorry @Eileenmcnaughton I'm coming back to this now; is the query in prod in the correct format, or was the patch to make the UI more SQL-like, or something different...? Thanks for the help! Either way, it's working now.
Jun 29 2021
@Eileenmcnaughton thanks! This is working in dev but not in prod. Here is the same searchkit reproduced in prod. I'm getting a spinning wheel when I click on 'search' and the results are unexpected: CIDs as old as 1 are showing up in the initial results, which never fully return.
Just a note as I saw this pop up - can we change the task title to New FY2021 Donors to avoid confusion? I think I understand from context that this is to be used in FY2122, but my two cents is that it will be hard to tell in the future what links these donors to FY2122. Thanks!
Jun 22 2021
I'm trying to test this under the Mailing Datas test searchkit and not seeing any results come up when query for 3 mailing_identifiers that were sent out around October - November
Jun 13 2021
The below query pulls a total of 832,108 donors. @LeanneS I realize this is lower than expected: there are a number of US individuals who are first time donors in FY2021 who are opted in (criteria mentioned above) who nonetheless do not have a listed street address, mostly due to matching gifts (at first glance).
Jun 11 2021
Thank you! Yes we can get this done in Q1.
@LeanneS where should the data pull live on the file server once I gather the data? I'll be working on this tomorrow.
May 6 2021
Fair enough, @Ottomata! Is Q1 too late for this self-imposed deadline?
@Ottomata when would this migration need to happen? If we have a deadline, we might be able to prioritize it and help find an owner.
Apr 16 2021
Apr 12 2021
Hi all! I am going to close out this ticket, since the main point - whitelisting the 3 mentioned schemas - has been resolved.
Apr 7 2021
Update for those on this task:
Apr 5 2021
Hi all! @DStrine : The weirdness you're referring to was isolated to mobile devices only. We saw a slight increase in mobile impressions and a simultaneous slight decrease in mobile donations, leading to an overall notable decrease in mobile donation rate. However, it doesn't appear there was a technical bug....that we've identified, yet.
Mar 26 2021
Hi, just pinging back on this ticket. @mforns are the new gerrit review requests looking better?
Mar 18 2021
Super! Thank you both. I have logged in successfully. I'll resolve the task!
Mar 17 2021
Hi @mforns - thanks again for the help and advice earlier this week. I have updated the gerrit reviews with the information we discussed earlier this week. I used the same branches - I hope that is okay? Please let me know if there is a preferred or better way to push changes - always want to learn more! Also, I am not 100% confident that I've spelled everything correctly!
Mar 16 2021
Got it, thanks @Eileenmcnaughton
Mar 10 2021
Hi @mforns getting back to you on this. I'll schedule a meeting for next week with you, myself, @Jdrewniak, @mpopov , and @Pcoombe please feel free to attend as well as you are a user of this data as well and know our needs (and future needs!) so well! I will look for a time.
Mar 9 2021
Thanks again @JAllemandou . This was confusing for me because, like you said, I was only aware of query optimization and it doesn't fit with my understanding of SQL - so I appreciate the information! I will keep this in mind and (if it comes up again) try to reorder and "trick" the execution.
Mar 5 2021
Thank you so much for the quick response, @mforns! I figured that I had missed the preservation window, but did want to ask.
@JAllemandou Phew! Thank you for this explanation, and thank you for adding event_l_card to the table definition. I am please to say that this dash is totally fixed! Thank you for your sustained help through several issues and lots of questions.
Hi all! Independent of the important security questions raised in the gerrit review, I want to ask about the possibility of accessing Portal data from Nov. 30, 2020 onward.
Mar 2 2021
I was thinking about this task the other day, and I don't think that killing running queries after 65 seconds will fix the user experience issue. Because of the way that Superset queues queries, even if everything in the processlist that is being executed by the research user is killed at 65 or 70 seconds, there are still queries waiting to start in the process list that both begin and timeout after the UI says that the query has timed out. Any ideas of how to kill those before they start?
Mar 1 2021
@mforns thanks for linking to T262433! I've added myself as a subscriber. Heads up for @mpopov, as I believe you authored this (?) that we do use this schema in Fundraising, and would like to use it more. Keeping eyes on the Portal is a good health metric for how users interact with Wikipedia, and the same can be said for the WikiApp and WikiAppiOS schemas. We run fundraising campaigns in each of these places, and keeping a close eye on how user behavior changes over time, across several years of fundraising campaigns, is important to ensuring that we are not under-serving a particular user base who sees our campaigns. We also want to ensure that there are no technical issues with our campaigns in a particular region / access method / etc.
Feb 23 2021
Thank you @mforns and apologies that it took me a while to return to this. I've submitted three individual reviews for the 3 schemas we would like to whitelist, and I'm looking forward to review.
Feb 22 2021
Thanks @elukey! I also wanted to ask about caching; it appears that caching is no longer working for this dashboard - do you know of a possible cause for this?
Feb 21 2021
Thanks again for helping with the authentication issue @elukey. I'm sure it's all fixed now, but I also wanted to note that it looks like my email address in Superset is listed as EYener@email.notfound - just in case this is helpful.
Feb 4 2021
@MDemosWMF I'd be interested to hear about other reports you're currently running in Civi UI. This is something I'm always mentioning to folks - if a report doesn't require individual record or donor lookup, it's a great candidate to be brought over to Superset. I'd be happy to hear more about it!
Thanks for the add @MDemosWMF. Is there a need to see individual contributions during this process? If not, I would advocate for keeping the report in Superset, which is great for running aggregated reports and adding / tweaking things as business requirements change. That would also remove the possibility of reports diverging as time goes on.
Jan 29 2021
Thanks, @Pcoombe I added that to the task description to make sure it's captured.
Jan 28 2021
Hi all, I spoke to @JAllemandou today about the issue I am seeing while editing this dashboard:
Jan 27 2021
Hi @JAllemandou thanks for the reply! I am pulling this task back up and opened the dashboard to implement these suggestions. However, I encountered a new error on all charts:
Jan 26 2021
Hmm I'm not a pro on that @KHaggard. Let me circle up with the FR tech folks in our next sync up to see if this is more of a Civi thing, a Superset thing, or an ad hoc request thing and we can get back to you. I'll close this out for now since it seems like it's no longer needed?
@KHaggard I wanted to circle back on this as it came up as a request in our Asana backlog as well.
Jan 19 2021
Jan 16 2021
That would be great @Dwisehaupt, I think MAX_STATEMENT_TIME would be great for the research user (maybe 65 seconds just to be safe?). I just want to be sure - is there a way to check that this user is isolated to Superset queries? I'm almost sure it is, but want to double check. If 'yes,' then this would be a great solution.
Jan 14 2021
Jan 12 2021
Really cool @Eileenmcnaughton & @DStrine . Sorry for belaboring the point....is this 450 day limit related to the created date? Or the date of the action taken? I can see it going either way and just want to make sure I understand. With the example that "if a mailing happened 460 days ago, but the open action happened yesterday" example, it seems like Acoustic would not store that because the storage limit is tied to the create date of 460 days ago - but I want to check.
Awesome, thank you @Eileenmcnaughton !
Jan 11 2021
Jan 7 2021
Hi all! CCing @AndyRussG - do we in Fundraising need IP data for either CentralNoticeImpressions or CentralNoticeBannerHistory? Having the geodata at the state/territory level is important for us for both reporting and diagnostic reasons, but the IP level is perhaps more micro than we need.
Jan 4 2021
@Jgreen would we be able to do this with a private repo?
Dec 21 2020
Dec 18 2020
Dec 11 2020
Sounds good, @KHaggard. What segment was it? We can put a note on any of our reports that this is still pending.
That would be a better question for @KHaggard and team to prioritize in terms of email tasks. I'm not sure how large this send is or when the Email team would want this data back for analysis.
Dec 9 2020
Hey FR-Tech team! I'm not seeing this in the database when I search for it, so I think it might be an import issue from Acoustic.