Page MenuHomePhabricator

Jgreen (Jeff Green)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Nov 25 2014, 1:54 PM (312 w, 6 d)
Availability
Available
IRC Nick
Jeff_Green
LDAP User
Jgreen
MediaWiki User
Jgreen (wmf) [ Global Accounts ]

Recent Activity

Yesterday

Jgreen moved T267969: frdev1001 ILO inaccessible from Triage to Up Next on the fundraising-tech-ops board.
Mon, Nov 23, 8:49 PM · fundraising-tech-ops

Thu, Nov 19

Jgreen claimed T268158: Cron syntax linter for frdeploy tools and/or process-control.

We talked about about a standalone lint script that can be used from git, fundraising_code_update, and process-control. The script would extract the 'schedule' value from a process-control yaml config file and sanity check it. It looks like python3-crontab will be a good start.

Thu, Nov 19, 8:11 PM · fundraising-tech-ops

Wed, Nov 18

Jgreen added a comment to T268158: Cron syntax linter for frdeploy tools and/or process-control.

Because we love regexes . . .

Wed, Nov 18, 6:03 PM · fundraising-tech-ops
Jgreen created T268158: Cron syntax linter for frdeploy tools and/or process-control.
Wed, Nov 18, 6:02 PM · fundraising-tech-ops

Tue, Nov 17

Jgreen closed T245557: break out fundraising outbound mail server to a separate host as Resolved.

frmx1001 & frmx2001 are in service and we're in the process of phasing mail over to them

Tue, Nov 17, 6:39 PM · fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Tue, Nov 17, 5:49 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Tue, Nov 17, 5:45 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Tue, Nov 17, 5:34 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T266057: Upgrade monitoring (frmon) servers to buster, a subtask of T254198: Epic: Upgrade fundraising servers to buster, as Resolved.
Tue, Nov 17, 5:07 PM · fundraising-tech-ops
Jgreen closed T266057: Upgrade monitoring (frmon) servers to buster as Resolved.

frmon1001 was upgraded during yesterday's maintenance window

Tue, Nov 17, 5:07 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Tue, Nov 17, 5:06 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Tue, Nov 17, 5:05 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Tue, Nov 17, 5:05 PM · fundraising-tech-ops
Jgreen closed T257242: frdb1001 hardware refresh, a subtask of T257244: 2020-2021 fundraising hardware refresh and capex, as Resolved.
Tue, Nov 17, 5:04 PM · fundraising-tech-ops
Jgreen closed T257242: frdb1001 hardware refresh as Resolved.

frdb1004 is in service, we'll task the frdb1001 decom separately in Jan 2021

Tue, Nov 17, 5:04 PM · fundraising-tech-ops
Jgreen closed T267944: Add Noah Israel as a recipient of fr-tech-acoustic-failmai as Resolved.

Done!

Tue, Nov 17, 4:31 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen claimed T267944: Add Noah Israel as a recipient of fr-tech-acoustic-failmai.
Tue, Nov 17, 4:02 PM · fundraising-tech-ops, Fundraising-Backlog

Mon, Nov 16

Jgreen created T267969: frdev1001 ILO inaccessible .
Mon, Nov 16, 8:27 PM · fundraising-tech-ops

Tue, Nov 10

Jgreen closed T266287: Upgrade fundraising bastion servers to buster, a subtask of T254198: Epic: Upgrade fundraising servers to buster, as Resolved.
Tue, Nov 10, 1:21 PM · fundraising-tech-ops
Jgreen closed T266287: Upgrade fundraising bastion servers to buster as Resolved.
Tue, Nov 10, 1:21 PM · fundraising-tech-ops
Jgreen renamed T266287: Upgrade fundraising bastion servers to buster from Upgrade frbast servers to buster to Upgrade fundraising bastion servers to buster.
Tue, Nov 10, 1:20 PM · fundraising-tech-ops

Mon, Nov 9

Jgreen moved T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet from In Progress to Done on the fundraising-tech-ops board.
Mon, Nov 9, 4:39 PM · fundraising-tech-ops, Operations
Jgreen closed T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet as Resolved.
Mon, Nov 9, 4:33 PM · fundraising-tech-ops, Operations
Jgreen updated the task description for T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet.
Mon, Nov 9, 4:17 PM · fundraising-tech-ops, Operations
Jgreen moved T267259: Create new job to send Civi thank you emails over smtp through frmx hosts from Up Next to In Progress on the fundraising-tech-ops board.
Mon, Nov 9, 4:02 PM · fundraising-tech-ops, Fundraising-Backlog

Wed, Nov 4

Jgreen moved T267259: Create new job to send Civi thank you emails over smtp through frmx hosts from Triage to Up Next on the fundraising-tech-ops board.
Wed, Nov 4, 8:03 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen updated the task description for T257242: frdb1001 hardware refresh.
Wed, Nov 4, 8:02 PM · fundraising-tech-ops
Jgreen moved T260655: Automate diff and commit of frack ACL from Backlog to Up Next on the fundraising-tech-ops board.
Wed, Nov 4, 7:59 PM · netops, Operations, fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Wed, Nov 4, 6:10 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Wed, Nov 4, 5:46 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Wed, Nov 4, 5:40 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Wed, Nov 4, 5:06 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Wed, Nov 4, 5:05 PM · fundraising-tech-ops

Mon, Nov 2

Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Mon, Nov 2, 8:08 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Mon, Nov 2, 8:07 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Mon, Nov 2, 7:12 PM · fundraising-tech-ops
Jgreen updated the task description for T254198: Epic: Upgrade fundraising servers to buster.
Mon, Nov 2, 7:11 PM · fundraising-tech-ops
Jgreen closed T265688: Upgrade payments_listener servers to Debian Buster as Resolved.

frpig1001 is done

Mon, Nov 2, 6:47 PM · fundraising-tech-ops
Jgreen closed T265688: Upgrade payments_listener servers to Debian Buster, a subtask of T254198: Epic: Upgrade fundraising servers to buster, as Resolved.
Mon, Nov 2, 6:47 PM · fundraising-tech-ops

Fri, Oct 30

Jgreen moved T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet from Triage to In Progress on the fundraising-tech-ops board.
Fri, Oct 30, 6:32 PM · fundraising-tech-ops, Operations
Jgreen added a project to T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet: fundraising-tech-ops.
Fri, Oct 30, 6:32 PM · fundraising-tech-ops, Operations
Jgreen placed T265086: (Need By: ASAP) rack/setup/install frdb1004.frack.eqiad.wmnet up for grabs.
Fri, Oct 30, 6:31 PM · fundraising-tech-ops, Operations
Jgreen added a comment to T264663: fr-tech to look into supporting mail fail over for civi.

I did a burst of one thank you mail from live on SMTP - @Dwisehaupt @Jgreen - is it visible to you? Still on php mail now - I used env per https://phabricator.wikimedia.org/T264663#6566786

Fri, Oct 30, 4:38 PM · Patch-For-Review, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint Upbeat Upholstery, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added a comment to T198876: Prometheus/Grafana snapshots for long term storage.

We are currently storing ~765 days of metrics with our current retention settings. They are currently only present that far back on frmon1001 as frmon2001 wasn't put into service until mid 2019. That level of retention takes up ~75G of disk space and as such is not currently mirrored or archived off the hosts.

Fri, Oct 30, 4:07 PM · fundraising-tech-ops

Thu, Oct 29

Jgreen added a parent task for T266481: (Need By: TBD) rack/setup/install payments100[5-8]: T257244: 2020-2021 fundraising hardware refresh and capex.
Thu, Oct 29, 8:01 PM · ops-eqiad, DC-Ops, Operations
Jgreen added subtasks for T257244: 2020-2021 fundraising hardware refresh and capex: Unknown Object (Task), T266481: (Need By: TBD) rack/setup/install payments100[5-8].
Thu, Oct 29, 8:01 PM · fundraising-tech-ops
Jgreen added a parent task for T266365: (Need By: TBD) rack/setup/install frqueue100[34]: T257244: 2020-2021 fundraising hardware refresh and capex.
Thu, Oct 29, 7:59 PM · Operations, ops-eqiad, DC-Ops
Jgreen added subtasks for T257244: 2020-2021 fundraising hardware refresh and capex: Unknown Object (Task), T266365: (Need By: TBD) rack/setup/install frqueue100[34].
Thu, Oct 29, 7:59 PM · fundraising-tech-ops
Jgreen added a subtask for T257244: 2020-2021 fundraising hardware refresh and capex: Unknown Object (Task).
Thu, Oct 29, 7:59 PM · fundraising-tech-ops
Jgreen moved T202419: Page fr-tech about spikes in frtechmail from Backlog to Done on the fundraising-tech-ops board.
Thu, Oct 29, 5:52 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T202419: Page fr-tech about spikes in frtechmail as Declined.

Doesn't seem to be much interest in this, closing.

Thu, Oct 29, 5:52 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T202419: Page fr-tech about spikes in frtechmail, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Declined.
Thu, Oct 29, 5:51 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T266794: remove fruec code and configuration from servers as Resolved.
Thu, Oct 29, 5:47 PM · fundraising-tech-ops
Jgreen moved T266794: remove fruec code and configuration from servers from Triage to In Progress on the fundraising-tech-ops board.
Thu, Oct 29, 5:06 PM · fundraising-tech-ops
Jgreen closed T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation, a subtask of T95841: Ensure all disaster recover documentation is in one central location, as Resolved.
Thu, Oct 29, 4:58 PM · Operations, fundraising-tech-ops
Jgreen closed T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation, a subtask of T209872: epic: frtech onwiki documentation overhaul, as Resolved.
Thu, Oct 29, 4:58 PM · Epic, fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation as Resolved.

The SRE portion is essentially done: https://collab.wikimedia.org/wiki/Fundraising/Engineering/Emergency_Datacenter_Cutover_Procedure.

Thu, Oct 29, 4:58 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T266810: fr-tech documentation needed for datacenter cutover procedure.
Thu, Oct 29, 4:57 PM · Fundraising-Backlog
Jgreen created T266794: remove fruec code and configuration from servers .
Thu, Oct 29, 3:10 PM · fundraising-tech-ops
Jgreen closed T264819: Investigate impact of buster upgrade on queue processing as Declined.

At this point I don't see a clear indication of a slowdown since the php 7.3 upgrade. As Dallas noted noted above we may be leaving 7.3 performance gains on the table by not enabling OPcache, but we don't think we can do that without sandbox testing.
Task T140311 is already tracking performance tuning issues/ideas.

Thu, Oct 29, 2:48 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen moved T246823: figure out Buster package for python3-mysql.connector for use with fruec from FR-Ops to Triage on the Fundraising-Backlog board.

Removing fundraising-tech-ops tag as this is a software development task. Dallas volunteered a patch, but it has been sitting unreviewed since June.

Thu, Oct 29, 2:26 PM · Patch-For-Review, Fundraising-Backlog
Jgreen removed a project from T246823: figure out Buster package for python3-mysql.connector for use with fruec: fundraising-tech-ops.
Thu, Oct 29, 2:22 PM · Patch-For-Review, Fundraising-Backlog

Wed, Oct 28

Jgreen closed T211594: reconfigure payments*:/srv to /var/log as Resolved.
Wed, Oct 28, 4:16 PM · fundraising-tech-ops
Jgreen updated the task description for T211594: reconfigure payments*:/srv to /var/log.
Wed, Oct 28, 3:57 PM · fundraising-tech-ops
Jgreen reopened T211594: reconfigure payments*:/srv to /var/log as "Open".

Reopening to fix payments100[1-3] since we did a dist-upgrade to Buster instead of a full reimage.

Wed, Oct 28, 3:25 PM · fundraising-tech-ops

Tue, Oct 27

Jgreen added a comment to T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation.

Tossed back into Fundraising-Backlot triage column b/c the next steps are to get FR Tech input to the areas mentioned above.

Tue, Oct 27, 8:59 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Oct 27, 8:58 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T246823: figure out Buster package for python3-mysql.connector for use with fruec from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Oct 27, 8:57 PM · Patch-For-Review, Fundraising-Backlog
Jgreen moved T264819: Investigate impact of buster upgrade on queue processing from FR-Ops to Triage on the Fundraising-Backlog board.
Tue, Oct 27, 8:56 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen triaged T260000: Investigate increase syslog volume on civi1001 after buster upgrade as High priority.

Setting priority to "High" because debug logging like this can become a performance issue during peak FR season.

Tue, Oct 27, 8:55 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T140311: [Epic] Payments performance tuning from Backlog to Watching on the fundraising-tech-ops board.
Tue, Oct 27, 8:42 PM · Epic, MediaWiki-extensions-DonationInterface, Performance Issue, fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T246711: TASK FORCE Investigate parent task "Adjust modsecurity rules to work for us" from Up Next to Backlog on the fundraising-tech-ops board.
Tue, Oct 27, 7:53 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T257242: frdb1001 hardware refresh from Backlog to Up Next on the fundraising-tech-ops board.
Tue, Oct 27, 7:01 PM · fundraising-tech-ops
Jgreen moved T264819: Investigate impact of buster upgrade on queue processing from Up Next to Watching on the fundraising-tech-ops board.
Tue, Oct 27, 6:45 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen added a comment to T264819: Investigate impact of buster upgrade on queue processing.

When doing some quick searches, I noticed many references to enabling the OPcache in php. I noticed that our configs do not currently have opcache enabled for any of the configs (apache/cgi/cli). Not sure if that is by design or not but may be something to look at.

Tue, Oct 27, 6:45 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen moved T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation from Up Next to Backlog on the fundraising-tech-ops board.
Tue, Oct 27, 6:41 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T265688: Upgrade payments_listener servers to Debian Buster from Backlog to Up Next on the fundraising-tech-ops board.
Tue, Oct 27, 6:41 PM · fundraising-tech-ops
Jgreen moved T266287: Upgrade fundraising bastion servers to buster from Backlog to Up Next on the fundraising-tech-ops board.
Tue, Oct 27, 6:41 PM · fundraising-tech-ops
Jgreen updated the task description for T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation.
Tue, Oct 27, 6:32 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T263686: add datacenter and server failover protocol to fundraising disaster protocol documentation from In Progress to Up Next on the fundraising-tech-ops board.

This is mostly done https://docs.google.com/document/d/1oIbiBFDkzIgliF_EaNmEzaUv_y2BW0QN9Fr5uyDKPoY/edit?usp=sharing but needs FR Tech help around the issues of validating redis incrementer (section 7) and migrating and validating queue consumer jobs (section 18).

Tue, Oct 27, 6:30 PM · Fundraising-Backlog, fundraising-tech-ops

Oct 21 2020

Jgreen added a comment to T260621: Test payments-wiki extensions on MediaWiki 1.35.

payments-wiki localsettings tree is updated to match payments-wiki-staging. In the process I noticed payments-wiki-staging/employers.csv is just the stub file, so I did not port that back to payments-wiki.

Oct 21 2020, 7:48 PM · Fundraising Sprint Upbeat Upholstery, fundraising sprint Theme songs for programming languages, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising Sprint Raw data never hurt anyone, Patch-For-Review, fundraising Sprint Quackery limited to ducks, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
Jgreen updated the task description for T256916: Create Grafana dashboard for Online Fundraising to monitor live campaigns .
Oct 21 2020, 7:20 PM · Fundraising-Backlog, Epic, observability, fundraising-tech-ops
Jgreen added a comment to T260621: Test payments-wiki extensions on MediaWiki 1.35.

I updated the frdeploy payments-wiki branch to match payments-wiki-staging (which is what we're currently running on). We still have to clean up the payments-wiki localsettings tree before we can switch over to the payments-wiki frdeploy project.

Oct 21 2020, 6:47 PM · Fundraising Sprint Upbeat Upholstery, fundraising sprint Theme songs for programming languages, Fundraising Sprint 🐍 is not a valid zipcode, Fundraising Sprint Raw data never hurt anyone, Patch-For-Review, fundraising Sprint Quackery limited to ducks, MediaWiki-extensions-DonationInterface, Fundraising-Backlog
Jgreen added a comment to T232386: Permissions to git-update staging civi for select FR-creative members.

I don't think we should broaden access to civicrm staging if it is avoidable.

Oct 21 2020, 6:34 PM · Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops
Jgreen moved T245557: break out fundraising outbound mail server to a separate host from Backlog to In Progress on the fundraising-tech-ops board.
Oct 21 2020, 6:26 PM · fundraising-tech-ops
Jgreen closed Restricted Task, a subtask of T109064: Untranslated Credit Card fields (GlobalCollect iframes), as Declined.
Oct 21 2020, 6:16 PM · Wikimedia-Fundraising
Jgreen closed Restricted Task, a subtask of T93019: Make worldpay forms for Japan, as Declined.
Oct 21 2020, 6:16 PM · Fundraising-Backlog
Jgreen closed Restricted Task, a subtask of T93301: Fundraise in France: October 31 - November 28, as Declined.
Oct 21 2020, 6:16 PM · Wikimedia-Fundraising-Campaigns
Jgreen closed T257245: procure and deploy new frmail server role, a subtask of T257244: 2020-2021 fundraising hardware refresh and capex, as Resolved.
Oct 21 2020, 4:04 PM · fundraising-tech-ops
Jgreen closed T257245: procure and deploy new frmail server role, a subtask of T245557: break out fundraising outbound mail server to a separate host, as Resolved.
Oct 21 2020, 4:04 PM · fundraising-tech-ops
Jgreen closed T257245: procure and deploy new frmail server role as Resolved.

Boxes are up and running and so far appear to deliver mail properly. The ops/sre part of the project is complete. There's a separate task to switch civicrm to use them.

Oct 21 2020, 4:04 PM · fundraising-tech-ops
Jgreen moved T262439: Upgrade Fundraising Superset to 0.37.x from Backlog to Up Next on the fundraising-tech-ops board.
Oct 21 2020, 4:01 PM · FR-Tech-Analytics, fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T260859: dovecot error messages after upgrade to buster as Declined.

This is all interaction of the stock Debian packages (dovecot-core vs systemd-?) packages. It looks like a debug/warning with no actual functionality impact, so I don't see any point in spending the time to fix it.

Oct 21 2020, 4:00 PM · fundraising-tech-ops
Jgreen placed T260183: (Need By: 2020-09-30) rack/setup/install frmx2001.frack.codfw.wmnet, frdata2001.frack.codfw.wmnet up for grabs.
Oct 21 2020, 3:48 PM · fundraising-tech-ops, ops-codfw, DC-Ops, Operations

Oct 20 2020

Jgreen added a comment to T264663: fr-tech to look into supporting mail fail over for civi.

@Eileenmcnaughton I did some testing and the phpmailer functionality seems reasonable. I have a couple observations.

Oct 20 2020, 8:02 PM · Patch-For-Review, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint Upbeat Upholstery, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Oct 19 2020

Jgreen moved T265688: Upgrade payments_listener servers to Debian Buster from Up Next to Backlog on the fundraising-tech-ops board.
Oct 19 2020, 7:09 PM · fundraising-tech-ops
Jgreen moved T265688: Upgrade payments_listener servers to Debian Buster from Triage to Up Next on the fundraising-tech-ops board.
Oct 19 2020, 7:08 PM · fundraising-tech-ops
Jgreen moved T264819: Investigate impact of buster upgrade on queue processing from Triage to Up Next on the fundraising-tech-ops board.
Oct 19 2020, 7:08 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen added a comment to T264819: Investigate impact of buster upgrade on queue processing.

I'm finding it hard to spot a clear trend with so much variation in the graph, also I'm not entirely clear on what the underlying metric is. These queries poked into grafana's explore tool could be useful:

Oct 19 2020, 6:43 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM

Oct 14 2020

Jgreen closed T265162: templatize smashpig main.yaml configuration in fundraising::smashpig as Resolved.
Oct 14 2020, 7:31 PM · fundraising-tech-ops