Page MenuHomePhabricator

Jgreen (Jeff Green)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Nov 25 2014, 1:54 PM (251 w, 5 d)
Availability
Available
IRC Nick
Jeff_Green
LDAP User
Jgreen
MediaWiki User
Jgreen (wmf) [ Global Accounts ]

Recent Activity

Fri, Sep 20

Jgreen closed T233237: For the new data pipeline, purge existing log files as Resolved.

I moved these files to a temporary directory /srv/purge_after_20191031 for now, which is out of the way of log collection, nfs, and log processing. We'll do a final purge sometime after that date.

Fri, Sep 20, 7:33 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T232633: fundraising dev/database access request for Mariana Suijkerbuijk.

Caitlin Cogdill sent an access authorization request to Lisa Gruwell earlier this week, we're waiting to hear back on that.

Fri, Sep 20, 4:43 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T176295: fundraising recurring_gc* queues metrics into prometheus .

They look interesting, but I'm not sure what they all are! Can you share what the queries look like?

Fri, Sep 20, 4:17 PM · observability, fundraising-tech-ops
Jgreen closed T212745: Create read-only user for civicrm devdb on staging as Resolved.

This is done, please reopen if there's any issue with the new mysql user.

Fri, Sep 20, 4:13 PM · FR-Q2-FY2019-20-cleanup-list, fundraising-tech-ops, Fundraising-Backlog
Jgreen updated subscribers of T176295: fundraising recurring_gc* queues metrics into prometheus .

@Ejegg is this something we still need? I know we're collecting a lot of metrics on civi1001 and am not sure if these queues are already included?

Fri, Sep 20, 4:12 PM · observability, fundraising-tech-ops
Jgreen closed T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Declined.
Fri, Sep 20, 4:09 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana as Declined.

Sticking with simple backup of the live host for now...

Fri, Sep 20, 4:09 PM · fundraising-tech-ops
Jgreen added a parent task for T233328: set up cross-host backups between frmon1001 and frmon2001: T91508: [Epic] overhaul fundraising cluster monitoring.
Fri, Sep 20, 4:08 PM · fundraising-tech-ops
Jgreen added a subtask for T91508: [Epic] overhaul fundraising cluster monitoring: T233328: set up cross-host backups between frmon1001 and frmon2001.
Fri, Sep 20, 4:08 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T233328: set up cross-host backups between frmon1001 and frmon2001 as Resolved.

Cross-host snapshot is set to run daily to back up /srv/prometheus.

Fri, Sep 20, 4:06 PM · fundraising-tech-ops

Thu, Sep 19

Jgreen added a comment to T233328: set up cross-host backups between frmon1001 and frmon2001.

commit cf298b3f3085d1862937754d3fa2908e8ba7e971
Author: Jeff Green <jgreen@wikimedia.org>
Date: Thu Sep 19 15:48:07 2019 +0000

Thu, Sep 19, 4:21 PM · fundraising-tech-ops
Jgreen added a subtask for T233328: set up cross-host backups between frmon1001 and frmon2001: Unknown Object (Task).
Thu, Sep 19, 4:20 PM · fundraising-tech-ops
Jgreen created T233328: set up cross-host backups between frmon1001 and frmon2001.
Thu, Sep 19, 4:19 PM · fundraising-tech-ops
Jgreen added a subtask for T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana: Unknown Object (Task).
Thu, Sep 19, 4:15 PM · fundraising-tech-ops
Jgreen added a subtask for T222109: decommission frav1001.frack.eqiad.wmnet: Unknown Object (Task).
Thu, Sep 19, 4:15 PM · decommission, Operations, fundraising-tech-ops, ops-eqiad, DC-Ops

Fri, Sep 13

Jgreen renamed T232633: fundraising dev/database access request for Mariana Suijkerbuijk from Access request - account Yubikey servers to fundraising dev/database access request for Mariana Suijkerbuijk.
Fri, Sep 13, 6:40 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T232633: fundraising dev/database access request for Mariana Suijkerbuijk.

Caitlin Cogdill sent an access authorization request to Lisa Gruwell earlier this week, we're waiting to hear back on that.

Fri, Sep 13, 6:39 PM · fundraising-tech-ops, Fundraising-Backlog

Thu, Sep 12

Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

Yubikey requested from OIT...

Thu, Sep 12, 8:11 PM · fundraising-tech-ops
Jgreen added a comment to T202419: Page fr-tech about spikes in frtechmail.

Reopening, now that we have our own prometheus/grafana instance, would it make sense to alert from it?

Thu, Sep 12, 6:21 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T202419: Page fr-tech about spikes in frtechmail.

is it possible to do this from grafana

Thu, Sep 12, 6:15 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen reopened T202419: Page fr-tech about spikes in frtechmail as "Open".
Thu, Sep 12, 6:10 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

I do not have a yubikey. My server access has only been to Fundraising up to this point. I have the Viscosity VPN set up on my laptop that allows me to connect automatically. Can I continue using this for access?

Thu, Sep 12, 2:54 PM · fundraising-tech-ops
Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

Access approval received.

Thu, Sep 12, 2:52 PM · fundraising-tech-ops
Jgreen added a comment to T232632: fundraising access request for Rosie Lewis.

Access approval received.

Thu, Sep 12, 2:52 PM · Fundraising-Backlog, fundraising-tech-ops

Wed, Sep 11

Jgreen added a comment to T232632: fundraising access request for Rosie Lewis.

Access request sent to Lisa . . .

Wed, Sep 11, 4:42 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T232632: fundraising access request for Rosie Lewis.
Wed, Sep 11, 4:41 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

I sent an access to Lisa Gruwell . . .

Wed, Sep 11, 4:19 PM · fundraising-tech-ops
Jgreen created T232623: fundraising access request for Nora Nichols.
Wed, Sep 11, 4:05 PM · fundraising-tech-ops
Jgreen updated subscribers of T232401: Access to the ssh frdev1001 server .

Leanne and Jerry already have fully set up accounts, so if either of them are having trouble connecting to frdev1001 easiest would be for them to ping @Jgreen or @Dwisehaupt directly to troubleshoot.

Wed, Sep 11, 4:02 PM · fundraising-tech-ops

Tue, Sep 10

Jgreen added a comment to T212745: Create read-only user for civicrm devdb on staging.

I created user dev_civicrm_read with select & create temporary tables on the dev_civicrm database. I'm not sure where this will be configured, so ping me for the password.

Tue, Sep 10, 9:14 PM · FR-Q2-FY2019-20-cleanup-list, fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T232529: dissolve frack unix group cps_data and related bits as Invalid.
Tue, Sep 10, 8:28 PM · fundraising-tech-ops
Jgreen created T232529: dissolve frack unix group cps_data and related bits.
Tue, Sep 10, 8:22 PM · fundraising-tech-ops
Jgreen assigned T232080: Access and onboarding for new fundraising analyst - Jerrie Kumalah to Dwisehaupt.

@jrobell sorry, I didn't manage to get started on this earlier! We're going to have Dallas handle the account setup so he can get familiar with all the moving parts.

Tue, Sep 10, 3:52 PM · fundraising-tech-ops, Fundraising-Backlog

Thu, Sep 5

Jgreen created T232094: civicrm failed outbound connections to https://www-prod.civicrm.osuosl.org.
Thu, Sep 5, 1:22 PM · Fundraising-Backlog

Wed, Sep 4

Jgreen moved T204188: MySQL replication broken on frdev1001 from Backlog to Done on the fundraising-tech-ops board.
Wed, Sep 4, 8:50 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen closed T204188: MySQL replication broken on frdev1001, a subtask of T204187: New logging tables don't pick up our new primary key, as Resolved.
Wed, Sep 4, 8:49 PM · FR-Q2-FY2019-20-cleanup-list, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen closed T204188: MySQL replication broken on frdev1001 as Resolved.

@Jgreen is this ticket still valid?

Wed, Sep 4, 8:49 PM · fundraising-tech-ops, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen created T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana.
Wed, Sep 4, 6:45 PM · fundraising-tech-ops
Jgreen closed T198648: Authentication for grafana, a subtask of T186073: Rack/setup frmon1001, as Resolved.
Wed, Sep 4, 6:43 PM · fundraising-tech-ops, Operations
Jgreen closed T198648: Authentication for grafana as Resolved.
Wed, Sep 4, 6:43 PM · fundraising-tech-ops, Operations
Jgreen added a comment to T198648: Authentication for grafana.

Currently implemented as client certificate authentication is sufficient for no-login read only access, and optional grafana accounts for editors/admins backending in sqlite db. We need to synchronize the sqlite db and other dynamic configuration between frmon1001 and frmon2001, tasking that separately.

Wed, Sep 4, 6:41 PM · fundraising-tech-ops, Operations
Jgreen added a parent task for T198876: Prometheus/Grafana snapshots for long term storage: T185134: Prometheus 2 breaking change.
Wed, Sep 4, 6:38 PM · fundraising-tech-ops
Jgreen removed a parent task for T185013: EPIC: migrate fundraising off of Debian Jessie: T185134: Prometheus 2 breaking change.
Wed, Sep 4, 6:38 PM · fundraising-tech-ops
Jgreen edited subtasks for T185134: Prometheus 2 breaking change, added: T198876: Prometheus/Grafana snapshots for long term storage; removed: T185013: EPIC: migrate fundraising off of Debian Jessie.
Wed, Sep 4, 6:38 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T231249: puppetize and configure fruec banner log parser/db injector script , a subtask of T183978: [Epic] Fundraising kafkatee changes, as Resolved.
Wed, Sep 4, 4:42 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
Jgreen closed T231249: puppetize and configure fruec banner log parser/db injector script as Resolved.
Wed, Sep 4, 4:42 PM · fundraising-tech-ops
Jgreen closed T231372: Onboarding Dallas Wisehaupt to Fundraising Tech Operations as Resolved.
Wed, Sep 4, 2:46 PM · fundraising-tech-ops

Fri, Aug 30

Jgreen moved T231249: puppetize and configure fruec banner log parser/db injector script from Backlog to In Progress on the fundraising-tech-ops board.
Fri, Aug 30, 9:36 PM · fundraising-tech-ops

Thu, Aug 29

Jgreen added a comment to T231249: puppetize and configure fruec banner log parser/db injector script .

frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists

Cool! K, I created that branch! :)

Thu, Aug 29, 3:34 PM · fundraising-tech-ops

Tue, Aug 27

Jgreen closed T231371: Add Dallas Wisehaupt (DWisehaupt) to WMF-NDA and Security phabricator projects, a subtask of T231372: Onboarding Dallas Wisehaupt to Fundraising Tech Operations, as Resolved.
Tue, Aug 27, 8:13 PM · fundraising-tech-ops
Jgreen closed T231371: Add Dallas Wisehaupt (DWisehaupt) to WMF-NDA and Security phabricator projects as Resolved.
Tue, Aug 27, 8:13 PM · fundraising-tech-ops
Jgreen added a member for Security: Dwisehaupt.
Tue, Aug 27, 8:12 PM
Jgreen added a member for WMF-NDA: Dwisehaupt.
Tue, Aug 27, 8:12 PM
Jgreen created T231375: Add Dallas Wisehaupt (DWisehaupt) to acl*WMF-FR phabricator project.
Tue, Aug 27, 7:03 PM · fundraising-tech-ops
Jgreen added a subtask for T231372: Onboarding Dallas Wisehaupt to Fundraising Tech Operations: T231371: Add Dallas Wisehaupt (DWisehaupt) to WMF-NDA and Security phabricator projects.
Tue, Aug 27, 7:00 PM · fundraising-tech-ops
Jgreen added a parent task for T231371: Add Dallas Wisehaupt (DWisehaupt) to WMF-NDA and Security phabricator projects: T231372: Onboarding Dallas Wisehaupt to Fundraising Tech Operations.
Tue, Aug 27, 7:00 PM · fundraising-tech-ops
Jgreen created T231372: Onboarding Dallas Wisehaupt to Fundraising Tech Operations.
Tue, Aug 27, 7:00 PM · fundraising-tech-ops
Jgreen created T231371: Add Dallas Wisehaupt (DWisehaupt) to WMF-NDA and Security phabricator projects.
Tue, Aug 27, 6:58 PM · fundraising-tech-ops
Jgreen added a member for fundraising-tech-ops: Dwisehaupt.
Tue, Aug 27, 6:08 PM
Jgreen removed a member for fundraising-tech-ops: cwdent.
Tue, Aug 27, 6:08 PM
Jgreen added a comment to T231249: puppetize and configure fruec banner log parser/db injector script .

fruec database is created per sql/create_tables.sql

Tue, Aug 27, 3:48 PM · fundraising-tech-ops

Mon, Aug 26

Jgreen updated subscribers of T231249: puppetize and configure fruec banner log parser/db injector script .

@AndyRussG checking in about these configuration settings:

Mon, Aug 26, 9:42 PM · fundraising-tech-ops
Jgreen added a comment to T231249: puppetize and configure fruec banner log parser/db injector script .
  • frdeploy configuration is puppetized and tested, needs to be adjusted to 'deployment' branch once that branch exists
Mon, Aug 26, 9:38 PM · fundraising-tech-ops
Jgreen updated the task description for T231249: puppetize and configure fruec banner log parser/db injector script .
Mon, Aug 26, 7:54 PM · fundraising-tech-ops
Jgreen added a parent task for T231249: puppetize and configure fruec banner log parser/db injector script : T183978: [Epic] Fundraising kafkatee changes.
Mon, Aug 26, 7:52 PM · fundraising-tech-ops
Jgreen added a subtask for T183978: [Epic] Fundraising kafkatee changes: T231249: puppetize and configure fruec banner log parser/db injector script .
Mon, Aug 26, 7:52 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
Jgreen updated the task description for T231249: puppetize and configure fruec banner log parser/db injector script .
Mon, Aug 26, 7:51 PM · fundraising-tech-ops
Jgreen created T231249: puppetize and configure fruec banner log parser/db injector script .
Mon, Aug 26, 7:50 PM · fundraising-tech-ops
Jgreen added a parent task for T224733: investigate recent divot in landingpage log activity: T183978: [Epic] Fundraising kafkatee changes.
Mon, Aug 26, 7:47 PM · Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen added a subtask for T183978: [Epic] Fundraising kafkatee changes: T224733: investigate recent divot in landingpage log activity.
Mon, Aug 26, 7:47 PM · Epic, Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Asymmetrical Earth Theory, Fundraising-Backlog
Jgreen closed T230553: some Fundraising users running macOS Mojave are having trouble with browser client certificates as Resolved.

As far as I can tell this is not an issue with Mojave or even High Sierra. It might be related to anti-malware software or other security settings.

Mon, Aug 26, 2:11 PM · fundraising-tech-ops

Aug 21 2019

Jgreen created T230952: PCI Gap Assessment auditor question about SNMP .
Aug 21 2019, 6:49 PM · netops, Operations

Aug 15 2019

Jgreen updated the task description for T230553: some Fundraising users running macOS Mojave are having trouble with browser client certificates.
Aug 15 2019, 1:42 PM · fundraising-tech-ops
Jgreen triaged T221003: upgrade fundraising databases from Debian Jessie to Stretch as Normal priority.
Aug 15 2019, 1:36 PM · fundraising-tech-ops
Jgreen triaged T211594: reconfigure payments*:/srv to /var/log as Low priority.
Aug 15 2019, 1:36 PM · fundraising-tech-ops
Jgreen triaged T222840: move main fundraising puppet repo to gerrit as Low priority.
Aug 15 2019, 1:36 PM · Gerrit, fundraising-tech-ops
Jgreen triaged T230553: some Fundraising users running macOS Mojave are having trouble with browser client certificates as Normal priority.
Aug 15 2019, 1:35 PM · fundraising-tech-ops
Jgreen moved T230553: some Fundraising users running macOS Mojave are having trouble with browser client certificates from Backlog to In Progress on the fundraising-tech-ops board.

OIT sent me a loaner Mac to test with. I upgraded it to Mojave, created a new user, and tested repeatedly with my cert. All of the issues I ran into had to do with the interface between the browser and login keychain. Safari is really buggy, and at one point corrupted the keychain causing this issue https://discussions.apple.com/thread/4010007. Also I ran into a situation where Safari would simply stall after entering https://civicrm.wikimedia.org, right where com.apple.WebKit.Networking or Safari would normally ask for auth to access the keychain, I think it may have been raising the dialog behind the browser window?

Aug 15 2019, 1:27 PM · fundraising-tech-ops
Jgreen created T230553: some Fundraising users running macOS Mojave are having trouble with browser client certificates.
Aug 15 2019, 1:27 PM · fundraising-tech-ops

Aug 9 2019

Jgreen added a comment to T230223: improve backups from frav1002 to archive.

Spelunked the Debian package, looks like log file encryption was disabled at 7.3.12 and never reenabled, so we'd need to build a custom package to support encryption. Probably not worth it.

Aug 9 2019, 7:13 PM · fundraising-tech-ops
Jgreen moved T230223: improve backups from frav1002 to archive from Backlog to In Progress on the fundraising-tech-ops board.
Aug 9 2019, 6:32 PM · fundraising-tech-ops
Jgreen triaged T230223: improve backups from frav1002 to archive as Normal priority.
Aug 9 2019, 5:47 PM · fundraising-tech-ops
Jgreen closed T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech as Resolved.

prometheus is up and running and collecting, there's still work to be done on the monitoring infrastructure around redundancy and reliability, but it's basically up

Aug 9 2019, 3:42 PM · ops-codfw, Operations, fundraising-tech-ops
Jgreen updated the task description for T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Aug 9 2019, 3:41 PM · ops-codfw, Operations, fundraising-tech-ops

Aug 8 2019

Jgreen moved T228509: puppetize dump_database cron job on fundraising build servers from In Progress to Done on the fundraising-tech-ops board.
Aug 8 2019, 1:32 PM · fundraising-tech-ops
Jgreen moved T228509: puppetize dump_database cron job on fundraising build servers from Backlog to In Progress on the fundraising-tech-ops board.
Aug 8 2019, 1:32 PM · fundraising-tech-ops
Jgreen closed T228509: puppetize dump_database cron job on fundraising build servers as Resolved.

done

Aug 8 2019, 1:32 PM · fundraising-tech-ops

Aug 1 2019

Jgreen closed T229504: frpig* timing out on tools project rsync as Resolved.

This is fixed. I tested a deploy to frpig2001. I did not deploy to frpig1001, in case you should be around to watch behavior when the new code goes out.

Aug 1 2019, 2:36 PM · fundraising-tech-ops
Jgreen closed T228710: Update data on staging, a subtask of T227513: Investigate: TargetSmart Civi import, as Resolved.
Aug 1 2019, 2:06 PM · Fundraising Sprint Office &nbsp;, Fundraising Sprint Never Ending Query, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen closed T228710: Update data on staging as Resolved.

done!

Aug 1 2019, 2:06 PM · Fundraising Sprint Office &nbsp;, fundraising-tech-ops, Fundraising Sprint Never Ending Query, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jul 31 2019

Jgreen closed T229076: Disable new wmf_donor triggers while we make them not lock the db as Resolved.
Jul 31 2019, 8:14 PM · fundraising-tech-ops
Jgreen closed T229439: puppetize install of python-pymysql/python3-pymysql where fundraising-tools code is deployed, a subtask of T213757: Replace MySQL-python with pymysql in fundraising python scripts, as Resolved.
Jul 31 2019, 8:14 PM · Fundraising Sprint Princess Mongodb, Fundraising Sprint Office &nbsp;, Fundraising Sprint Never Ending Query, Fundraising Sprint Men In Slack, Fundraising-Backlog
Jgreen closed T229439: puppetize install of python-pymysql/python3-pymysql where fundraising-tools code is deployed as Resolved.

installed on civicrm and payments-listener roles where the 'tools' project is deployed

Jul 31 2019, 8:14 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen edited projects for T229439: puppetize install of python-pymysql/python3-pymysql where fundraising-tools code is deployed, added: fundraising-tech-ops; removed Fundraising Sprint Office &nbsp;, Fundraising Sprint Never Ending Query, Fundraising Sprint Men In Slack.
Jul 31 2019, 8:13 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen created T229439: puppetize install of python-pymysql/python3-pymysql where fundraising-tools code is deployed.
Jul 31 2019, 3:39 PM · fundraising-tech-ops, Fundraising-Backlog

Jul 30 2019

Jgreen added a comment to T229273: Bug: Trigger caused db locks.

@Jgreen on live there is now a trigger file sites/all/modules/wmf_civicrm/scripts/contribution_triggers.mysql which has the triggers minus the ones I think most likely to be problematic - if you can reload that

Jul 30 2019, 8:52 PM · Fundraising Sprint Rocky Horror Presentation Layer, Fundraising Sprint Quick and the Deadlocked, Fundraising Sprint Princess Mongodb, Fundraising Sprint Office &nbsp;, Unplanned-Sprint-Work, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jul 29 2019

Jgreen triaged T228710: Update data on staging as Normal priority.
Jul 29 2019, 1:39 PM · Fundraising Sprint Office &nbsp;, fundraising-tech-ops, Fundraising Sprint Never Ending Query, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen claimed T228710: Update data on staging.
Jul 29 2019, 1:38 PM · Fundraising Sprint Office &nbsp;, fundraising-tech-ops, Fundraising Sprint Never Ending Query, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog

Jul 26 2019

Jgreen added a comment to T229076: Disable new wmf_donor triggers while we make them not lock the db.

Triggers are removed as of 08:43:23 PM EDT.

Jul 26 2019, 12:48 AM · fundraising-tech-ops

Jul 22 2019

Jgreen claimed T196476: rack/setup/install Prometeuse/Grafana host frmon2001 for fr-tech.
Jul 22 2019, 8:28 PM · ops-codfw, Operations, fundraising-tech-ops