Page MenuHomePhabricator

Jgreen (Jeff Green)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Nov 25 2014, 1:54 PM (260 w, 14 h)
Availability
Available
IRC Nick
Jeff_Green
LDAP User
Jgreen
MediaWiki User
Jgreen (wmf) [ Global Accounts ]

Recent Activity

Fri, Nov 15

Jgreen renamed T232137: rack/setup/install frnetmon1001.frack.eqiad.wmnet from rack/setup/install frnetmon1001 to rack/setup/install frnetmon1001.frack.eqiad.wmnet.
Fri, Nov 15, 6:54 PM · fundraising-tech-ops, Operations, ops-eqiad
Jgreen renamed T234069: rack/setup/install frban2001.frack.codfw.wmnet from rack/setup/install frban2001.codfw.wmnet to rack/setup/install frban2001.frack.codfw.wmnet.
Fri, Nov 15, 6:54 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen closed T221116: Investigate r-cran-shiny and/or RStudio shiny-server for Fundraising team use. as Declined.

Closing this, there's no longer a push for this specific direction.

Fri, Nov 15, 6:43 PM · fundraising-tech-ops
Jgreen added a comment to T238395: Analytics Infrastructure and Setup.

From an Ops/SRE perspective we're thinking the architecture should be a separate application server (like T237442) vs database server (like T237437).

Fri, Nov 15, 5:36 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T238392: Modules for python3 on frdev1001.

@Jgreen can you install SQLAlchemy as well, please?
Apologies that this ask is coming more piecemeal than the R packages request.

Fri, Nov 15, 4:21 PM · fundraising-tech-ops
Jgreen closed T238392: Modules for python3 on frdev1001 as Resolved.

Done!

Fri, Nov 15, 1:26 PM · fundraising-tech-ops

Thu, Nov 14

Jgreen added a subtask for T186550: Anycast recdns: Unknown Object (Task).
Thu, Nov 14, 9:26 PM · Patch-For-Review, netops, Operations, Traffic

Wed, Nov 13

Jgreen reassigned T238233: decommission alnilam.frack.codfw.wmnet from Jgreen to Papaul.
Wed, Nov 13, 6:50 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Jgreen updated the task description for T238233: decommission alnilam.frack.codfw.wmnet.
Wed, Nov 13, 6:50 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Jgreen updated the task description for T238233: decommission alnilam.frack.codfw.wmnet.
Wed, Nov 13, 6:50 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Jgreen updated the task description for T238233: decommission alnilam.frack.codfw.wmnet.
Wed, Nov 13, 5:17 PM · fundraising-tech-ops, Operations, DC-Ops, decommission
Jgreen created T238233: decommission alnilam.frack.codfw.wmnet.
Wed, Nov 13, 4:43 PM · fundraising-tech-ops, Operations, DC-Ops, decommission

Thu, Nov 7

Jgreen added a comment to T237582: frqueue1001 system battery needs replacement.
Thu, Nov 7, 5:51 PM · ops-eqiad, Operations
Jgreen added a comment to T237648: investigate remote logging or polling of iDRAC and ILO for hardware issues .

Note an enterprise license is required for iDRAC, afaict this was included in recent server orders.

Thu, Nov 7, 4:07 PM · fundraising-tech-ops
Jgreen reassigned T236739: frdb1001 has suffered a raid event resulting in /dev/sda going read only from Jclark-ctr to Dwisehaupt.

Reassigning to @Dwisehaupt since he's done the heavy lifting on the reimage/recommissions.

Thu, Nov 7, 4:04 PM · fundraising-tech-ops, Fundraising-Backlog, DC-Ops
Jgreen triaged T237648: investigate remote logging or polling of iDRAC and ILO for hardware issues as Normal priority.
Thu, Nov 7, 3:59 PM · fundraising-tech-ops
Jgreen closed Unknown Object (Task), a subtask of T236739: frdb1001 has suffered a raid event resulting in /dev/sda going read only, as Resolved.
Thu, Nov 7, 3:54 PM · fundraising-tech-ops, Fundraising-Backlog, DC-Ops
Jgreen added a comment to T237582: frqueue1001 system battery needs replacement.

@Jgreen - looks like the warranty ended for the server a few months ago in May. Let me know if you're looking to decommission this server soon or if you would like us to purchase the replacement part.

Thu, Nov 7, 3:01 PM · ops-eqiad, Operations

Wed, Nov 6

Jgreen created T237582: frqueue1001 system battery needs replacement.
Wed, Nov 6, 9:42 PM · ops-eqiad, Operations
Jgreen raised the priority of T235676: dwisehaupt needs access to iginca for frack hosts from Normal to Needs Triage.
Wed, Nov 6, 4:45 PM · LDAP-Access-Requests, Icinga, Operations

Wed, Oct 30

Jgreen removed a subtask for T221008: upgrade fundraising queue servers from Debian Jessie: T216633: Redis queues should use a key prefix.
Wed, Oct 30, 8:15 PM · fundraising-tech-ops
Jgreen removed a parent task for T216633: Redis queues should use a key prefix: T221008: upgrade fundraising queue servers from Debian Jessie.
Wed, Oct 30, 8:15 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen removed a project from T235676: dwisehaupt needs access to iginca for frack hosts: fundraising-tech-ops.
Wed, Oct 30, 8:12 PM · LDAP-Access-Requests, Icinga, Operations
Jgreen closed T236750: R packages to be added to frdev1001 server as Resolved.

Done!

Wed, Oct 30, 7:17 PM · fundraising-tech-ops
Jgreen moved T236750: R packages to be added to frdev1001 server from Backlog to In Progress on the fundraising-tech-ops board.
Wed, Oct 30, 7:06 PM · fundraising-tech-ops
Jgreen edited projects for T236750: R packages to be added to frdev1001 server, added: fundraising-tech-ops; removed Operations.
Wed, Oct 30, 7:06 PM · fundraising-tech-ops
Jgreen added a comment to T236739: frdb1001 has suffered a raid event resulting in /dev/sda going read only.

In addition to the repair, we're looking at adding another db system to the cluster for capacity/redundancy expansion. See T236920

Wed, Oct 30, 3:46 PM · fundraising-tech-ops, Fundraising-Backlog, DC-Ops

Tue, Oct 29

Jgreen closed T212252: Reconfigure fundraising check_endpoints, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Resolved.
Tue, Oct 29, 8:11 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T212252: Reconfigure fundraising check_endpoints as Resolved.

Got api.paypal.com working with client certs on payments and civi. Leaving the ingenico check as a pass with 404, since it at least it shows us the endpoint is online.

Tue, Oct 29, 8:11 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T212252: Reconfigure fundraising check_endpoints, a subtask of T207511: Sort out fr-tech work phone situation, as Resolved.
Tue, Oct 29, 8:11 PM · observability, Fundraising-Backlog
Jgreen triaged T212252: Reconfigure fundraising check_endpoints as Normal priority.
Tue, Oct 29, 5:55 PM · Fundraising-Backlog, fundraising-tech-ops

Mon, Oct 28

Jgreen closed T221003: upgrade fundraising databases from Debian Jessie to Stretch as Resolved.

done when bringing frdb1001 back from RAID system crash

Mon, Oct 28, 11:43 PM · fundraising-tech-ops
Jgreen closed T221003: upgrade fundraising databases from Debian Jessie to Stretch, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Mon, Oct 28, 11:43 PM · fundraising-tech-ops

Tue, Oct 22

Jgreen added a comment to T236096: Cannot add or change anything on the prospect tab.

@DStrine I've upped this to 'unbreak now' as there is something actively broken. We discussed on IRC & it seemed like it didn't merit disturbing Jeff or Dallas after hours but ideally first thing when they start their day . I think the file to load should be safe enough to 'just do'

Tue, Oct 22, 3:47 PM · Unplanned-Sprint-Work, Fundraising Sprint Usual Subscripts, FR-Civi-Prospect, Fundraising-Backlog

Oct 18 2019

Jgreen closed T165393: reimage americium to Debian Stretch, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Declined.
Oct 18 2019, 2:53 PM · fundraising-tech-ops
Jgreen closed T165393: reimage americium to Debian Stretch as Declined.
Oct 18 2019, 2:53 PM · fundraising-tech-ops
Jgreen closed T230223: improve backups from frav1002 to archive as Resolved.

Used compresscmd to encrpyt/compress at log rotation, and a separate cron job to sweep those to /srv/archive/logs where they're picked up a day later by archive_sync and stored on the logger/archive hosts.

Oct 18 2019, 2:51 PM · fundraising-tech-ops

Oct 17 2019

Jgreen closed T232623: fundraising access request for Nora Nichols, a subtask of T232401: Access to the ssh frdev1001 server , as Resolved.
Oct 17 2019, 6:30 PM · fundraising-tech-ops
Jgreen closed T232623: fundraising access request for Nora Nichols as Resolved.

Closing this task, what remains is mostly user-side stuff

Oct 17 2019, 6:30 PM · fundraising-tech-ops

Oct 11 2019

Jgreen added a comment to T234992: New credential for Engage: Naomi K.

cert and password sent

Oct 11 2019, 5:49 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen closed T234992: New credential for Engage: Naomi K as Resolved.
Oct 11 2019, 5:49 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen claimed T234992: New credential for Engage: Naomi K.

Received authorization from Lisa Gruwell - Date: Thu, 10 Oct 2019 14:52:52 -0700

Oct 11 2019, 5:36 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added a comment to T232630: rack/setup/install frqueue2001.
  • bonded ethernet configuration done
  • redis replication appears to be working now that firewall policy is deployed
  • added to icinga
Oct 11 2019, 5:35 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen updated the task description for T232630: rack/setup/install frqueue2001.
Oct 11 2019, 5:18 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen closed Restricted Task, a subtask of T232630: rack/setup/install frqueue2001, as Resolved.
Oct 11 2019, 5:14 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen closed Restricted Task, a subtask of T232137: rack/setup/install frnetmon1001.frack.eqiad.wmnet, as Resolved.
Oct 11 2019, 5:14 PM · fundraising-tech-ops, Operations, ops-eqiad
Jgreen closed Restricted Task, a subtask of T234068: rack/setup/install frban1001.eqiad.wmnet, as Resolved.
Oct 11 2019, 5:14 PM · Operations, fundraising-tech-ops, ops-eqiad
Jgreen closed Restricted Task, a subtask of T234069: rack/setup/install frban2001.frack.codfw.wmnet, as Resolved.
Oct 11 2019, 5:14 PM · Operations, fundraising-tech-ops, ops-codfw

Oct 10 2019

Jgreen moved T234992: New credential for Engage: Naomi K from Backlog to In Progress on the fundraising-tech-ops board.
Oct 10 2019, 8:22 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added a project to T234992: New credential for Engage: Naomi K: fundraising-tech-ops.
Oct 10 2019, 8:21 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen triaged T234992: New credential for Engage: Naomi K as Normal priority.
Oct 10 2019, 8:21 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen added a comment to T234992: New credential for Engage: Naomi K.

@LeanneS she should have a civi email now but I assume you need @Jgreen to do the certificate & he'll need an email confirmation from Lisa I guess

Oct 10 2019, 8:21 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM, Fundraising-Backlog
Jgreen triaged T232623: fundraising access request for Nora Nichols as Normal priority.
Oct 10 2019, 5:13 PM · fundraising-tech-ops
Jgreen closed T232401: Access to the ssh frdev1001 server as Resolved.
Oct 10 2019, 5:12 PM · fundraising-tech-ops
Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

yes thank you!

Oct 10 2019, 5:12 PM · fundraising-tech-ops
Jgreen added a comment to T212252: Reconfigure fundraising check_endpoints.

@Ejegg this is mostly done, with some notes:

Oct 10 2019, 4:44 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen renamed T212252: Reconfigure fundraising check_endpoints from Create icinga alert for connectivity to Ingenico Connect endpoint to Reconfigure fundraising check_endpoints.
Oct 10 2019, 3:30 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T235171: test database for fundraising analytics queries from In Progress to Done on the fundraising-tech-ops board.
Oct 10 2019, 2:11 PM · fundraising-tech-ops
Jgreen closed T235171: test database for fundraising analytics queries as Resolved.
Oct 10 2019, 2:10 PM · fundraising-tech-ops
Jgreen added a comment to T235171: test database for fundraising analytics queries.

I'm going to backup and repurpose dev_analytics for this, which hasn't been touched since 2017.

Oct 10 2019, 1:43 PM · fundraising-tech-ops
Jgreen triaged T235171: test database for fundraising analytics queries as Normal priority.
Oct 10 2019, 1:27 PM · fundraising-tech-ops
Jgreen created T235171: test database for fundraising analytics queries.
Oct 10 2019, 1:27 PM · fundraising-tech-ops

Oct 8 2019

Jgreen closed T185134: Prometheus 2 breaking change, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Declined.
Oct 8 2019, 3:33 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T185134: Prometheus 2 breaking change as Declined.

There's not really anything to be done here. We've archived the v1 historical data, and could, in theory, spin up a v1 instance to feed it to grafana, but it's just not worth the effort. It's a good reminder not to think of prometheus as a store for historical data, and to consider alternatives.

Oct 8 2019, 3:33 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T234918: set up prometheus server snapshots for backups.
Oct 8 2019, 12:46 PM · fundraising-tech-ops
Jgreen added a parent task for T202419: Page fr-tech about spikes in frtechmail: T91508: [Epic] overhaul fundraising cluster monitoring.
Oct 8 2019, 12:45 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a parent task for T207511: Sort out fr-tech work phone situation: T91508: [Epic] overhaul fundraising cluster monitoring.
Oct 8 2019, 12:45 PM · observability, Fundraising-Backlog
Jgreen added a parent task for T212252: Reconfigure fundraising check_endpoints: T91508: [Epic] overhaul fundraising cluster monitoring.
Oct 8 2019, 12:45 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added subtasks for T91508: [Epic] overhaul fundraising cluster monitoring: T202419: Page fr-tech about spikes in frtechmail, T212252: Reconfigure fundraising check_endpoints, T207511: Sort out fr-tech work phone situation.
Oct 8 2019, 12:45 PM · Epic, observability, fundraising-tech-ops

Oct 7 2019

Jgreen added a subtask for T232630: rack/setup/install frqueue2001: Unknown Object (Task).
Oct 7 2019, 5:26 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen added a subtask for T234068: rack/setup/install frban1001.eqiad.wmnet: Unknown Object (Task).
Oct 7 2019, 5:26 PM · Operations, fundraising-tech-ops, ops-eqiad
Jgreen added a subtask for T234069: rack/setup/install frban2001.frack.codfw.wmnet: Unknown Object (Task).
Oct 7 2019, 5:25 PM · Operations, fundraising-tech-ops, ops-codfw
Jgreen added a subtask for T232137: rack/setup/install frnetmon1001.frack.eqiad.wmnet: Unknown Object (Task).
Oct 7 2019, 5:25 PM · fundraising-tech-ops, ops-eqiad, Operations
Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

@NNichols circling back on this task, did you ever receive your yubikey?

Oct 7 2019, 4:55 PM · fundraising-tech-ops
Jgreen closed T234592: check_log_messages may not be correctly setting or checking it's lock file as Resolved.

Working correctly now.

Oct 7 2019, 4:52 PM · Fundraising-Backlog, fundraising-tech-ops

Oct 4 2019

Jgreen claimed T234592: check_log_messages may not be correctly setting or checking it's lock file.

Ah HA! Both scripts indeed suffered the same bug and were thus colliding. Both are fixed, leaving the task open until we're sure they're running successfully.

Oct 4 2019, 8:51 PM · Fundraising-Backlog, fundraising-tech-ops

Oct 2 2019

Jgreen moved T212252: Reconfigure fundraising check_endpoints from Backlog to In Progress on the fundraising-tech-ops board.
Oct 2 2019, 7:49 PM · Fundraising-Backlog, fundraising-tech-ops

Oct 1 2019

Jgreen claimed T212252: Reconfigure fundraising check_endpoints.
Oct 1 2019, 1:10 PM · Fundraising-Backlog, fundraising-tech-ops

Sep 30 2019

Jgreen added a comment to T212252: Reconfigure fundraising check_endpoints.

We already get a lot of middle-of-the-night alerts about hiccups in endpoint connectivity, regardless of whether banners are up or whether we're currently using a particular endpoint. Can we explore other ways to do this? Random ideas:

  • hook into the payments-wiki config to limit 'critical' alerts to endpoints we're actively using
  • have payments-wiki trigger an alert based on feedback from the client re. success in loading the iframe
  • don't notify unless there is payments traffic happening
Sep 30 2019, 2:41 PM · Fundraising-Backlog, fundraising-tech-ops

Sep 24 2019

Jgreen triaged T233672: possible routing issue between eqiad and Maxmind network as Unbreak Now! priority.

Flipping this to "Unbreak Now!" since it's a timely issue, and service outage interfering with the donation pipeline. We do have some donation activity at the moment.

Sep 24 2019, 1:12 AM · Operations, fundraising-tech-ops, netops
Jgreen created T233672: possible routing issue between eqiad and Maxmind network.
Sep 24 2019, 1:08 AM · Operations, fundraising-tech-ops, netops

Sep 20 2019

Jgreen closed T233237: For the new data pipeline, purge existing log files as Resolved.

I moved these files to a temporary directory /srv/purge_after_20191031 for now, which is out of the way of log collection, nfs, and log processing. We'll do a final purge sometime after that date.

Sep 20 2019, 7:33 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T232633: fundraising dev/database access request for Mariana Suijkerbuijk.

Caitlin Cogdill sent an access authorization request to Lisa Gruwell earlier this week, we're waiting to hear back on that.

Sep 20 2019, 4:43 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T176295: fundraising recurring_gc* queues metrics into prometheus .

They look interesting, but I'm not sure what they all are! Can you share what the queries look like?

Sep 20 2019, 4:17 PM · observability, fundraising-tech-ops
Jgreen closed T212745: Create read-only user for civicrm devdb on staging as Resolved.

This is done, please reopen if there's any issue with the new mysql user.

Sep 20 2019, 4:13 PM · FR-Q2-FY2019-20-cleanup-list, fundraising-tech-ops, Fundraising-Backlog
Jgreen updated subscribers of T176295: fundraising recurring_gc* queues metrics into prometheus .

@Ejegg is this something we still need? I know we're collecting a lot of metrics on civi1001 and am not sure if these queues are already included?

Sep 20 2019, 4:12 PM · observability, fundraising-tech-ops
Jgreen closed T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Declined.
Sep 20 2019, 4:09 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana as Declined.

Sticking with simple backup of the live host for now...

Sep 20 2019, 4:09 PM · fundraising-tech-ops
Jgreen added a parent task for T233328: set up cross-host backups between frmon1001 and frmon2001: T91508: [Epic] overhaul fundraising cluster monitoring.
Sep 20 2019, 4:08 PM · fundraising-tech-ops
Jgreen added a subtask for T91508: [Epic] overhaul fundraising cluster monitoring: T233328: set up cross-host backups between frmon1001 and frmon2001.
Sep 20 2019, 4:08 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T233328: set up cross-host backups between frmon1001 and frmon2001 as Resolved.

Cross-host snapshot is set to run daily to back up /srv/prometheus.

Sep 20 2019, 4:06 PM · fundraising-tech-ops

Sep 19 2019

Jgreen added a comment to T233328: set up cross-host backups between frmon1001 and frmon2001.

commit cf298b3f3085d1862937754d3fa2908e8ba7e971
Author: Jeff Green <jgreen@wikimedia.org>
Date: Thu Sep 19 15:48:07 2019 +0000

Sep 19 2019, 4:21 PM · fundraising-tech-ops
Jgreen added a subtask for T233328: set up cross-host backups between frmon1001 and frmon2001: Unknown Object (Task).
Sep 19 2019, 4:20 PM · fundraising-tech-ops
Jgreen created T233328: set up cross-host backups between frmon1001 and frmon2001.
Sep 19 2019, 4:19 PM · fundraising-tech-ops
Jgreen added a subtask for T232029: synchronize frmon1001:/var/lib/grafana to frmon2001:/var/lib/grafana: Unknown Object (Task).
Sep 19 2019, 4:15 PM · fundraising-tech-ops
Jgreen added a subtask for T222109: decommission frav1001.frack.eqiad.wmnet: Unknown Object (Task).
Sep 19 2019, 4:15 PM · decommission, Operations, fundraising-tech-ops, ops-eqiad, DC-Ops

Sep 13 2019

Jgreen renamed T232633: fundraising dev/database access request for Mariana Suijkerbuijk from Access request - account Yubikey servers to fundraising dev/database access request for Mariana Suijkerbuijk.
Sep 13 2019, 6:40 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen added a comment to T232633: fundraising dev/database access request for Mariana Suijkerbuijk.

Caitlin Cogdill sent an access authorization request to Lisa Gruwell earlier this week, we're waiting to hear back on that.

Sep 13 2019, 6:39 PM · fundraising-tech-ops, Fundraising-Backlog

Sep 12 2019

Jgreen added a comment to T232623: fundraising access request for Nora Nichols.

Yubikey requested from OIT...

Sep 12 2019, 8:11 PM · fundraising-tech-ops
Jgreen added a comment to T202419: Page fr-tech about spikes in frtechmail.

Reopening, now that we have our own prometheus/grafana instance, would it make sense to alert from it?

Sep 12 2019, 6:21 PM · Fundraising-Backlog, fundraising-tech-ops