Page MenuHomePhabricator

Jgreen (Jeff Green)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Nov 25 2014, 1:54 PM (293 w, 6 h)
Availability
Available
IRC Nick
Jeff_Green
LDAP User
Jgreen
MediaWiki User
Jgreen (wmf) [ Global Accounts ]

Recent Activity

Today

Jgreen moved T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing from Stalled to Up Next on the fundraising-tech-ops board.
Tue, Jul 7, 6:23 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T254298: Upgrade payments to MediaWiki 1.35 from Watching to Up Next on the fundraising-tech-ops board.
Tue, Jul 7, 6:22 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster, a subtask of T243019: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data , from Stalled to Open.
Tue, Jul 7, 6:21 PM · fr-tech-ops-okr, Fundraising-Backlog, fundraising-tech-ops
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster, a subtask of T254298: Upgrade payments to MediaWiki 1.35, from Stalled to Open.
Tue, Jul 7, 6:21 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster from Stalled to Open.
Tue, Jul 7, 6:21 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen created T257352: Create fundraising MediaWiki 1.35 payments deploy branch.
Tue, Jul 7, 6:19 PM · Fundraising-Backlog
Jgreen moved T256146: Upgrade payments servers to Debian Buster from Stalled to Up Next on the fundraising-tech-ops board.
Tue, Jul 7, 6:16 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen moved T254940: Test CiviCRM on new box from Stalled to In Progress on the fundraising-tech-ops board.
Tue, Jul 7, 6:14 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen moved T242257: 2019-2020 Q3 fundraising hardware refresh and capex from Stalled to Backlog on the fundraising-tech-ops board.
Tue, Jul 7, 6:13 PM · fundraising-tech-ops
Jgreen removed a project from T256146: Upgrade payments servers to Debian Buster: Fundraising-Backlog.
Tue, Jul 7, 6:07 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen removed a project from T170318: payments-listener server high availability: Fundraising-Backlog.
Tue, Jul 7, 6:03 PM · fundraising-tech-ops
Jgreen closed T162687: tweak process-control email template to be inbox friendlier as Declined.

Closing this task since it didn't get picked up after three years.

Tue, Jul 7, 5:58 PM · Fundraising-Backlog
Jgreen changed the status of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, a subtask of T243019: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data , from Open to Stalled.
Tue, Jul 7, 5:05 PM · fr-tech-ops-okr, Fundraising-Backlog, fundraising-tech-ops
Jgreen changed the status of T242257: 2019-2020 Q3 fundraising hardware refresh and capex from Open to Stalled.
Tue, Jul 7, 5:05 PM · fundraising-tech-ops
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster, a subtask of T243019: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data , from Open to Stalled.
Tue, Jul 7, 5:05 PM · fr-tech-ops-okr, Fundraising-Backlog, fundraising-tech-ops
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster, a subtask of T254298: Upgrade payments to MediaWiki 1.35, from Open to Stalled.
Tue, Jul 7, 5:05 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen changed the status of T256146: Upgrade payments servers to Debian Buster from Open to Stalled.
Tue, Jul 7, 5:05 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen changed the status of T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing, a subtask of T254298: Upgrade payments to MediaWiki 1.35, from Open to Stalled.
Tue, Jul 7, 5:04 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen changed the status of T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing from Open to Stalled.
Tue, Jul 7, 5:04 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen changed the status of T254940: Test CiviCRM on new box, a subtask of T242270: rack/setup/install civi2001.frack.codfw.wmnet, from Open to Stalled.
Tue, Jul 7, 5:04 PM · Patch-For-Review, fundraising-tech-ops
Jgreen changed the status of T254940: Test CiviCRM on new box from Open to Stalled.
Tue, Jul 7, 5:04 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen changed the status of T242270: rack/setup/install civi2001.frack.codfw.wmnet, a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, from Open to Stalled.
Tue, Jul 7, 5:04 PM · fundraising-tech-ops
Jgreen changed the status of T242270: rack/setup/install civi2001.frack.codfw.wmnet from Open to Stalled.
Tue, Jul 7, 5:04 PM · Patch-For-Review, fundraising-tech-ops
Jgreen added a subtask for T245557: break out fundraising outbound mail server to a separate host: T257245: procure and deploy new frmail server role.
Tue, Jul 7, 1:42 PM · fundraising-tech-ops
Jgreen added a parent task for T257245: procure and deploy new frmail server role: T245557: break out fundraising outbound mail server to a separate host.
Tue, Jul 7, 1:42 PM · fundraising-tech-ops
Jgreen added subtasks for T257245: procure and deploy new frmail server role: Unknown Object (Task), Unknown Object (Task).
Tue, Jul 7, 1:42 PM · fundraising-tech-ops

Yesterday

Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Mon, Jul 6, 8:12 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Mon, Jul 6, 7:59 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T257232: map out a civicrm.log_% table data expiration/delete strategy.
Mon, Jul 6, 7:57 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T257245: procure and deploy new frmail server role.
Mon, Jul 6, 7:17 PM · fundraising-tech-ops
Jgreen added a subtask for T257242: frdb1001 hardware refresh: Unknown Object (Task).
Mon, Jul 6, 7:15 PM · fundraising-tech-ops
Jgreen added a parent task for T257242: frdb1001 hardware refresh: T257244: 2010-2021 fundraising hardware refresh and capex.
Mon, Jul 6, 7:15 PM · fundraising-tech-ops
Jgreen added a subtask for T257244: 2010-2021 fundraising hardware refresh and capex: T257242: frdb1001 hardware refresh.
Mon, Jul 6, 7:15 PM · fundraising-tech-ops
Jgreen updated the task description for T257244: 2010-2021 fundraising hardware refresh and capex.
Mon, Jul 6, 7:14 PM · fundraising-tech-ops
Jgreen created T257244: 2010-2021 fundraising hardware refresh and capex.
Mon, Jul 6, 7:13 PM · fundraising-tech-ops
Jgreen created T257242: frdb1001 hardware refresh.
Mon, Jul 6, 7:00 PM · fundraising-tech-ops
Jgreen triaged T257232: map out a civicrm.log_% table data expiration/delete strategy as Medium priority.
Mon, Jul 6, 5:54 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T256924: Add logging and lockfiles to fr analytics scripts where needed from Triage to Up Next on the fundraising-tech-ops board.
Mon, Jul 6, 5:54 PM · fundraising-tech-ops
Jgreen moved T257232: map out a civicrm.log_% table data expiration/delete strategy from Triage to Up Next on the fundraising-tech-ops board.
Mon, Jul 6, 5:54 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T257232: map out a civicrm.log_% table data expiration/delete strategy.
Mon, Jul 6, 5:24 PM · Fundraising-Backlog, fundraising-tech-ops

Thu, Jul 2

Jgreen updated the task description for T243019: OKR 2019-2020 Q3: Support fundraising through maintaining infrastructure for payments and donor data .
Thu, Jul 2, 3:00 PM · fr-tech-ops-okr, Fundraising-Backlog, fundraising-tech-ops
Jgreen placed T256146: Upgrade payments servers to Debian Buster up for grabs.
Thu, Jul 2, 2:59 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen added a comment to T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster.

faulkner database is removed everywhere except frdb1003 (fr analytics db) and the archived backup

Thu, Jul 2, 2:58 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T256670: fundraising database "faulkner" end of life, a subtask of T255456: investigate moving non-essential databases (faulkner, pgehres, fredge) off of fundraising database cluster, as Resolved.
Thu, Jul 2, 2:57 PM · FR-Tech-Analytics, Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T256670: fundraising database "faulkner" end of life as Resolved.
Thu, Jul 2, 2:57 PM · fundraising-tech-ops
Jgreen updated the task description for T256670: fundraising database "faulkner" end of life.
Thu, Jul 2, 2:57 PM · fundraising-tech-ops
Jgreen updated the task description for T256670: fundraising database "faulkner" end of life.
Thu, Jul 2, 2:54 PM · fundraising-tech-ops
Jgreen claimed T256670: fundraising database "faulkner" end of life.
Thu, Jul 2, 2:49 PM · fundraising-tech-ops
Jgreen moved T256670: fundraising database "faulkner" end of life from Up Next to In Progress on the fundraising-tech-ops board.
Thu, Jul 2, 2:48 PM · fundraising-tech-ops
Jgreen added a project to T256916: Create Grafana dashboard for Online Fundraising to monitor live campaigns : Fundraising-Backlog.
Thu, Jul 2, 2:32 PM · Fundraising-Backlog, Epic, observability, fundraising-tech-ops

Wed, Jul 1

Jgreen updated the task description for T256916: Create Grafana dashboard for Online Fundraising to monitor live campaigns .
Wed, Jul 1, 8:16 PM · Fundraising-Backlog, Epic, observability, fundraising-tech-ops
Jgreen moved T256916: Create Grafana dashboard for Online Fundraising to monitor live campaigns from Triage to Up Next on the fundraising-tech-ops board.
Wed, Jul 1, 7:47 PM · Fundraising-Backlog, Epic, observability, fundraising-tech-ops
Jgreen created T256916: Create Grafana dashboard for Online Fundraising to monitor live campaigns .
Wed, Jul 1, 7:46 PM · Fundraising-Backlog, Epic, observability, fundraising-tech-ops
Jgreen moved T256670: fundraising database "faulkner" end of life from Triage to Up Next on the fundraising-tech-ops board.
Wed, Jul 1, 7:30 PM · fundraising-tech-ops
Jgreen renamed T162946: collect SmashPig database/queue metrics to fundraising prometheus/grafana from Write Grafana monitors for SmashPig database things to collect SmashPig database/queue metrics to fundraising prometheus/grafana.
Wed, Jul 1, 7:29 PM · fundraising-tech-ops, Fundraising-Backlog, FR-Smashpig
Jgreen added a parent task for T162946: collect SmashPig database/queue metrics to fundraising prometheus/grafana: T91508: [Epic] overhaul fundraising cluster monitoring.
Wed, Jul 1, 7:28 PM · fundraising-tech-ops, Fundraising-Backlog, FR-Smashpig
Jgreen added a subtask for T91508: [Epic] overhaul fundraising cluster monitoring: T162946: collect SmashPig database/queue metrics to fundraising prometheus/grafana.
Wed, Jul 1, 7:28 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T239564: Monitor and investigate possible event dropping by Kafkatee as Resolved.

We've improved kafkatee monitoring significantly since this task was created. We are now catching cases where kafkatee does not properly connect to a broker, stops logging its stats, or fails to write content to log files. We've also got detailed metrics collecting to fundraising prometheus/grafana which could be used to alert on error counters or similar, once we actually see the type of data loss hypothesized and observe how it plays out in the gathered metrics.

Wed, Jul 1, 7:26 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen placed T242270: rack/setup/install civi2001.frack.codfw.wmnet up for grabs.
Wed, Jul 1, 7:15 PM · Patch-For-Review, fundraising-tech-ops
Jgreen placed T226749: fundraising tech operations GDPR tracking task up for grabs.
Wed, Jul 1, 7:14 PM · fundraising-tech-ops
Jgreen placed T77912: [epic] PCI gap analysis and Improvements up for grabs.
Wed, Jul 1, 7:14 PM · fundraising-tech-ops, Epic, Fundraising-Backlog
Jgreen placed T222840: move main fundraising puppet repo to gerrit up for grabs.
Wed, Jul 1, 7:13 PM · Gerrit, fundraising-tech-ops
Jgreen added a subtask for T228765: Epic: matching gifts HEP integration (was employer field V2): Unknown Object (Task).
Wed, Jul 1, 4:40 PM · Epic, Fundraising-Backlog

Mon, Jun 29

Jgreen closed T256683: fundraising kafkatee ca certificate has expired as Resolved.
Mon, Jun 29, 8:49 PM · fundraising-tech-ops
Jgreen added a comment to T256683: fundraising kafkatee ca certificate has expired.

Refreshed CA cert from puppetmaster1001:/srv/private/modules/secret/secrets/certificates/kafka_fundraising_client which has fixed the problem.

Mon, Jun 29, 8:49 PM · fundraising-tech-ops
Jgreen updated the task description for T256683: fundraising kafkatee ca certificate has expired.
Mon, Jun 29, 8:37 PM · fundraising-tech-ops
Jgreen added a comment to T256683: fundraising kafkatee ca certificate has expired.

https://wikitech.wikimedia.org/wiki/Cergen

Mon, Jun 29, 8:32 PM · fundraising-tech-ops
Jgreen triaged T256683: fundraising kafkatee ca certificate has expired as Unbreak Now! priority.
Mon, Jun 29, 8:21 PM · fundraising-tech-ops
Jgreen created T256683: fundraising kafkatee ca certificate has expired.
Mon, Jun 29, 8:20 PM · fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 6:05 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen created T256670: fundraising database "faulkner" end of life.
Mon, Jun 29, 6:03 PM · fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 4:18 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 4:16 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 4:01 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 4:00 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 3:19 PM · Epic, Fundraising-Backlog, fundraising-tech-ops
Jgreen updated the task description for T241083: Epic: fundraising database approaching data partition capacity .
Mon, Jun 29, 3:17 PM · Epic, Fundraising-Backlog, fundraising-tech-ops

Fri, Jun 26

Jgreen moved T256420: Determine reason for daily increasing proc count on fran1001 from Triage to Up Next on the fundraising-tech-ops board.
Fri, Jun 26, 7:00 PM · fundraising-tech-ops

Thu, Jun 25

Jgreen triaged T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync as Medium priority.
Thu, Jun 25, 8:48 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen closed T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync , a subtask of T254940: Test CiviCRM on new box, as Resolved.
Thu, Jun 25, 8:47 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen closed T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync as Resolved.
Thu, Jun 25, 8:47 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen added a comment to T254940: Test CiviCRM on new box.

@Jgreen it would be nicer if they shared that dir, but it might not be too bad. We'll just have to remember to do a manual sync if we have to fail over. The audit jobs run once a day, so we will have some time to do that.

Thu, Jun 25, 8:47 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen moved T256146: Upgrade payments servers to Debian Buster from Triage to In Progress on the fundraising-tech-ops board.
Thu, Jun 25, 6:23 PM · fr-tech-ops-okr, fundraising-tech-ops
Jgreen moved T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing from Triage to In Progress on the fundraising-tech-ops board.
Thu, Jun 25, 6:23 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T254940: Test CiviCRM on new box from Triage to In Progress on the fundraising-tech-ops board.
Thu, Jun 25, 6:23 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen moved T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync from Triage to In Progress on the fundraising-tech-ops board.
Thu, Jun 25, 6:23 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen closed T256318: modify pc_log_archiver to add hostname to the target filename on log rotation as Resolved.
Thu, Jun 25, 6:23 PM · fundraising-tech-ops
Jgreen closed T256318: modify pc_log_archiver to add hostname to the target filename on log rotation, a subtask of T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync , as Resolved.
Thu, Jun 25, 6:23 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen claimed T256318: modify pc_log_archiver to add hostname to the target filename on log rotation.
Thu, Jun 25, 3:07 PM · fundraising-tech-ops

Wed, Jun 24

Jgreen created T256318: modify pc_log_archiver to add hostname to the target filename on log rotation.
Wed, Jun 24, 8:36 PM · fundraising-tech-ops
Jgreen removed projects from T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync : Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog.
Wed, Jun 24, 8:30 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen created T256317: refactor civi1001 archive_sync jobs to also keep civi1001 and civi2001 audit data in sync .
Wed, Jun 24, 8:29 PM · fundraising-tech-ops, Wikimedia-Fundraising-CiviCRM
Jgreen added a project to T254940: Test CiviCRM on new box: fundraising-tech-ops.
Wed, Jun 24, 8:20 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen merged task T253159: test civicrm and peripheral software on Debian Buster into T254940: Test CiviCRM on new box.
Wed, Jun 24, 8:20 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen merged T253159: test civicrm and peripheral software on Debian Buster into T254940: Test CiviCRM on new box.
Wed, Jun 24, 8:20 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen added a subtask for T242270: rack/setup/install civi2001.frack.codfw.wmnet: T254940: Test CiviCRM on new box.
Wed, Jun 24, 8:20 PM · Patch-For-Review, fundraising-tech-ops
Jgreen added a parent task for T254940: Test CiviCRM on new box: T242270: rack/setup/install civi2001.frack.codfw.wmnet.
Wed, Jun 24, 8:20 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen added a comment to T254940: Test CiviCRM on new box.

@Ejegg how significant is it that civi1001:/var/spool/audit vs civi2001:/var/spool/audit are not in sync? Right now synchronizing those before/after audit job runs is a totally manual and there's no mechanism or procedure to deal with it.

Wed, Jun 24, 5:02 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM
Jgreen added a comment to T254940: Test CiviCRM on new box.

IIRC audit processing (or maybe just orphan-slaying?) also uses logs that we push over from the central logger, which is done by archive_sync on frlog*, but we hadn't set this up yet for civi2001. This is fixed.

Wed, Jun 24, 4:42 PM · fundraising-tech-ops, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising Sprint Lazy Loading Life, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM

Tue, Jun 23

Jgreen added a comment to T256153: Configure payments2xxx for MediaWiki 1.35 upgrade testing.

@Ejegg ok we're set to use the payments-staging frdeploy project for this, we just need to set it to the right branch and make the config changes for 1.35.

Tue, Jun 23, 8:51 PM · fundraising-tech-ops, Fundraising-Backlog