Page MenuHomePhabricator

Jgreen (Jeff Green)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Nov 25 2014, 1:54 PM (234 w, 3 d)
Availability
Available
IRC Nick
Jeff_Green
LDAP User
Jgreen
MediaWiki User
Jgreen (wmf) [ Global Accounts ]

Recent Activity

Yesterday

Jgreen closed Restricted Task, a subtask of T91508: [Epic] overhaul fundraising cluster monitoring, as Resolved.
Thu, May 23, 8:39 PM · Epic, observability, fundraising-tech-ops
Jgreen closed T222761: Update civi1001 django_settings.py, backfill banner impressions as Resolved.
Thu, May 23, 8:37 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T183138: fundraising database improvements for 2018 as Resolved.
Thu, May 23, 8:37 PM · Fundraising-Backlog, fundraising-tech-ops, Epic

Wed, May 22

Jgreen added a comment to T218261: Build out Debian Buster test instances in virtualbox dev environment.

build: done & prod-worthy
siem: done & prod-worth

Wed, May 22, 6:41 PM · fundraising-tech-ops
Jgreen lowered the priority of T183138: fundraising database improvements for 2018 from High to Normal.
Wed, May 22, 5:35 PM · Fundraising-Backlog, fundraising-tech-ops, Epic
Jgreen moved T183138: fundraising database improvements for 2018 from Backlog to Done on the fundraising-tech-ops board.
Wed, May 22, 5:23 PM · Fundraising-Backlog, fundraising-tech-ops, Epic
Jgreen removed a subtask for T183138: fundraising database improvements for 2018 : T170321: encrypt fundraising database client->server communication.
Wed, May 22, 5:23 PM · Fundraising-Backlog, fundraising-tech-ops, Epic
Jgreen removed a parent task for T170321: encrypt fundraising database client->server communication: T183138: fundraising database improvements for 2018 .
Wed, May 22, 5:23 PM · fundraising-tech-ops
Jgreen moved T222974: install nginx 1.14.1 from stretch-backports on payments servers, enable TLSv1.3 from Backlog to Done on the fundraising-tech-ops board.
Wed, May 22, 4:43 PM · fundraising-tech-ops, FR-Ingenico, Fundraising-Backlog
Jgreen closed T222974: install nginx 1.14.1 from stretch-backports on payments servers, enable TLSv1.3 as Declined.

Our current thinking is that this is unlikely to help with SSL negotiation failures, and we'd rather stick with the stock version otherwise.

Wed, May 22, 4:42 PM · fundraising-tech-ops, FR-Ingenico, Fundraising-Backlog

Tue, May 21

Jgreen added a project to T107384: Redirect old PayPal IPN URL to active listener: fundraising-tech-ops.
Tue, May 21, 6:09 PM · fundraising-tech-ops, Recurring-Donations, Fundraising-Backlog
Jgreen added a comment to T216633: Redis queues should use a key prefix.

I linked this to T221008 because we're talking about making this change at the same time as we upgrade the queue servers and redis, since we'll need downtime and empty queues for both tasks.

Tue, May 21, 5:56 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a parent task for T216633: Redis queues should use a key prefix: T221008: upgrade fundraising queue servers from Debian Jessie.
Tue, May 21, 5:55 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a subtask for T221008: upgrade fundraising queue servers from Debian Jessie: T216633: Redis queues should use a key prefix.
Tue, May 21, 5:55 PM · fundraising-tech-ops
Jgreen added a comment to T223658: Enable Content Security Policy on donate-wiki.

@Jgreen This look identical to the ticket referenced. can you give more info?

Tue, May 21, 5:49 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen renamed T223658: Enable Content Security Policy on donate-wiki from Enable Content Security Policy on payments-wiki to Enable Content Security Policy on donate-wiki.
Tue, May 21, 5:48 PM · fundraising-tech-ops, Fundraising-Backlog

Fri, May 17

Jgreen created T223658: Enable Content Security Policy on donate-wiki.
Fri, May 17, 7:05 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen renamed T190183: Enable Content Security Policy on payments-wiki from Enable Content Security Policy on payments-wiki and donate-wiki to Enable Content Security Policy on payments-wiki.
Fri, May 17, 7:03 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T222618: schedule ~15 minute payments outage to switch clusters from codfw to eqiad as Resolved.
Fri, May 17, 12:02 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen moved T222618: schedule ~15 minute payments outage to switch clusters from codfw to eqiad from In Progress to Done on the fundraising-tech-ops board.
Fri, May 17, 12:01 PM · Fundraising-Backlog, fundraising-tech-ops

Wed, May 15

Jgreen added a comment to T222618: schedule ~15 minute payments outage to switch clusters from codfw to eqiad .

We're planning for 2PM Thurs 5/16/2019.

Wed, May 15, 2:45 PM · Fundraising-Backlog, fundraising-tech-ops

Fri, May 10

hashar awarded T222840: move main fundraising puppet repo to gerrit a Love token.
Fri, May 10, 8:58 PM · Gerrit, fundraising-tech-ops
Jgreen created T222974: install nginx 1.14.1 from stretch-backports on payments servers, enable TLSv1.3.
Fri, May 10, 6:13 PM · fundraising-tech-ops, FR-Ingenico, Fundraising-Backlog

Thu, May 9

Jgreen removed a subtask for T184460: Upgrade PaymentsWiki to Mediawiki 1.31 (new LTS): T185547: Use Composer merge plugin for payments-wiki.
Thu, May 9, 3:48 PM · Fundraising Sprint Hansel and grep -l, Fundraising Sprint Greps of Wrath, Fundraising Sprint Fistful of $variables, Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Da Vinci Coder, Fundraising Sprint Casino Royale With Cheese, MW-1.33-notes (1.33.0-wmf.16; 2019-02-05), Fundraising Sprint Bert and Ernie's Excellent Adventure, Patch-For-Review, Fundraising Sprint A series of unfortunate event handlers, Fundraising-Backlog
Jgreen removed a parent task for T185547: Use Composer merge plugin for payments-wiki: T184460: Upgrade PaymentsWiki to Mediawiki 1.31 (new LTS).
Thu, May 9, 3:48 PM · Fundraising Sprint Queue is pronounced GJif, Patch-For-Review, Fundraising-Backlog, MediaWiki-extensions-DonationInterface
Jgreen closed T220453: upgrade civi1001 and mintaka to Stretch, possibly Buster, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Thu, May 9, 3:47 PM · fundraising-tech-ops
Jgreen closed T220453: upgrade civi1001 and mintaka to Stretch, possibly Buster as Resolved.

civi1001 is done

Thu, May 9, 3:47 PM · fundraising-tech-ops
Jgreen moved T220452: upgrade frdev1001 to Debian Stretch from Backlog to Done on the fundraising-tech-ops board.
Thu, May 9, 3:47 PM · fundraising-tech-ops
Jgreen renamed T220452: upgrade frdev1001 to Debian Stretch from upgrade frdev1001 to Debian Buster to upgrade frdev1001 to Debian Stretch.
Thu, May 9, 3:45 PM · fundraising-tech-ops
Jgreen closed T220452: upgrade frdev1001 to Debian Stretch, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Thu, May 9, 3:45 PM · fundraising-tech-ops

Wed, May 8

Jgreen created T222840: move main fundraising puppet repo to gerrit.
Wed, May 8, 8:05 PM · Gerrit, fundraising-tech-ops
Jgreen closed T222789: check_bannerimpressions.py 'no entries' alerts after upgrading frdev1001 to stretch as Resolved.

This was fixed by T222761

Wed, May 8, 3:30 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen updated subscribers of T222789: check_bannerimpressions.py 'no entries' alerts after upgrading frdev1001 to stretch.
Wed, May 8, 12:00 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T222789: check_bannerimpressions.py 'no entries' alerts after upgrading frdev1001 to stretch.
Wed, May 8, 12:00 PM · Fundraising-Backlog, fundraising-tech-ops

Mon, May 6

Jgreen closed T186018: add "frtech_mail" syslog filter to thulium as Declined.
Mon, May 6, 2:56 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T188561: SSL cert for links.email.wikimedia.org.

@BBlack circling back on this, do you still see any issue now after the Silverpop SSL improvements?

Mon, May 6, 2:54 PM · Operations, Traffic, fundraising-tech-ops, Fundraising-Backlog
Jgreen moved T222618: schedule ~15 minute payments outage to switch clusters from codfw to eqiad from Backlog to In Progress on the fundraising-tech-ops board.
Mon, May 6, 2:47 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen created T222618: schedule ~15 minute payments outage to switch clusters from codfw to eqiad .
Mon, May 6, 2:41 PM · Fundraising-Backlog, fundraising-tech-ops

Thu, May 2

Jgreen moved T220453: upgrade civi1001 and mintaka to Stretch, possibly Buster from Backlog to In Progress on the fundraising-tech-ops board.
Thu, May 2, 7:28 PM · fundraising-tech-ops
Jgreen moved T222001: Outage to reload triggers from Backlog to In Progress on the fundraising-tech-ops board.
Thu, May 2, 7:28 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T222001: Outage to reload triggers.

@CCogdill_WMF it looks like Ops Do have another outage need so this should be able to be done then - @cwd @Jgreen hopefully this is on your radar

Thu, May 2, 6:20 PM · Fundraising-Backlog, fundraising-tech-ops

Wed, May 1

Jgreen moved T221003: upgrade fundraising databases from Debian Jessie to Stretch from Backlog to In Progress on the fundraising-tech-ops board.
Wed, May 1, 6:54 PM · fundraising-tech-ops
Jgreen moved T221116: Investigate r-cran-shiny and/or RStudio shiny-server for Fundraising team use. from Backlog to In Progress on the fundraising-tech-ops board.
Wed, May 1, 6:54 PM · fundraising-tech-ops

Mon, Apr 29

Jgreen created T222109: decommission frav1001.frack.eqiad.wmnet.
Mon, Apr 29, 8:53 PM · Operations, fundraising-tech-ops, ops-eqiad, DC-Ops
Jgreen closed T221274: configure and install frav1002.frack.eqiad.wmnet as Resolved.

this is done

Mon, Apr 29, 8:47 PM · fundraising-tech-ops
Jgreen closed T214525: remove IBM/Silverpop 1024-bit domain key as Resolved.

this is deployed

Mon, Apr 29, 5:38 PM · Patch-For-Review, Traffic, Operations, fundraising-tech-ops, DNS
Jgreen closed T222070: frbackup2001 data partition is at 95% capacity as Resolved.
Mon, Apr 29, 1:23 PM · fundraising-tech-ops
Jgreen added a comment to T222070: frbackup2001 data partition is at 95% capacity .

This is done, now it's down to 74%.

Mon, Apr 29, 1:23 PM · fundraising-tech-ops
Jgreen added a project to T222070: frbackup2001 data partition is at 95% capacity : fundraising-tech-ops.
Mon, Apr 29, 1:23 PM · fundraising-tech-ops
Jgreen created T222070: frbackup2001 data partition is at 95% capacity .
Mon, Apr 29, 1:23 PM · fundraising-tech-ops

Apr 18 2019

Jgreen added a comment to T221274: configure and install frav1002.frack.eqiad.wmnet.

frav1002 is up and running

  • aiderator is set up and seems to be working correctly
  • dmarc parser is set up and seems to be working correctly
Apr 18 2019, 9:10 PM · fundraising-tech-ops
Jgreen closed T221276: iptables policy for frav1002.frack.eqiad.wmnet as Resolved.

deployed and working

Apr 18 2019, 8:52 PM · fundraising-tech-ops
Jgreen closed T221276: iptables policy for frav1002.frack.eqiad.wmnet, a subtask of T221274: configure and install frav1002.frack.eqiad.wmnet, as Resolved.
Apr 18 2019, 8:52 PM · fundraising-tech-ops

Apr 17 2019

Jgreen added a comment to T221276: iptables policy for frav1002.frack.eqiad.wmnet.

commit 154be961aa593905778a0479699ab0434551c11d
Author: Jeff Green <jgreen@wikimedia.org>
Date: Wed Apr 17 19:23:13 2019 +0000

Apr 17 2019, 7:27 PM · fundraising-tech-ops
Jgreen created T221276: iptables policy for frav1002.frack.eqiad.wmnet.
Apr 17 2019, 7:27 PM · fundraising-tech-ops
Jgreen closed T213104: rack and cable frav1002.frack.eqiad.wmnet as Resolved.

Runs good!

Apr 17 2019, 7:26 PM · Patch-For-Review, Operations, fundraising-tech-ops, ops-eqiad
Jgreen closed T213104: rack and cable frav1002.frack.eqiad.wmnet, a subtask of T221274: configure and install frav1002.frack.eqiad.wmnet, as Resolved.
Apr 17 2019, 7:26 PM · fundraising-tech-ops
Jgreen added a parent task for T213104: rack and cable frav1002.frack.eqiad.wmnet: T221274: configure and install frav1002.frack.eqiad.wmnet.
Apr 17 2019, 7:26 PM · Patch-For-Review, Operations, fundraising-tech-ops, ops-eqiad
Jgreen added a subtask for T221274: configure and install frav1002.frack.eqiad.wmnet: T213104: rack and cable frav1002.frack.eqiad.wmnet.
Apr 17 2019, 7:26 PM · fundraising-tech-ops
Jgreen created T221274: configure and install frav1002.frack.eqiad.wmnet.
Apr 17 2019, 7:25 PM · fundraising-tech-ops
Jgreen renamed T206870: decommission betelgeuse.frack.codfw.wmnet from decommission betelgeuse to decommission betelgeuse.frack.codfw.wmnet.
Apr 17 2019, 3:42 PM · Patch-For-Review, decommission, fundraising-tech-ops, ops-codfw, Operations
Jgreen created T221232: configure switch ports for frav1002.frack.eqiad.wmnet.
Apr 17 2019, 1:56 PM · Operations, netops, fundraising-tech-ops
Jgreen closed T215482: Update settings for environment indicator to reflect new format as Resolved.

@Jgreen looks good & any time is fine - it affects the icon on the browser tab - ie

Apr 17 2019, 1:40 PM · Fundraising-Backlog, fundraising-tech-ops

Apr 16 2019

Jgreen added a comment to T213104: rack and cable frav1002.frack.eqiad.wmnet.

@Jgreen I need an mgmt IP for this.... I apologize but I don't recall any of the fundraising mgmt info.

please provide IP, Subnet, and Gateway

Apr 16 2019, 9:11 PM · Patch-For-Review, Operations, fundraising-tech-ops, ops-eqiad
Jgreen closed T217413: cronspam: file size changed while zipping as Declined.

I think this is just par for the course when backing up directories where writes may be happening.

Apr 16 2019, 8:41 PM · fundraising-tech-ops
Jgreen added a comment to T216633: Redis queues should use a key prefix.

Alternate deploy strategy: update /etc/smashpig/main.yaml on payments first, wait till queues are empty before updating the same config file on civi1001.

Apr 16 2019, 8:37 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen triaged T221008: upgrade fundraising queue servers from Debian Jessie as High priority.
Apr 16 2019, 8:33 PM · fundraising-tech-ops
Jgreen added a comment to T221008: upgrade fundraising queue servers from Debian Jessie.

Versions available with Debian:

Apr 16 2019, 6:38 PM · fundraising-tech-ops
Jgreen added a comment to T221116: Investigate r-cran-shiny and/or RStudio shiny-server for Fundraising team use..

Note that we're using shiny server elsewhere at WMF and production puppet has modules/shiny_server already.

Apr 16 2019, 5:46 PM · fundraising-tech-ops
Jgreen added a comment to T221116: Investigate r-cran-shiny and/or RStudio shiny-server for Fundraising team use..

Upon first pass through the docs and software, the major difference as I see it is the authentication and user-access features--i.e. the emphasis on being able to run backend processes as different unix user based on i.e. PAM authentication. Also SSL support. It's not clear to me that we need to compartmentalize to different users. Also I think we may be able to put it behind a proxy for SSL and client authentication. We're going to set up a call to talk about technical options.

Apr 16 2019, 5:28 PM · fundraising-tech-ops
Jgreen created T221116: Investigate r-cran-shiny and/or RStudio shiny-server for Fundraising team use..
Apr 16 2019, 5:23 PM · fundraising-tech-ops
Jgreen claimed T215482: Update settings for environment indicator to reflect new format.
Apr 16 2019, 4:13 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen added a comment to T215482: Update settings for environment indicator to reflect new format.

@Eileenmcnaughton is this correct?

Apr 16 2019, 4:08 PM · Fundraising-Backlog, fundraising-tech-ops
Jgreen closed T216559: disable rschoenbaechler in fundraising as Resolved.

done

Apr 16 2019, 3:56 PM · fundraising-tech-ops
Jgreen closed T217342: Deploy inconsistency between civi1001 and mintaka as Resolved.

I fixed this by puppetizing creation of /srv/org.wikimedia.civicrm-files/civicrm/templates_c.

Apr 16 2019, 3:51 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T217341: tmp on civi1001 is large as Declined.

It seems to be running fine now, if it crops up again we can reopen take another look.

Apr 16 2019, 3:38 PM · fundraising-tech-ops, Fundraising-Backlog
Jgreen closed T219280: sort out jessie vs jesse-backports vs openssl pinning issue as Resolved.

Adjusted apt sources and pin settings short term, while we finish phasing out jessie hosts.

Apr 16 2019, 3:27 PM · fundraising-tech-ops
Jgreen closed T219280: sort out jessie vs jesse-backports vs openssl pinning issue , a subtask of T219333: apt-get update broken on jessie: jessie-updates and jessie-backports removed by Debian, as Resolved.
Apr 16 2019, 3:27 PM · Patch-For-Review, Operations
Jgreen added a comment to T219280: sort out jessie vs jesse-backports vs openssl pinning issue .

Is this still an issue, the openssl package is provided via jessi-wikimedia/main

jbond@jbond-jessie:~$ apt-cache policy openssl
openssl:
  Installed: 1.0.2r-1~wmf1
  Candidate: 1.0.2r-1~wmf1
  Version table:
 *** 1.0.2r-1~wmf1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

If so can you provide me with an example machine

Thanks John

Apr 16 2019, 3:21 PM · fundraising-tech-ops
Jgreen closed T221036: upgrade payments servers from Debian Jessie to Stretch as Resolved.

Done. payments1001 is configured as payments mariadb master for both sites.

Apr 16 2019, 1:08 PM · fundraising-tech-ops
Jgreen closed T221036: upgrade payments servers from Debian Jessie to Stretch, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Apr 16 2019, 1:08 PM · fundraising-tech-ops

Apr 15 2019

Jgreen closed T202290: test and deploy payments on Debian Stretch/PHP7.0 as Resolved.

We've worked out all the identified upgrade-related issues with the test cluster, and are ready to upgrade the main cluster.

Apr 15 2019, 7:59 PM · fundraising-tech-ops
Jgreen closed T202290: test and deploy payments on Debian Stretch/PHP7.0, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Apr 15 2019, 7:59 PM · fundraising-tech-ops
Jgreen created T221036: upgrade payments servers from Debian Jessie to Stretch.
Apr 15 2019, 7:55 PM · fundraising-tech-ops
Jgreen created T221008: upgrade fundraising queue servers from Debian Jessie.
Apr 15 2019, 3:53 PM · fundraising-tech-ops
Jgreen added a comment to T165393: reimage americium to Debian Stretch.

Americium is actually due for replacement in FY 2019-20, tentatively planned for first quarter. It would be a whole lot less disruptive to spin up the new host and switch to it than to do a Trusty to Stretch upgrade.

Apr 15 2019, 3:39 PM · fundraising-tech-ops
Jgreen added a comment to T221003: upgrade fundraising databases from Debian Jessie to Stretch.

frdb1002 is done

Apr 15 2019, 3:37 PM · fundraising-tech-ops
Jgreen added a comment to T221003: upgrade fundraising databases from Debian Jessie to Stretch.

frdb2001 is done

Apr 15 2019, 3:36 PM · fundraising-tech-ops
Jgreen created T221003: upgrade fundraising databases from Debian Jessie to Stretch.
Apr 15 2019, 3:36 PM · fundraising-tech-ops

Apr 10 2019

Jgreen added a comment to T220412: Identify appropriate SPF record for domain wikimediafoundation.org.

I noticed there is a DKIM record in the wikimediafoundation.org zone, I wonder what this is/was used by?

Apr 10 2019, 6:33 PM · Patch-For-Review, fundraising-tech-ops, Fundraising-Backlog, Mail, Operations
Jgreen added a comment to T220453: upgrade civi1001 and mintaka to Stretch, possibly Buster.

mintaka is upgraded to stretch

Apr 10 2019, 5:20 PM · fundraising-tech-ops

Apr 9 2019

Jgreen closed T220516: whitelist extensions/FundraisingEmailUnsubscribe at payments reverse proxy as Resolved.

Fixed.

Apr 9 2019, 3:16 PM · fundraising-tech-ops
Jgreen added a project to T220516: whitelist extensions/FundraisingEmailUnsubscribe at payments reverse proxy: fundraising-tech-ops.
Apr 9 2019, 3:07 PM · fundraising-tech-ops
Jgreen created T220516: whitelist extensions/FundraisingEmailUnsubscribe at payments reverse proxy.
Apr 9 2019, 3:06 PM · fundraising-tech-ops
Jgreen renamed T220453: upgrade civi1001 and mintaka to Stretch, possibly Buster from upgrade frdev1001 and mintaka to Stretch, possibly Buster to upgrade civi1001 and mintaka to Stretch, possibly Buster.
Apr 9 2019, 2:57 PM · fundraising-tech-ops
Jgreen merged task T220100: upgrade pay-lvs servers to stretch into T185014: migrate pay-lvs servers to Debian Stretch.
Apr 9 2019, 2:56 PM · fundraising-tech-ops
Jgreen merged T220100: upgrade pay-lvs servers to stretch into T185014: migrate pay-lvs servers to Debian Stretch.
Apr 9 2019, 2:56 PM · fundraising-tech-ops
Jgreen closed T185014: migrate pay-lvs servers to Debian Stretch as Resolved.

pay-lvs1001 and 1002 are done

Apr 9 2019, 12:35 PM · fundraising-tech-ops
Jgreen closed T185014: migrate pay-lvs servers to Debian Stretch, a subtask of T176260: build & deploy latest pybal for frack, as Resolved.
Apr 9 2019, 12:35 PM · fundraising-tech-ops
Jgreen closed T185014: migrate pay-lvs servers to Debian Stretch, a subtask of T185013: EPIC: migrate fundraising off of Debian Jessie, as Resolved.
Apr 9 2019, 12:35 PM · fundraising-tech-ops