Page MenuHomePhabricator

jhathaway (Jesse Hathaway)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Nov 22 2021, 10:00 PM (125 w, 16 h)
Availability
Available
LDAP User
JHathaway
MediaWiki User
JHathaway (WMF) [ Global Accounts ]

Recent Activity

Yesterday

jhathaway triaged T362109: dcl: add support for calico as Low priority.
Mon, Apr 15, 2:24 PM · Patch-For-Review, Infrastructure-Foundations

Tue, Apr 9

jhathaway closed T361750: Site: eqiad, codfw 2 VM request for postfix mx-out as Resolved.
Tue, Apr 9, 4:15 PM · Patch-For-Review, vm-requests, Infrastructure-Foundations, SRE

Mon, Apr 8

jhathaway added a comment to T362109: dcl: add support for calico.

patch is merged, waiting on a new release, https://github.com/kubernetes/minikube/pull/18233

Mon, Apr 8, 8:35 PM · Patch-For-Review, Infrastructure-Foundations
jhathaway created T362109: dcl: add support for calico.
Mon, Apr 8, 8:29 PM · Patch-For-Review, Infrastructure-Foundations
jhathaway closed T111595: Do not apply spam headers on email assessed NOT to be spam as Declined.

@bcampbell setting this to declined, please reopen, if this is still a concern

Mon, Apr 8, 3:23 PM · Infrastructure-Foundations, Mail, SRE
jhathaway changed the status of T360804: macOS aarch64 support from Open to Stalled.

After doing some exploration I don't think emulation is viable due to the complexity and the performance, so blocking this task until T320811 is resolved. In the interim running dcl on WMCS looks to be the best alternative.

Mon, Apr 8, 2:33 PM · ARM support, Infrastructure-Foundations
jhathaway lowered the priority of T360804: macOS aarch64 support from Medium to Low.
Mon, Apr 8, 2:33 PM · ARM support, Infrastructure-Foundations
jhathaway changed the status of T360804: macOS aarch64 support, a subtask of T349450: dcl: Version one feedback, from Open to Stalled.
Mon, Apr 8, 2:33 PM · Infrastructure-Foundations
jhathaway triaged T360804: macOS aarch64 support as Medium priority.
Mon, Apr 8, 2:28 PM · ARM support, Infrastructure-Foundations
jhathaway triaged T360146: Proposal: implement `dcl ssh` subcommand as Low priority.
Mon, Apr 8, 2:27 PM · Infrastructure-Foundations

Fri, Apr 5

jhathaway created P59690 (An Untitled Masterwork).
Fri, Apr 5, 4:12 PM
jhathaway added a comment to T361860: Old "Email this user" email is repeatedly resent.

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

Fri, Apr 5, 3:48 PM · Mail, Infrastructure-Foundations, MediaWiki-Email, SRE

Wed, Apr 3

jhathaway renamed T361750: Site: eqiad, codfw 2 VM request for postfix mx-out from Site: eqiad, codfw 2 VM request for postfix mta-out to Site: eqiad, codfw 2 VM request for postfix mx-out.
Wed, Apr 3, 9:04 PM · Patch-For-Review, vm-requests, Infrastructure-Foundations, SRE
jhathaway triaged T361750: Site: eqiad, codfw 2 VM request for postfix mx-out as Medium priority.
Wed, Apr 3, 9:01 PM · Patch-For-Review, vm-requests, Infrastructure-Foundations, SRE
jhathaway renamed T361750: Site: eqiad, codfw 2 VM request for postfix mx-out from Site: eqiad, codfw 1 VM request for postfix mta-out to Site: eqiad, codfw 2 VM request for postfix mta-out.
Wed, Apr 3, 8:58 PM · Patch-For-Review, vm-requests, Infrastructure-Foundations, SRE
jhathaway created T361750: Site: eqiad, codfw 2 VM request for postfix mx-out.
Wed, Apr 3, 8:51 PM · Patch-For-Review, vm-requests, Infrastructure-Foundations, SRE
jhathaway closed T332718: eqiad: (1) VM for MySQL Orchestrator as Declined.

part of bookworm upgrade sprint week, but I ran out of time, not currently prioritizing this work.

Wed, Apr 3, 7:16 PM · vm-requests, Infrastructure-Foundations, SRE

Tue, Apr 2

jhathaway added a comment to T360804: macOS aarch64 support.

FYI, this is going to be so pretty slow (anecdotal stories speak of standard puppet CI taking up to 15mins instead of a few secs, other workloads vary). See T320811 for the bigger picture, the rest of the tasks in ARM support spell out the various other problems met by developers.

Tue, Apr 2, 2:33 PM · ARM support, Infrastructure-Foundations

Mon, Apr 1

jhathaway added a comment to T356920: Access to DMARCIAN.

@DBu-WMF the current dmarc monitoring is still a work in progress. ITS has purchased a subscription to dmarcdigests via the security budget, which is currently active, but they are not intending to keep the service long term. However you can open a ticket with ITS to request access, while it is still available. As another route @Jgreen wrote some scripts for ingesting the reports for their own needs and we briefly discussed reusing that tooling outside of fund raising tech, but the work has not been planned as of yet.

Mon, Apr 1, 4:15 PM · Infrastructure-Foundations, Mail, SRE
jhathaway added a comment to T356920: Access to DMARCIAN.

@jhathaway: Another question is why the task is in S4 (Hardware Procurement) while it seems to have nothing to do with hardware procurement?

Mon, Apr 1, 4:07 PM · Infrastructure-Foundations, Mail, SRE

Fri, Mar 22

jhathaway added a comment to T338032: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost.

Thanks for the poke @RoySmith, ITS obtained this information from Zendesk on how Zendesk's spam marking system operates:

Fri, Mar 22, 8:43 PM · Mail, Infrastructure-Foundations, Trust-and-Safety
jhathaway renamed T360804: macOS aarch64 support from macOS arm support to macOS aarch64 support.
Fri, Mar 22, 5:21 PM · ARM support, Infrastructure-Foundations
jhathaway updated subscribers of T360804: macOS aarch64 support.

@SLyngshede-WMF let's move discussion to phabricator. I tried the updated patch on my x86_64 hackintosh with minikube 1.32:

Fri, Mar 22, 5:20 PM · ARM support, Infrastructure-Foundations
jhathaway added a comment to T360804: macOS aarch64 support.

patch, https://gitlab.wikimedia.org/jhathaway/dcl/-/merge_requests/2

Fri, Mar 22, 5:12 PM · ARM support, Infrastructure-Foundations
jhathaway created T360804: macOS aarch64 support.
Fri, Mar 22, 5:12 PM · ARM support, Infrastructure-Foundations

Feb 26 2024

jhathaway added a comment to T358398: corto-test.
  1. Incident metadata
Feb 26 2024, 11:00 PM · Trash
jhathaway claimed T232343: Consider Postfix as MTA for our MXes (and OTRS/Mailman/Phab).
Feb 26 2024, 3:46 PM · Infrastructure-Foundations, User-MoritzMuehlenhoff, Mail, SRE
jhathaway added a project to T358462: CORTOTEST3: Trash.
Feb 26 2024, 5:25 AM · Trash
jhathaway closed T358462: CORTOTEST3 as Invalid.
Feb 26 2024, 5:24 AM · Trash

Feb 24 2024

jhathaway added a comment to T358398: corto-test.

butter:

Feb 24 2024, 4:47 PM · Trash

Feb 23 2024

jhathaway added a comment to T358398: corto-test.

well how does PHID-TASK-5tckxnd23nugdvfo3ziu, relate to T358398?

Feb 23 2024, 11:38 PM · Trash
jhathaway added a comment to T355755: Create phabricator bot user for 'corto'.

great, thanks!

Feb 23 2024, 5:32 PM · Incident Tooling
jhathaway created T358355: Integrations tests.
Feb 23 2024, 3:48 PM · Infrastructure-Foundations, Mail, SRE

Feb 22 2024

jhathaway added a comment to T355755: Create phabricator bot user for 'corto'.

@fgiunchedi any update on the phab acccount, should I reach out to someone?

Feb 22 2024, 11:10 PM · Incident Tooling
jhathaway closed T355756: Create & test Google service account for 'corto' as Resolved.

obtained, and seems to work, closing

Feb 22 2024, 11:10 PM · Incident Tooling
jhathaway closed T355756: Create & test Google service account for 'corto', a subtask of T356790: introducing corto internal incident response workflow automation, as Resolved.
Feb 22 2024, 11:10 PM · Incident Tooling, SRE-OnFire

Feb 15 2024

jhathaway added a comment to T357711: Disk space idp1002:9100:/ 4.817% free.

For reasons that are not clear, we stopped compressing on May 30th of 2023 we also don't seem to have any deletion rules in place.

Feb 15 2024, 6:11 PM · Infrastructure-Foundations
jhathaway created T357711: Disk space idp1002:9100:/ 4.817% free.
Feb 15 2024, 6:10 PM · Infrastructure-Foundations

Feb 14 2024

jhathaway added a comment to T338032: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost.

@RoySmith I also asked ITS if we could use phabricator to communicate, since it is accessible by volunteers.

Feb 14 2024, 5:08 PM · Mail, Infrastructure-Foundations, Trust-and-Safety
jhathaway added a comment to T341056: volatile: We need to configure the volatile endpoint on puppetserveres.

thanks for the additional context @Muehlenhoff!

Feb 14 2024, 3:37 PM · Patch-For-Review, Puppet-Infrastructure, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE

Feb 13 2024

jhathaway updated subscribers of T341056: volatile: We need to configure the volatile endpoint on puppetserveres.

@Muehlenhoff I think that makes sense, are the updates run manually when Debian issues a release?

Feb 13 2024, 11:11 PM · Patch-For-Review, Puppet-Infrastructure, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE

Feb 9 2024

jhathaway added a comment to T338032: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost.

@jhathaway Did you open one and have the number?

Feb 9 2024, 6:15 PM · Mail, Infrastructure-Foundations, Trust-and-Safety
jhathaway added a comment to T338032: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost.

@RoySmith did you ever here anything back from the ticket you opened? Also, you said this was mentioned in another forum, in what context, was there another message marked as spam?

Feb 9 2024, 4:29 PM · Mail, Infrastructure-Foundations, Trust-and-Safety

Feb 8 2024

jhathaway added a comment to T356991: Repurpose puppetmaster2003 as puppetserver2003.

This set of steps looks correct to me, based on what @jbond did on T345067

Feb 8 2024, 4:06 PM · Puppet-Infrastructure, Puppet (Puppet 7.0), Infrastructure-Foundations, SRE

Feb 6 2024

jhathaway added a comment to T356221: Ensure donate.wikimedia.org adheres to Google's sender guidelines.

sounds good, happy to help

Feb 6 2024, 10:29 PM · donate.wikimedia.org
jhathaway added a comment to T356221: Ensure donate.wikimedia.org adheres to Google's sender guidelines.

@Jgreen the tool only provides a graph and a percentage, I have added you as a viewer of the domain, https://postmaster.google.com. If that doesn't work I am happy to screen share, or drop some pics on this ticket.

Feb 6 2024, 10:21 PM · donate.wikimedia.org
jhathaway updated subscribers of T356221: Ensure donate.wikimedia.org adheres to Google's sender guidelines.

@Dwisehaupt postmaster tools shows dmarc failures for this domain. Do you know where it is being actively used in emails being sent out?

Feb 6 2024, 9:51 PM · donate.wikimedia.org
jhathaway claimed T356221: Ensure donate.wikimedia.org adheres to Google's sender guidelines.
Feb 6 2024, 9:50 PM · donate.wikimedia.org
jhathaway closed T355449: Ensure wikimedia.org adheres to Google's sender guidelines, a subtask of T355712: Ensure Wikimedia complies with Google's new email sender guidelines, as Resolved.
Feb 6 2024, 9:40 PM · Foundational Technology Requests
jhathaway closed T355449: Ensure wikimedia.org adheres to Google's sender guidelines as Resolved.

other than one-click unsubscribe, we are in compliance

Feb 6 2024, 9:40 PM · Infrastructure-Foundations
jhathaway closed T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines, a subtask of T355712: Ensure Wikimedia complies with Google's new email sender guidelines, as Resolved.
Feb 6 2024, 9:39 PM · Foundational Technology Requests
jhathaway closed T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines as Resolved.

@bcampbell I assume this is resolved, please go ahead and reopen if it is not.

Feb 6 2024, 9:39 PM · Traffic, SRE, DNS, Foundational Technology Requests
jhathaway claimed T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines.
Feb 6 2024, 9:35 PM · Traffic, SRE, DNS, Foundational Technology Requests
jhathaway renamed T355756: Create & test Google service account for 'corto' from Create Google service account for 'corto' to Create & test Google service account for 'corto'.
Feb 6 2024, 4:57 PM · Incident Tooling
jhathaway added a subtask for T356790: introducing corto internal incident response workflow automation: T355758: Create "corto" Phabricator bot account for Corto.
Feb 6 2024, 4:56 PM · Incident Tooling, SRE-OnFire
jhathaway added a parent task for T355758: Create "corto" Phabricator bot account for Corto: T356790: introducing corto internal incident response workflow automation.
Feb 6 2024, 4:56 PM · Release-Engineering-Team (Now this 🫠), User-brennen, Phabricator-Bot-Requests, Incident Tooling
jhathaway added a subtask for T356790: introducing corto internal incident response workflow automation: T355755: Create phabricator bot user for 'corto'.
Feb 6 2024, 4:55 PM · Incident Tooling, SRE-OnFire
jhathaway added a parent task for T355755: Create phabricator bot user for 'corto': T356790: introducing corto internal incident response workflow automation.
Feb 6 2024, 4:55 PM · Incident Tooling
jhathaway added a subtask for T356790: introducing corto internal incident response workflow automation: T355756: Create & test Google service account for 'corto'.
Feb 6 2024, 4:54 PM · Incident Tooling, SRE-OnFire
jhathaway added a parent task for T355756: Create & test Google service account for 'corto': T356790: introducing corto internal incident response workflow automation.
Feb 6 2024, 4:54 PM · Incident Tooling
jhathaway triaged T355756: Create & test Google service account for 'corto' as Medium priority.
Feb 6 2024, 4:15 PM · Incident Tooling
jhathaway moved T355756: Create & test Google service account for 'corto' from Inbox to Prioritized on the Incident Tooling board.
Feb 6 2024, 4:14 PM · Incident Tooling
jhathaway updated subscribers of T355756: Create & test Google service account for 'corto'.

obtained the credentials from @bcampbell, now I need to test them!

Feb 6 2024, 4:14 PM · Incident Tooling

Feb 2 2024

jhathaway updated subscribers of T356459: Password exposure on PCC when adding new host to LB pool?.

@ssingh thanks for the analysis, I am pretty sure it is correct. @bking as to your question, redacting passwords would be preferred. However, before we figure out if that is necessary, @Joe do you think you could weigh in on the use of wmflib::autogen_password, it wasn't clear to me from the git history as to its intended use?

Feb 2 2024, 10:27 PM · Vuln-Infoleak, SecTeam-Processed, Infrastructure-Foundations, Security
jhathaway added a comment to T38937: Allow for group admins to create project repositories inside a parent project.

Has anyone ever looked at the project-group-structure plugin, which allows for access scoped project creation?

Feb 2 2024, 5:01 PM · Upstream, Gerrit

Jan 31 2024

jhathaway lowered the priority of T355449: Ensure wikimedia.org adheres to Google's sender guidelines from High to Medium.
Jan 31 2024, 11:26 PM · Infrastructure-Foundations
jhathaway closed T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines as Resolved.

Since VRTS is sending under <SUPPORT LIST>@wikimedia.org domain, most of what applies to the wikimedia.org domain applies here. Google sees the domain as in compliance with dkim, spf & dmarc.

Jan 31 2024, 6:22 PM · collaboration-services
jhathaway closed T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines, a subtask of T355712: Ensure Wikimedia complies with Google's new email sender guidelines, as Resolved.
Jan 31 2024, 6:22 PM · Foundational Technology Requests
jhathaway updated the task description for T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines.
Jan 31 2024, 6:20 PM · collaboration-services
jhathaway added a comment to T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines.

auth results on my personal gmail instance:

Jan 31 2024, 6:16 PM · collaboration-services
jhathaway added a comment to T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines.

@bcampbell patch is merged, if you want to give it a whirl!

Jan 31 2024, 5:33 PM · Traffic, SRE, DNS, Foundational Technology Requests

Jan 30 2024

jhathaway added a comment to T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines.

@bcampbell patch is out for review

Jan 30 2024, 10:50 PM · Traffic, SRE, DNS, Foundational Technology Requests
jhathaway created T356221: Ensure donate.wikimedia.org adheres to Google's sender guidelines.
Jan 30 2024, 9:35 PM · donate.wikimedia.org
jhathaway added a comment to T355691: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines.

ah, that seems like a bug? However, dkim signature should be okay, since our dmarc record is set to relaxed. Which allows subdomain matching, i.e. dkim: wikimedia.org aligns with from: no-reply@phabricator.wikimedia.org.

Jan 30 2024, 8:52 PM · Release-Engineering-Team (Radar), collaboration-services, Upstream, Phabricator (Upstream), User-brennen, Infrastructure-Foundations
jhathaway added a comment to T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines.

After reading this post on reddit, I think I finally grok the intended setup, apologies for being a bit dense on this one:

Jan 30 2024, 8:40 PM · Traffic, SRE, DNS, Foundational Technology Requests
jhathaway added a comment to T355691: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines.

Is it known/ok that Phab is sending mail from phabricator.wikimedia.org but the DKIM signature is for wikimedia.org?

Jan 30 2024, 7:38 PM · Release-Engineering-Team (Radar), collaboration-services, Upstream, Phabricator (Upstream), User-brennen, Infrastructure-Foundations
jhathaway added a comment to T356118: Google Workspace for the Foundation does not check dmarc.

Does it matter that the mx*.wikimedia.org servers do not currently add an Authentication-Results header?

Jan 30 2024, 4:00 PM · Foundational Technology Requests
jhathaway claimed T356185: operations/deployment-charts can't be checked out on Windows due to a file named aux.yaml.
Jan 30 2024, 3:33 PM · SRE
jhathaway added a comment to T356118: Google Workspace for the Foundation does not check dmarc.

I believe so, for instance here what I get from a mail from services.discover.com on my personal account:

Jan 30 2024, 3:29 PM · Foundational Technology Requests

Jan 29 2024

jhathaway updated subscribers of T356118: Google Workspace for the Foundation does not check dmarc.

@bcampbell do you have any idea why this might be? Not a critical issue, but it is nice to be able to check how a single message is validated by Google.

Jan 29 2024, 10:12 PM · Foundational Technology Requests
jhathaway created T356118: Google Workspace for the Foundation does not check dmarc.
Jan 29 2024, 10:11 PM · Foundational Technology Requests
jhathaway added a comment to T355833: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines.

It appears to point to an SPF record:

Jan 29 2024, 4:58 PM · Traffic, SRE, DNS, Foundational Technology Requests
jhathaway triaged T355691: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines as High priority.
Jan 29 2024, 4:24 PM · Release-Engineering-Team (Radar), collaboration-services, Upstream, Phabricator (Upstream), User-brennen, Infrastructure-Foundations
jhathaway claimed T352604: Improve early detection of hiera data type errors.
Jan 29 2024, 4:11 PM · Puppet-Infrastructure, Puppet-Core, Infrastructure-Foundations, User-jbond
jhathaway claimed T350008: Puppet: drop realm.pp from the manifest directory.
Jan 29 2024, 4:01 PM · Patch-For-Review, Infrastructure-Foundations, Puppet-Core
jhathaway claimed T308639: Remove legacy functions.
Jan 29 2024, 3:59 PM · Puppet-Core, Patch-For-Review, Infrastructure-Foundations, User-jbond
jhathaway claimed T351027: Received 4 mail notifications but only 2 actual mails.
Jan 29 2024, 3:45 PM · Infrastructure-Foundations, Mail
jhathaway added a comment to T351027: Received 4 mail notifications but only 2 actual mails.

@AlexisJazz were you able to reproduce?

Jan 29 2024, 3:45 PM · Infrastructure-Foundations, Mail
jhathaway claimed T347831: Exim: add lists and auto-generated headers.
Jan 29 2024, 3:41 PM · Infrastructure-Foundations, Mail
jhathaway triaged T355450: mediawiki support for one-click unsubscribe as High priority.
Jan 29 2024, 3:39 PM · MediaWiki-Platform-Team, User-ArielGlenn, MediaWiki-Engineering, Growth-Team, Notifications, MediaWiki-Email
jhathaway triaged T320937: admin: Add validation checks for missing realname and email in data.yaml as Low priority.
Jan 29 2024, 3:38 PM · Infrastructure-Foundations, Patch-For-Review, SRE
jhathaway triaged T319041: Remove wikivoyage-ev.org mail aliases from wikivoyage.org & wikivoyage.de as Low priority.
Jan 29 2024, 3:38 PM · Mail, Infrastructure-Foundations
jhathaway triaged T226884: Some emails coming from Gerrit are being tagged as suspicious by Gmail as Medium priority.
Jan 29 2024, 3:38 PM · Infrastructure-Foundations, Mail
jhathaway triaged T306472: Troubleshooting Mail Delivery Issues from Coupa as Medium priority.
Jan 29 2024, 3:37 PM · Infrastructure-Foundations, Mail
jhathaway triaged T207650: Email sent from wikipedia UI seems to use nondeliverable sender: 550 Administrative prohibition as Medium priority.
Jan 29 2024, 3:37 PM · Infrastructure-Foundations, Mail
jhathaway triaged T325394: Replace Exim with Postfix on mail servers as Medium priority.
Jan 29 2024, 3:36 PM · Infrastructure-Foundations, Mail, SRE
jhathaway triaged T325403: MTA Provisioning as Medium priority.
Jan 29 2024, 3:36 PM · Infrastructure-Foundations, Mail, SRE
jhathaway triaged T325405: Provision mta-outbound-lists as Medium priority.
Jan 29 2024, 3:36 PM · Infrastructure-Foundations, Mail, SRE
jhathaway triaged T325404: Provision mta-inbound-lists as Low priority.
Jan 29 2024, 3:36 PM · Infrastructure-Foundations, Mail, SRE
jhathaway triaged T325395: Puppetry as Medium priority.
Jan 29 2024, 3:35 PM · Patch-For-Review, Infrastructure-Foundations, Mail, SRE