Page MenuHomePhabricator

herron (Keith Herron)
Ops Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
May 30 2017, 5:25 PM (128 w, 10 h)
Availability
Available
IRC Nick
herron
LDAP User
Herron
MediaWiki User
Unknown

Recent Activity

Fri, Nov 8

herron updated the task description for T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal).
Fri, Nov 8, 9:22 PM · Patch-For-Review, User-herron, Goal, observability
herron closed T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal), a subtask of T228878: Reduce Icinga alert noise, as Resolved.
Fri, Nov 8, 9:22 PM · User-fgiunchedi, Goal, observability
herron closed T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal) as Resolved.

https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status probably needs some cleanup (some of the graphs are empty, there's a note there to ignore icinga errors, etc). Also fix missing doc link on the alert?

Fri, Nov 8, 9:22 PM · Patch-For-Review, User-herron, Goal, observability

Thu, Nov 7

herron added a comment to T236497: cp3056 hardware issue.

Sorry I missed that you already had a patch! But in any case, we only need commenting from cache::nodes to fix up this case (there's no good reason to e.g. churn it out of conftool or the various iptables rules defined from the other stuff).

Thu, Nov 7, 4:25 PM · DC-Ops, ops-esams, Traffic, Operations
herron added a comment to T236497: cp3056 hardware issue.

Since it looks like cp3056 might be down for some time could we remove it from the config until fixed? It would be good to let the ipsec checks in icinga return to green.

Thu, Nov 7, 3:25 PM · DC-Ops, ops-esams, Traffic, Operations

Wed, Nov 6

herron added a comment to T237587: Determine & implement near-term method for escalating network alerts.

In terms of “what” should be escalated, so far we discussed

Wed, Nov 6, 10:43 PM · Operations, netops, observability
herron triaged T237587: Determine & implement near-term method for escalating network alerts as Normal priority.
Wed, Nov 6, 10:37 PM · Operations, netops, observability

Tue, Nov 5

herron closed T233318: scs monitoring missing in Icinga as Resolved.

Yes, I think we're in good shape here

Tue, Nov 5, 3:28 PM · Icinga, observability, Operations

Mon, Nov 4

herron added a comment to T233318: scs monitoring missing in Icinga.

Host monitoring for SCS systems has been added to icinga

Mon, Nov 4, 2:59 PM · Icinga, observability, Operations

Fri, Nov 1

herron updated the task description for T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task).
Fri, Nov 1, 2:03 PM · User-herron, Operations
herron added a comment to T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.).

Essentially a duplicate of T220390 where audit work has gone into documenting clusters and use cases

Fri, Nov 1, 2:02 PM · Operations
herron merged T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.) into T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.
Fri, Nov 1, 2:01 PM · Operations
herron merged task T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.) into T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.
Fri, Nov 1, 2:01 PM · Operations
herron changed the status of T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases, a subtask of T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task), from Open to Stalled.
Fri, Nov 1, 2:00 PM · User-herron, Operations
herron changed the status of T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases from Open to Stalled.
Fri, Nov 1, 2:00 PM · Operations
herron added a comment to T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.

The document https://docs.google.com/document/d/1mr217D6eyoGvGUG31M-FVMve9MCOCRKZxUWFZOZirQw/edit#heading=h.mbousz3hsm22 was created & shared via the goal meetings during Q4. Time permitting this probably could use another round of comments to extend/finalize and remove any info that's gone stale by now.

Fri, Nov 1, 2:00 PM · Operations
herron updated the task description for T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task).
Fri, Nov 1, 1:48 PM · User-herron, Operations
herron closed T226274: (Need By: June 30) rack/setup/install kafka-main100[1-5] as Resolved.
Fri, Nov 1, 1:46 PM · User-herron, Operations
herron added a parent task for T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345]: T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Fri, Nov 1, 1:46 PM · Patch-For-Review, Services (watching), Core Platform Team Legacy (Watching / External), Analytics, User-herron, Operations
herron added a subtask for T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019.: T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].
Fri, Nov 1, 1:46 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron closed T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019., a subtask of T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task), as Resolved.
Fri, Nov 1, 1:46 PM · User-herron, Operations
herron closed T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. as Resolved.

To circle back on this, we moved forward with option 2 and are using task T225005 to track the migration effort

Fri, Nov 1, 1:46 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron merged T220389: Review current architecture/capacity and establish plan for Kafka main cluster upgrade/refresh to cover needs for next 2-3 years into T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Fri, Nov 1, 1:42 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron merged task T220389: Review current architecture/capacity and establish plan for Kafka main cluster upgrade/refresh to cover needs for next 2-3 years into T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Fri, Nov 1, 1:42 PM · Operations

Wed, Oct 30

herron updated the task description for T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].
Wed, Oct 30, 6:38 PM · Patch-For-Review, Services (watching), Core Platform Team Legacy (Watching / External), Analytics, User-herron, Operations
herron removed a watcher for Wikimedia-Mailing-lists: herron.
Wed, Oct 30, 3:18 PM
herron removed a watcher for Puppet: herron.
Wed, Oct 30, 3:18 PM

Mon, Oct 28

herron renamed T236478: update failed puppet checks so that they go critical 24 hours from update failed puppet checkes so that they go critical 24 hours to update failed puppet checks so that they go critical 24 hours.
Mon, Oct 28, 7:02 PM · User-jbond, Puppet, Operations, observability
herron updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Mon, Oct 28, 1:47 PM · DC-Ops, Operations, ops-eqiad

Fri, Oct 18

herron added a comment to T235260: Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users).

Hello! @gsingers, as a last step could you please review and sign the L3 document? Once that's done (and the related patch has a +1 from a peer within SRE) we'll be ready to merge and deploy analytics-privatedata-users group membership.

Fri, Oct 18, 2:12 PM · LDAP-Access-Requests, Operations, SRE-Access-Requests, Analytics-Kanban, Analytics
herron renamed T235260: Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users) from Analytics Access for Grant to Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users).
Fri, Oct 18, 2:11 PM · LDAP-Access-Requests, Operations, SRE-Access-Requests, Analytics-Kanban, Analytics

Wed, Oct 16

herron closed T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin as Resolved.

Transitioning this resolved as all subtasks have now been resolved. If additional follow-up is needed, please don't hesitate to re-open. Thanks!

Wed, Oct 16, 2:57 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron added a parent task for T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener: T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin .
Wed, Oct 16, 2:56 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a parent task for T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah: T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin .
Wed, Oct 16, 2:56 PM · SRE-Access-Requests, Operations
herron added subtasks for T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin : T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener, T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah.
Wed, Oct 16, 2:56 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron closed T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener as Resolved.

The requested group memberships have been provisioned. I'll transition this to resolved now, but please don't hesitate to re-open if any follow up is necessary. Thanks!

Wed, Oct 16, 2:54 PM · Patch-For-Review, SRE-Access-Requests, Operations

Oct 11 2019

herron updated the task description for T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.
Oct 11 2019, 8:33 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron updated subscribers of T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.

Great, thank you!

Oct 11 2019, 8:32 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron closed T234473: Requesting access to analytics cluster for Djellel Difallah as Resolved.

Access has been granted. Transitioning this to resolved now, but if any follow-up is needed please don't hesitate to re-open. Thanks!

Oct 11 2019, 8:29 PM · Research, SRE-Access-Requests, Operations
herron added a comment to T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.

Regarding chat I'd encourage them to reach out with any questions via IRC. Details about available channels and their associated topics can be fount at https://meta.wikimedia.org/wiki/IRC/Channels

Oct 11 2019, 8:28 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a comment to T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.

Hi @Nuria could you please review this group request for approval?

Oct 11 2019, 8:23 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a comment to T232417: mass Yahoo / AOL bounces mailman.

! In T232417#5567208, @aezell wrote:
tl:dr; Contacting someone in the abuse department at Yahoo/AOL is probably the best bet to figure this out.

Oct 11 2019, 8:00 PM · Mail, Operations, Wikimedia-Mailing-lists
herron added a comment to T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array.

! In T234564#5565056, @Krinkle wrote:
Would it be possible to give type:mediawiki channel:(error OR exception OR fatal) a separate index as well? These are the only critical ones involved in deployment and should not suffer due to spam from random info/debug channels.
We might want to include type:syslog program:php72-fpm and type:scap in there as well.

Oct 11 2019, 7:18 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), User-Ryasmeen, MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), Patch-For-Review, Wikimedia-production-error, Performance-Team (Radar), Deployments, Wikimedia-Logstash, VisualEditor
herron updated the task description for T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.
Oct 11 2019, 5:18 PM · Patch-For-Review, SRE-Access-Requests, Operations

Oct 10 2019

herron updated the task description for T234854: Upgrade ELK Stack.
Oct 10 2019, 4:29 PM · Patch-For-Review, Operations, Wikimedia-Logstash
herron moved T235136: LDAP membership for new employee Nikki Nikkhoui from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Oct 10 2019, 4:01 PM · Operations, LDAP-Access-Requests
herron added a comment to T235136: LDAP membership for new employee Nikki Nikkhoui.

Hello, could you please expand on this request? What resources are meant to be accessed, and do you know specifically what LDAP group?

Oct 10 2019, 4:00 PM · Operations, LDAP-Access-Requests
herron moved T234429: Requesting access to view EventLogging data for Co_WMDE from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Oct 10 2019, 3:59 PM · WMF-Legal, Operations, SRE-Access-Requests
herron added a comment to T234429: Requesting access to view EventLogging data for Co_WMDE.

Hello, @CorinnaHillebrand_WMDE, could you please review and sign the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document, update the task description with your desired shell username and SSH public key (must be a unique key only to be used in wmf production), and coordinate a comment of approval on this task from your manager?

Oct 10 2019, 3:58 PM · WMF-Legal, Operations, SRE-Access-Requests
herron updated the task description for T234429: Requesting access to view EventLogging data for Co_WMDE.
Oct 10 2019, 3:53 PM · WMF-Legal, Operations, SRE-Access-Requests
herron moved T234209: Grant LDAP groups and deployment shell access to Kevin Bazira from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Oct 10 2019, 3:48 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron moved T234209: Grant LDAP groups and deployment shell access to Kevin Bazira from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Oct 10 2019, 3:48 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron updated the task description for T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.
Oct 10 2019, 3:48 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron updated the task description for T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.
Oct 10 2019, 3:48 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron added a comment to T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.

@kevinbazira could you please review and sign the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document, add details to the task description outlining high level reasoning for the access, and coordinate a comment of approval from your manager?

Oct 10 2019, 3:47 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron moved T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Oct 10 2019, 3:28 PM · SRE-Access-Requests, Operations
herron updated subscribers of T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah.

Hi @Nuria could you please review this for approval?

Oct 10 2019, 3:28 PM · SRE-Access-Requests, Operations
herron moved T234473: Requesting access to analytics cluster for Djellel Difallah from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Oct 10 2019, 3:16 PM · Research, SRE-Access-Requests, Operations
herron moved T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Oct 10 2019, 3:16 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a comment to T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.

@EYener in the task description it looks like the ssh key fingerprint was provided, instead of the ssh public key itself. Could you please update with your ssh public key? It should begin with "ssh-ed25519" or "ssh-rsa" . Thanks in advance!

Oct 10 2019, 3:16 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron updated the task description for T234473: Requesting access to analytics cluster for Djellel Difallah.
Oct 10 2019, 3:01 PM · Research, SRE-Access-Requests, Operations
herron added a comment to T234473: Requesting access to analytics cluster for Djellel Difallah.

Hello, I've uploaded a patch set for this access. Typically yes @Nuria approves additions to analytics groups. Once that's done we should be in good shape to move forward. Thanks!

Oct 10 2019, 3:01 PM · Research, SRE-Access-Requests, Operations

Oct 9 2019

herron added a project to T235124: Move kafka100[123] to logstash102[012]: DC-Ops.
Oct 9 2019, 8:10 PM · DC-Ops, Operations, ops-eqiad
herron triaged T235125: Move kafka200[123] to logstash202[012] as Normal priority.
Oct 9 2019, 8:10 PM · DC-Ops, Operations, ops-codfw
herron triaged T235124: Move kafka100[123] to logstash102[012] as Normal priority.
Oct 9 2019, 8:09 PM · DC-Ops, Operations, ops-eqiad
herron added a comment to T233883: disable WMFSF, keep archives.

Hi @Varnent, the old list address is disabled and messages sent there will held in moderation indefinitely. The communication mail that was sent out about this IMO is clear that the old list address has been replaced with the new address. But to help with usability I've added an auto-response to remind users who email the old address of the change, just in case. Also, if desired, current list admins can still log in and update the description that users see via the web, and tend to any moderated messages during the transition time.

Oct 9 2019, 7:56 PM · Operations, Wikimedia-Mailing-lists
herron added a comment to T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array.

Actually pulling an example raw message from kafka should answer my own question about an example problem message

Oct 9 2019, 7:10 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), User-Ryasmeen, MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), Patch-For-Review, Wikimedia-production-error, Performance-Team (Radar), Deployments, Wikimedia-Logstash, VisualEditor
herron added a comment to T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array.

Looking backwards through logs I see:

Oct 9 2019, 5:23 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), User-Ryasmeen, MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), Patch-For-Review, Wikimedia-production-error, Performance-Team (Radar), Deployments, Wikimedia-Logstash, VisualEditor

Oct 8 2019

herron updated the task description for T234854: Upgrade ELK Stack.
Oct 8 2019, 3:14 PM · Patch-For-Review, Operations, Wikimedia-Logstash
herron closed T233883: disable WMFSF, keep archives as Resolved.

Hello, the WMFSF list has been disabled and archives will remain in place. I'll transition to resolved now. Thanks!

Oct 8 2019, 1:54 PM · Operations, Wikimedia-Mailing-lists

Oct 7 2019

herron triaged T234854: Upgrade ELK Stack as Normal priority.
Oct 7 2019, 8:05 PM · Patch-For-Review, Operations, Wikimedia-Logstash

Oct 3 2019

herron awarded T224033: Fix operations/puppet.git "rebase hell" a Like token.
Oct 3 2019, 3:52 PM · Release-Engineering-Team (Development services), Gerrit, Release-Engineering-Team-TODO, Continuous-Integration-Config, Operations

Oct 2 2019

herron closed T233134: logstash-beta.wmflabs.org does not receive any mediawiki events as Resolved.

This occurred in prod as well after host reboots, and a fix has been deployed in puppet. The fix moves include ::profile::rsyslog::udp_json_logback_compat from profile::elasticsearch to profile::elasticsearch::cirrus which prevents the rsyslog udp 11514 listener from being deployed to logstash collectors where it conflicts with logstash. I think we're in good shape here now, setting this to resolved.

Oct 2 2019, 3:50 PM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure

Oct 1 2019

herron added a comment to T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin .

There's two issues with the patch merged for Erin Yener: (1) If contractors have a @wikimedia.org address, they should be added to cn=wmf, not cn=nda. (2) Contractors need an entry in data.yaml with the contract end and a person of contact (expiry_date, expiry_contact fields). Otherwise we'll miss dropping their credentials when the contract expires (we ping the point of contact one week before the contract expires and will extend access if the contract is contuining)

Oct 1 2019, 7:54 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron closed T233235: Add Urbanecm to #mediawiki_security as Resolved.

done!

Oct 1 2019, 4:10 PM · Operations
herron awarded T233991: Vendor's Emails Not Coming Through a Like token.
Oct 1 2019, 4:05 PM · Operations, Mail

Sep 30 2019

herron closed T233780: Turnilo access for Jerrie Kumalah and Erin Yener (fundraising analysts), a subtask of T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin , as Resolved.
Sep 30 2019, 7:22 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron closed T233780: Turnilo access for Jerrie Kumalah and Erin Yener (fundraising analysts) as Resolved.

The resolution of parent task T233636 should address this ask as well. Please re-open if any follow up is needed. Thanks!

Sep 30 2019, 7:22 PM · Operations, LDAP-Access-Requests
herron closed T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin as Resolved.

jkumalah has been added to ldap group wmf, eyener has been added to ldap group nda, and both added to ldap_only_users via puppet.

Sep 30 2019, 7:20 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron awarded T207200: Revisit the logging work done on Q1 2017-2018 for the standard pod setup a Party Time token.
Sep 30 2019, 4:52 PM · serviceops, Release-Engineering-Team (Pipeline), Release-Engineering-Team-TODO, Core Platform Team Legacy (Watching / External), Services (watching), Release Pipeline, Operations
herron added a comment to T233828: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash.

Copying excepetion.message to message looks to have made an improvement here. I see no results in the past 15 minutes when querying for normalized_message:%{messsage}

Sep 30 2019, 2:56 PM · Patch-For-Review, Wikimedia-Logstash, serviceops, Operations, observability

Sep 27 2019

herron added a comment to T233828: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash.

A few ideas to address this:

Sep 27 2019, 9:17 PM · Patch-For-Review, Wikimedia-Logstash, serviceops, Operations, observability
herron added a comment to T233828: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash.

These logs appear to be nesting the message field inside the exception field, and the message field at the root is not present. Which should explain why normalized_message is containing a literal %{message}.

Sep 27 2019, 9:05 PM · Patch-For-Review, Wikimedia-Logstash, serviceops, Operations, observability
herron triaged T233883: disable WMFSF, keep archives as Normal priority.
Sep 27 2019, 7:20 PM · Operations, Wikimedia-Mailing-lists
herron triaged T233839: Puppet systemd::mask is an anti pattern that has unwanted side effect as Normal priority.
Sep 27 2019, 7:19 PM · serviceops, Traffic, Puppet, Operations
herron triaged T233843: Convert glam@wikimedia.org OTRS into a Google Group as Normal priority.
Sep 27 2019, 7:17 PM · Office-IT, Operations, OTRS
herron triaged T233991: Vendor's Emails Not Coming Through as Normal priority.
Sep 27 2019, 7:17 PM · Operations, Mail
herron added a comment to T233991: Vendor's Emails Not Coming Through.

Hello, yes generally speaking based upon the production mail logs I am seeing mail from lawroom.com being accepted and sent onwards to google for final delivery. Messages from this domain appears to occur in bursts, with the majority of recent activity in the logs on Sept 17th, 23rd, and 25th.

Sep 27 2019, 7:16 PM · Operations, Mail
herron triaged T234047: Extend firewall rules for new corp LDAP replicas as Normal priority.
Sep 27 2019, 6:34 PM · Operations
herron closed T216172: Set up basic email infra for w.wiki domain as Resolved.

Thanks for the ping/reminder! Basic aliasing for w.wiki has been deployed and successfully tested.

Sep 27 2019, 5:16 PM · Traffic, Operations, Mail

Sep 26 2019

herron added a project to T233828: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash: Wikimedia-Logstash.
Sep 26 2019, 5:20 PM · Patch-For-Review, Wikimedia-Logstash, serviceops, Operations, observability
herron triaged T233921: Further steps for CAS/web SSO as Normal priority.
Sep 26 2019, 5:19 PM · User-jbond, Operations
herron triaged T233930: Create a staging environment for CAS as Normal priority.
Sep 26 2019, 5:19 PM · User-jbond, Operations
herron triaged T233931: Cross data center setup for CAS as Normal priority.
Sep 26 2019, 5:19 PM · User-jbond, Operations
herron triaged T233933: Replicated ticket registry as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Operations
herron triaged T233934: Collects metrics for CAS as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Operations
herron triaged T233935: Icinga Monitoring for CAS as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Operations
herron triaged T233936: Integrate CAS into backup infrastructure as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Operations
herron triaged T233937: Add U2F/FIDO as second factor for CAS as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Patch-For-Review, Operations
herron triaged T233938: SSO kill switch for crucial services as Normal priority.
Sep 26 2019, 5:18 PM · User-jbond, Operations