Page MenuHomePhabricator

herron (Keith Herron)
Ops Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
May 30 2017, 5:25 PM (137 w, 3 d)
Availability
Available
IRC Nick
herron
LDAP User
Herron
MediaWiki User
Unknown

Recent Activity

Wed, Jan 15

herron reassigned T239732: (No Need By Date Provided) codfw: rack/setup/install puppetmaster2003.codfw.wmnet from Papaul to jbond.

@herron thanks in that case you can just add the server to site.pp with the role ( spare::system) and assign the task to @jbond

Wed, Jan 15, 9:03 PM · Operations, ops-codfw
herron added a comment to T239732: (No Need By Date Provided) codfw: rack/setup/install puppetmaster2003.codfw.wmnet.

Hey @Papaul, I don't think there is any specific urgency to this and it can wait until he's back, but if it needs to go sooner I could work on it.

Wed, Jan 15, 8:36 PM · Operations, ops-codfw
herron created T242885: Expand Eqiad Ganeti row_A capacity.
Wed, Jan 15, 4:30 PM · hardware-requests, Operations

Tue, Jan 14

herron added a comment to T242770: Logstash for MediaWiki is down in Beta Cluster.

This should be fixed now.

Tue, Jan 14, 5:37 PM · observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure

Wed, Jan 8

herron added a comment to T240906: CA App Synthetic Monitor Mail (SMTP): Connection timed out; connect(): -2.

I'd like to rule out possible hardware issues by migrating this VM to another Ganeti host, and seeing if that makes any improvement.

Wed, Jan 8, 4:17 PM · Operations, Mail
herron added a comment to T228924: rack/setup/install ganeti10([09]|1[0-8]).eqiad.wmnet.

The row_A ganeti group is running low on memory capacity (please see T239151#5707691) . Should we allocate a few of these new hosts to expand the existing row_A ganeti group?

Wed, Jan 8, 4:10 PM · ops-eqiad, vm-requests, Operations

Tue, Jan 7

herron added a comment to T228099: rack/setup/install ganeti500[123].eqsin.wmnet.

The eqsin ganeti cluster is now up and running, and a first VM netflow5001 has been created.

Tue, Jan 7, 10:38 PM · Operations
herron updated the task description for T228099: rack/setup/install ganeti500[123].eqsin.wmnet.
Tue, Jan 7, 10:35 PM · Operations
herron added a comment to T239151: Gerrit VM to test data migration.

ganeti-test.wikimedia.org VM has been created on row_C, and I've uploaded a patch to assign it role::gerrit with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562587/

Tue, Jan 7, 7:22 PM · Patch-For-Review, Gerrit, Operations, vm-requests

Fri, Jan 3

herron added a comment to T240906: CA App Synthetic Monitor Mail (SMTP): Connection timed out; connect(): -2.

There has been multiple of mx1001 issues lately (even if that is unreliable, it is worth noting). My suggestion would be, at least initially, to detect the same issue, if real, on icinga.

Fri, Jan 3, 11:00 PM · Operations, Mail
herron triaged T240341: redirect non-existing wikimania2020.wikimedia.org to wikimania.wikimedia.org as Medium priority.
Fri, Jan 3, 8:33 PM · Traffic, Operations, DNS
herron triaged T240495: investigate making 'notrack' the default on our ferm rules as Medium priority.
Fri, Jan 3, 8:33 PM · Operations
herron triaged T240824: PHP Fatal error: Allowed memory size of 524288000 bytes exhausted (tried to allocate 20480 bytes) in /var/www/php-monitoring/lib.php on line 35 as Medium priority.
Fri, Jan 3, 7:45 PM · serviceops, Operations
herron triaged T240843: Track services without a native systemd unit as Medium priority.
Fri, Jan 3, 7:44 PM · Operations
herron triaged T241309: Add more detailed instructions to the "sec-advice" page as Medium priority.
Fri, Jan 3, 7:44 PM · Operations, Traffic
herron triaged T241494: Degraded RAID on cloudvirt1014 as High priority.
Fri, Jan 3, 7:44 PM · ops-eqiad, Operations
herron triaged T241719: Migrate Cloud VPS to Puppet 5 / facter 3 as Medium priority.
Fri, Jan 3, 7:43 PM · Operations, cloud-services-team
herron triaged T241838: Requesting access to EventLogging data for knissen as Medium priority.
Fri, Jan 3, 7:43 PM · Operations, SRE-Access-Requests
herron added a comment to T241096: Requesting access to analytics-privatedata-users and researchers for Aroraakhil.

Hi @Nuria, a friendly ping/bump for approval on this. Happy new year!

Fri, Jan 3, 7:42 PM · Operations, SRE-Access-Requests, Research
herron moved T241838: Requesting access to EventLogging data for knissen from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Fri, Jan 3, 7:39 PM · Operations, SRE-Access-Requests
herron updated the task description for T241838: Requesting access to EventLogging data for knissen.
Fri, Jan 3, 7:38 PM · Operations, SRE-Access-Requests
herron added a comment to T240250: Convert the existing access request documentation into a Phab template.

I'd like to edit the form but don't currently have permission. Primarily I'd like to add the clinic duty checklist and clarify a few prerequisites for the requestor to complete. These are things that we currently do manually via back-and-forth comments. Adding them to the template should save time on every request. I'd like to update the template like so:

Fri, Jan 3, 7:31 PM · WMF-CTO-Team-Backlog, Product-Analytics
herron updated the task description for T241838: Requesting access to EventLogging data for knissen.
Fri, Jan 3, 7:14 PM · Operations, SRE-Access-Requests
herron added a comment to T241722: NDA for Superset Request from WMDE Employee - Kris Litson.

Thanks for the update @Kris_Litson_WMDE

Fri, Jan 3, 4:36 PM · Operations, LDAP-Access-Requests
herron placed T240250: Convert the existing access request documentation into a Phab template up for grabs.
Fri, Jan 3, 4:16 PM · WMF-CTO-Team-Backlog, Product-Analytics

Thu, Jan 2

herron reassigned T240929: Migrate archives of the OKFN-hosted Open-GLAM mailing list to Wikimedia's mailman from herron to jcrespo.

Sounds good @jcrespo, please pass back to me when you've received the export and uploaded it to the mailman host and I'll see what I can do to import. Thanks!

Thu, Jan 2, 3:46 PM · Operations, Wikimedia-Mailing-lists
herron moved T241722: NDA for Superset Request from WMDE Employee - Kris Litson from Backlog to NDA Pending on the LDAP-Access-Requests board.
Thu, Jan 2, 3:39 PM · Operations, LDAP-Access-Requests
herron updated subscribers of T241722: NDA for Superset Request from WMDE Employee - Kris Litson.

Hello! Looping in @RStallman-legalteam to coordinate getting your NDA on file.

Thu, Jan 2, 3:36 PM · Operations, LDAP-Access-Requests
herron removed a project from T223463: (2019-09) Create secteam groups in admin.yaml and define permissions: SRE-Access-Requests.

Removing the SRE-Access-Requests project tag for now. Please update and re-add if/when any further action is needed. Thanks!

Thu, Jan 2, 2:55 PM · Operations, Security-Team, Patch-For-Review

Thu, Dec 19

herron added a comment to T241166: Sync new ganeti clusters with netbox.

esams and ulsfo are online now, and eqsin should be shortly. Not sure if it's best to do all at once, or per-site, but wanted to get a task created to keep tabs on it.

Thu, Dec 19, 7:04 PM · Operations, netbox
herron triaged T241166: Sync new ganeti clusters with netbox as Medium priority.
Thu, Dec 19, 7:03 PM · Operations, netbox
herron updated subscribers of T236216: rack/setup/install ganeti300[123].

The esams ganeti cluster is now up and running, and netflow3001 has been created there as a first VM.

Thu, Dec 19, 6:16 PM · Operations, ops-esams
herron added a comment to T236216: rack/setup/install ganeti300[123].

These hosts have been reimaged with buster, certs created, and patches uploaded to enable ganeti on these hosts.

Thu, Dec 19, 5:58 AM · Operations, ops-esams
herron committed rLPRI8cfae6c0f64b: add dummy esams and eqsin ganeti keys to pacify PCC (authored by herron).
add dummy esams and eqsin ganeti keys to pacify PCC
Thu, Dec 19, 5:08 AM
herron added a comment to T228099: rack/setup/install ganeti500[123].eqsin.wmnet.

Hey @RobH, T229243 is encouraging. How are these hosts looking now?

Thu, Dec 19, 4:56 AM · Operations
herron added a comment to T226444: rack/setup/install ganeti400[123].

Actually since netflow4001 is not yet puppetized the instance has been shut down. https://gerrit.wikimedia.org/r/559330 should unblock the first puppet run, and can re-start the instance after its merged.

Thu, Dec 19, 4:50 AM · Traffic, Operations
herron added a comment to T226444: rack/setup/install ganeti400[123].

For sure, but its a work in progress currently. Basically I'd like a sanity check that the manual steps make sense and aren't already automated, or are better handled, in a way that I'm not aware of.

Thu, Dec 19, 3:43 AM · Traffic, Operations
herron changed the status of T226444: rack/setup/install ganeti400[123] from Stalled to Open.

The ulsfo buster ganeti cluster is up and running now, and netflow4001 has been created there as a first VM.

Thu, Dec 19, 2:44 AM · Traffic, Operations

Dec 16 2019

herron added a comment to T240906: CA App Synthetic Monitor Mail (SMTP): Connection timed out; connect(): -2.

Looked into these alerts a bit, and pulled the source IP addresses for these checks from watchmouse, but I don't see these IPs appearing in the mx logs. I think it is because the exim mx logs are not currently detailed enough. So I'll make the logs a bit more verbose and review again after more log information has been gathered.

Dec 16 2019, 9:58 PM · Operations, Mail
herron triaged T240906: CA App Synthetic Monitor Mail (SMTP): Connection timed out; connect(): -2 as Medium priority.
Dec 16 2019, 9:52 PM · Operations, Mail
herron committed rLPRIe3dbfc3a2e5c: add dummy ulsfo ganeti RAPI key to pacify PCC (authored by herron).
add dummy ulsfo ganeti RAPI key to pacify PCC
Dec 16 2019, 7:57 PM
herron added a comment to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.

Yes, we will need a second logstash stretch instance, and to migrate the Kafka broker ID from deployment-logstash2 to the new host.

Dec 16 2019, 3:40 PM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure

Dec 10 2019

herron added a comment to T234854: Upgrade ELK Stack.

@elukey hey, yes that's been fixed by making a newer version of curator available to the new clusters. Haven't seen cron errors from these since Dec 5. Thanks for cleaning up the "config does not exist" entries!

Dec 10 2019, 12:27 PM · Operations, Wikimedia-Logstash

Dec 5 2019

herron added a comment to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.

Looking more closely the problem was due to a Broker: Leader not available issue in the deployment-prep kafka logging cluster. After starting deployment-logstash2 back up (the instance had been stopped) logs are flowing again. Longer term we'll likely need another logstash stretch instance and to migrate over the broker id from deployment-logstash2 to the new instance.

Dec 5 2019, 5:33 PM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure
herron added a comment to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.

seeing rsyslog complaining about "omkafka: kafka delivery FAIL" on deployment-prep hosts.

Dec 5 2019, 2:57 PM · Release-Engineering-Team-TODO, observability, Wikimedia-Logstash, Beta-Cluster-Infrastructure

Dec 4 2019

herron updated the task description for T234854: Upgrade ELK Stack.
Dec 4 2019, 3:35 PM · Operations, Wikimedia-Logstash

Nov 26 2019

herron updated subscribers of T239121: VE edit data stopped due to statsv falling over (?) on webperf1001.
Nov 26 2019, 7:39 PM · Performance-Team (Radar), observability, Analytics, Editing-team
herron added a comment to T226444: rack/setup/install ganeti400[123].

Ok, for my own edification, how would the private only LVS model work if we wanted to stand up a public facing non HTTP(S) service in a VM at one+ of these sites?

Nov 26 2019, 3:13 PM · Traffic, Operations
herron added a comment to T226444: rack/setup/install ganeti400[123].

Will this Ganeti cluster use vlan tagged interfaces, or will separate physical interfaces connect to both public and private vlans? If tagging, are the switchports configured for that yet?

Nov 26 2019, 2:55 PM · Traffic, Operations

Nov 19 2019

herron added a comment to T237587: Determine & implement near-term method for escalating network alerts.

Friendly ping to @Volans about @fgiunchedi question above

Nov 19 2019, 9:12 PM · Operations, netops, observability
herron added a comment to T230492: Requesting SRE permissions to create Gerrit projects under operations/debs.

Thanks for the ping, I missed the question. Sure, being added to the Gerrit Manager that would work for me!

Nov 19 2019, 6:08 PM · Gerrit-Privilege-Requests

Nov 15 2019

herron added a comment to T238416: Logstash doesn't parse ulogd source and destination ports.

https://gerrit.wikimedia.org/r/551270 should do the trick for source/dest ports. I don't recall why these weren't parsed out in the first place. While we're at it would any of the other parts the ulogd/iptables events be useful as fields?

Nov 15 2019, 9:32 PM · Operations, observability

Nov 13 2019

herron added a comment to T235891: Ingest production logs with ELK7.

re: bridging the gap with non-kafka inputs, my current thinking is to output all logs with deprecated-input tag back into kafka-logging on a separate topic and consume that from the new cluster. cc @herron @colewhite

Nov 13 2019, 3:54 PM · User-fgiunchedi, Patch-For-Review, Operations, Wikimedia-Logstash

Nov 8 2019

herron updated the task description for T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal).
Nov 8 2019, 9:22 PM · User-herron, Goal, observability
herron closed T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal), a subtask of T228878: Reduce Icinga alert noise, as Resolved.
Nov 8 2019, 9:22 PM · User-fgiunchedi, Goal, observability
herron closed T230236: De-noise ipsec alerts (Reduce Icinga alert noise goal) as Resolved.

https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status probably needs some cleanup (some of the graphs are empty, there's a note there to ignore icinga errors, etc). Also fix missing doc link on the alert?

Nov 8 2019, 9:22 PM · User-herron, Goal, observability

Nov 7 2019

herron added a comment to T236497: cp3056 hardware issue.

Sorry I missed that you already had a patch! But in any case, we only need commenting from cache::nodes to fix up this case (there's no good reason to e.g. churn it out of conftool or the various iptables rules defined from the other stuff).

Nov 7 2019, 4:25 PM · DC-Ops, ops-esams, Operations, Traffic
herron added a comment to T236497: cp3056 hardware issue.

Since it looks like cp3056 might be down for some time could we remove it from the config until fixed? It would be good to let the ipsec checks in icinga return to green.

Nov 7 2019, 3:25 PM · DC-Ops, ops-esams, Operations, Traffic

Nov 6 2019

herron added a comment to T237587: Determine & implement near-term method for escalating network alerts.

In terms of “what” should be escalated, so far we discussed

Nov 6 2019, 10:43 PM · Operations, netops, observability
herron triaged T237587: Determine & implement near-term method for escalating network alerts as Medium priority.
Nov 6 2019, 10:37 PM · Operations, netops, observability

Nov 5 2019

herron closed T233318: scs monitoring missing in Icinga as Resolved.

Yes, I think we're in good shape here

Nov 5 2019, 3:28 PM · Icinga, observability, Operations

Nov 4 2019

herron added a comment to T233318: scs monitoring missing in Icinga.

Host monitoring for SCS systems has been added to icinga

Nov 4 2019, 2:59 PM · Icinga, observability, Operations

Nov 1 2019

herron updated the task description for T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task).
Nov 1 2019, 2:03 PM · User-herron, Operations
herron added a comment to T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.).

Essentially a duplicate of T220390 where audit work has gone into documenting clusters and use cases

Nov 1 2019, 2:02 PM · Operations
herron merged T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.) into T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.
Nov 1 2019, 2:01 PM · Operations
herron merged task T220391: Establish guideline documentation for Kafka cluster use cases (main, jumbo, logging, etc.) into T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.
Nov 1 2019, 2:01 PM · Operations
herron changed the status of T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases, a subtask of T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task), from Open to Stalled.
Nov 1 2019, 2:00 PM · User-herron, Operations
herron changed the status of T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases from Open to Stalled.
Nov 1 2019, 2:00 PM · Operations
herron added a comment to T220390: Audit existing Kafka main producers/consumers and document their configuration and use cases.

The document https://docs.google.com/document/d/1mr217D6eyoGvGUG31M-FVMve9MCOCRKZxUWFZOZirQw/edit#heading=h.mbousz3hsm22 was created & shared via the goal meetings during Q4. Time permitting this probably could use another round of comments to extend/finalize and remove any info that's gone stale by now.

Nov 1 2019, 2:00 PM · Operations
herron updated the task description for T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task).
Nov 1 2019, 1:48 PM · User-herron, Operations
herron closed T226274: (Need By: June 30) rack/setup/install kafka-main100[1-5] as Resolved.
Nov 1 2019, 1:46 PM · User-herron, Operations
herron added a parent task for T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345]: T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Nov 1 2019, 1:46 PM · Patch-For-Review, Services (watching), Core Platform Team Legacy (Watching / External), Analytics, User-herron, Operations
herron added a subtask for T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019.: T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].
Nov 1 2019, 1:46 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron closed T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019., a subtask of T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task), as Resolved.
Nov 1 2019, 1:46 PM · User-herron, Operations
herron closed T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. as Resolved.

To circle back on this, we moved forward with option 2 and are using task T225005 to track the migration effort

Nov 1 2019, 1:46 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron merged T220389: Review current architecture/capacity and establish plan for Kafka main cluster upgrade/refresh to cover needs for next 2-3 years into T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Nov 1 2019, 1:42 PM · CPT Initiatives (Modern Event Platform (TEC2)), User-herron, Services (watching), Event-Platform, Analytics, Operations
herron merged task T220389: Review current architecture/capacity and establish plan for Kafka main cluster upgrade/refresh to cover needs for next 2-3 years into T217359: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019..
Nov 1 2019, 1:42 PM · Operations

Oct 30 2019

herron updated the task description for T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].
Oct 30 2019, 6:38 PM · Patch-For-Review, Services (watching), Core Platform Team Legacy (Watching / External), Analytics, User-herron, Operations
herron removed a watcher for Wikimedia-Mailing-lists: herron.
Oct 30 2019, 3:18 PM
herron removed a watcher for Puppet: herron.
Oct 30 2019, 3:18 PM

Oct 28 2019

herron renamed T236478: update failed puppet checks so that they go critical 24 hours from update failed puppet checkes so that they go critical 24 hours to update failed puppet checks so that they go critical 24 hours.
Oct 28 2019, 7:02 PM · User-jbond, Puppet, Operations, observability
herron updated the task description for T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).
Oct 28 2019, 1:47 PM · DC-Ops, Operations, ops-eqiad

Oct 18 2019

herron added a comment to T235260: Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users).

Hello! @gsingers, as a last step could you please review and sign the L3 document? Once that's done (and the related patch has a +1 from a peer within SRE) we'll be ready to merge and deploy analytics-privatedata-users group membership.

Oct 18 2019, 2:12 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Analytics-Kanban, Analytics
herron renamed T235260: Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users) from Analytics Access for Grant to Analytics Access for Grant (groups cn=wmf and analytics-privatedata-users).
Oct 18 2019, 2:11 PM · LDAP-Access-Requests, SRE-Access-Requests, Operations, Analytics-Kanban, Analytics

Oct 16 2019

herron closed T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin as Resolved.

Transitioning this resolved as all subtasks have now been resolved. If additional follow-up is needed, please don't hesitate to re-open. Thanks!

Oct 16 2019, 2:57 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron added a parent task for T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener: T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin .
Oct 16 2019, 2:56 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a parent task for T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah: T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin .
Oct 16 2019, 2:56 PM · SRE-Access-Requests, Operations
herron added subtasks for T233636: Banner History and page view data access for fundraising analysts - Jerrie and Erin : T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener, T234433: Requesting access to 'analytics-privatedata-users' and 'researchers' for Jerrie Kumalah.
Oct 16 2019, 2:56 PM · Analytics, Operations, SRE-Access-Requests, Fundraising-Backlog
herron closed T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener as Resolved.

The requested group memberships have been provisioned. I'll transition this to resolved now, but please don't hesitate to re-open if any follow up is necessary. Thanks!

Oct 16 2019, 2:54 PM · Patch-For-Review, SRE-Access-Requests, Operations

Oct 11 2019

herron updated the task description for T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.
Oct 11 2019, 8:33 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron updated subscribers of T234209: Grant LDAP groups and deployment shell access to Kevin Bazira.

Great, thank you!

Oct 11 2019, 8:32 PM · SRE-Access-Requests, Operations, LDAP-Access-Requests, Scoring-platform-team
herron closed T234473: Requesting access to analytics cluster for Djellel Difallah as Resolved.

Access has been granted. Transitioning this to resolved now, but if any follow-up is needed please don't hesitate to re-open. Thanks!

Oct 11 2019, 8:29 PM · Research, Operations, SRE-Access-Requests
herron added a comment to T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.

Regarding chat I'd encourage them to reach out with any questions via IRC. Details about available channels and their associated topics can be fount at https://meta.wikimedia.org/wiki/IRC/Channels

Oct 11 2019, 8:28 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a comment to T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.

Hi @Nuria could you please review this group request for approval?

Oct 11 2019, 8:23 PM · Patch-For-Review, SRE-Access-Requests, Operations
herron added a comment to T232417: mass Yahoo / AOL bounces mailman.

! In T232417#5567208, @aezell wrote:
tl:dr; Contacting someone in the abuse department at Yahoo/AOL is probably the best bet to figure this out.

Oct 11 2019, 8:00 PM · Mail, Operations, Wikimedia-Mailing-lists
herron added a comment to T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array.

! In T234564#5565056, @Krinkle wrote:
Would it be possible to give type:mediawiki channel:(error OR exception OR fatal) a separate index as well? These are the only critical ones involved in deployment and should not suffer due to spam from random info/debug channels.
We might want to include type:syslog program:php72-fpm and type:scap in there as well.

Oct 11 2019, 7:18 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (201910), User-Ryasmeen, MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), Patch-For-Review, Wikimedia-production-error, Performance-Team (Radar), Deployments, Wikimedia-Logstash, VisualEditor
herron updated the task description for T234529: Requesting access to 'analytics-privatedata-users' and 'researchers' for Erin Yener.
Oct 11 2019, 5:18 PM · Patch-For-Review, SRE-Access-Requests, Operations

Oct 10 2019

herron updated the task description for T234854: Upgrade ELK Stack.
Oct 10 2019, 4:29 PM · Operations, Wikimedia-Logstash
herron moved T235136: LDAP membership for new employee Nikki Nikkhoui from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Oct 10 2019, 4:01 PM · Operations, LDAP-Access-Requests
herron added a comment to T235136: LDAP membership for new employee Nikki Nikkhoui.

Hello, could you please expand on this request? What resources are meant to be accessed, and do you know specifically what LDAP group?

Oct 10 2019, 4:00 PM · Operations, LDAP-Access-Requests