Page MenuHomePhabricator

elukey (Luca Toscano)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Jan 5 2016, 9:54 PM (183 w, 6 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
LToscano (WMF) [ Global Accounts ]

Recent Activity

Today

elukey added a comment to T227141: a5-eqiad pdu refresh.

A single Hadoop worker node for analytics, all good.

Tue, Jul 16, 10:05 AM · DC-Ops, Operations, ops-eqiad
elukey updated the task description for T227141: a5-eqiad pdu refresh.
Tue, Jul 16, 10:05 AM · DC-Ops, Operations, ops-eqiad
elukey added a comment to T227140: a4-eqiad pdu refresh.

I replaced the Analytics tag for kafka1001 with @herron since the kafka main cluster is now handled by infrastructure foundations.

Tue, Jul 16, 10:04 AM · DC-Ops, Operations, ops-eqiad
elukey updated the task description for T227140: a4-eqiad pdu refresh.
Tue, Jul 16, 10:02 AM · DC-Ops, Operations, ops-eqiad
elukey updated the task description for T227140: a4-eqiad pdu refresh.
Tue, Jul 16, 10:02 AM · DC-Ops, Operations, ops-eqiad
elukey updated the task description for T227140: a4-eqiad pdu refresh.
Tue, Jul 16, 10:01 AM · DC-Ops, Operations, ops-eqiad
elukey added a comment to T227139: a3-eqiad pdu refresh.

All the analytics nodes are hadoop workers, not a big deal if they loose power.

Tue, Jul 16, 10:00 AM · DC-Ops, Operations, ops-eqiad
elukey added a comment to T227138: a2-eqiad pdu refresh.

The kafka hosts are going to be decommed in T226517, so not a concern. The other hosts can go down without horrible consequences :)

Tue, Jul 16, 9:57 AM · DC-Ops, Operations, ops-eqiad
elukey added a comment to T225642: Allow async foreign set/delete WAN cache operations in mcrouter.

We can check per-hosts metrics with:

Tue, Jul 16, 9:27 AM · User-Elukey, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
elukey added a comment to T225642: Allow async foreign set/delete WAN cache operations in mcrouter.

In light of what I wrote above:

Tue, Jul 16, 9:22 AM · User-Elukey, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
elukey added a comment to T225642: Allow async foreign set/delete WAN cache operations in mcrouter.

All right this is a clear PEBCAK (problem exists between computer and keyboard). First of all my tests were not correct:

Tue, Jul 16, 8:57 AM · User-Elukey, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
elukey added a comment to T225642: Allow async foreign set/delete WAN cache operations in mcrouter.

This is happening on mw1261 (currently depooled):

Tue, Jul 16, 8:01 AM · User-Elukey, Patch-For-Review, Performance-Team (Radar), Availability (MediaWiki-MultiDC), Operations
elukey moved T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN from In Progress to Done on the Analytics-Kanban board.
Tue, Jul 16, 6:59 AM · Analytics-Kanban, Analytics
elukey set the point value for T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN to 5.
Tue, Jul 16, 6:59 AM · Analytics-Kanban, Analytics
elukey added a comment to T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN.

The services in analytics that use LDAP are:

Tue, Jul 16, 6:59 AM · Analytics-Kanban, Analytics

Yesterday

elukey moved T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports. from In Progress to Done on the User-Elukey board.
Mon, Jul 15, 4:40 PM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey moved T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports. from In Progress to Done on the Analytics-Kanban board.
Mon, Jul 15, 2:57 PM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey set the point value for T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports. to 5.
Mon, Jul 15, 2:57 PM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey moved T227941: mylvmbackup on an-coord1001 not working from Next Up to Done on the Analytics-Kanban board.
Mon, Jul 15, 2:54 PM · Analytics-Kanban, Analytics
elukey claimed T227941: mylvmbackup on an-coord1001 not working.
Mon, Jul 15, 2:54 PM · Analytics-Kanban, Analytics
elukey changed the point value for T227941: mylvmbackup on an-coord1001 not working from 0 to 3.
Mon, Jul 15, 2:54 PM · Analytics-Kanban, Analytics
elukey added a comment to T227941: mylvmbackup on an-coord1001 not working.

To keep archives happy:

Mon, Jul 15, 2:54 PM · Analytics-Kanban, Analytics
elukey added a comment to T226035: Dropping data from druid takes down aqs hosts .

@Nuria probably yes, I agree!

Mon, Jul 15, 1:13 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey moved T227425: codfw: 1 misc node for the Kerberos KDC service from Backlog to Kerberos on the User-Elukey board.
Mon, Jul 15, 10:10 AM · hardware-requests, Operations, User-Elukey, Analytics
elukey moved T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN from Next Up to In Progress on the Analytics-Kanban board.
Mon, Jul 15, 9:46 AM · Analytics-Kanban, Analytics
elukey claimed T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN.
Mon, Jul 15, 9:46 AM · Analytics-Kanban, Analytics
elukey added a comment to T223863: memkeys segfaults on Debian Stretch.

Opened https://github.com/bmatheny/memkeys/issues/25

Mon, Jul 15, 8:51 AM · User-Elukey, Operations
elukey added a comment to T228015: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin.

From my home ipv6 address (removed the first hops):

Mon, Jul 15, 8:46 AM · Operations, netops
elukey updated the task description for T228015: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin.
Mon, Jul 15, 8:13 AM · Operations, netops
elukey triaged T228015: IPv6 packet loss registered by the Ripe Atlas anchor in eqsin as High priority.
Mon, Jul 15, 8:10 AM · Operations, netops

Sun, Jul 14

elukey triaged T227967: mr1-eqsin.oob IPv6 connectivity flapping as High priority.
Sun, Jul 14, 8:05 AM · Operations, netops
elukey created T227967: mr1-eqsin.oob IPv6 connectivity flapping.
Sun, Jul 14, 8:05 AM · Operations, netops

Sat, Jul 13

elukey triaged T227941: mylvmbackup on an-coord1001 not working as High priority.
Sat, Jul 13, 9:01 AM · Analytics-Kanban, Analytics

Fri, Jul 12

elukey added a subtask for T213089: Upgrade memcached for Debian Stretch/Buster: T223863: memkeys segfaults on Debian Stretch.
Fri, Jul 12, 2:48 PM · User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
elukey added a parent task for T223863: memkeys segfaults on Debian Stretch: T213089: Upgrade memcached for Debian Stretch/Buster.
Fri, Jul 12, 2:48 PM · User-Elukey, Operations
elukey updated the task description for T214275: Deprecate the usage of nutcracker for memcached.
Fri, Jul 12, 2:46 PM · Wikimedia-General-or-Unknown, serviceops, Performance-Team (Radar), User-Elukey, Operations
elukey added a comment to T214275: Deprecate the usage of nutcracker for memcached.

The latter should be doable, but the former seems a bit more complicated. Is there any plan to deprecate the labswiki infra and fold it into the appserver layer?

There definitely is such a plan, although it will be quite a while (maybe end of this FY?) before we're able to move forward.

Fri, Jul 12, 2:45 PM · Wikimedia-General-or-Unknown, serviceops, Performance-Team (Radar), User-Elukey, Operations
elukey added a comment to T225059: Consider adding per-shard metrics to the prometheus mcrouter exporter.

@aaron I added two new rows to https://grafana.wikimedia.org/dashboard/db/mcrouter with new per-shard metrics. Let me know what you think about it and if anything is missing.

Fri, Jul 12, 2:40 PM · Patch-For-Review, Performance-Team (Radar), observability, User-Elukey, Operations
elukey added a comment to T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models.

Recap of what it has been done so fare in various (sub) tasks:

Fri, Jul 12, 2:35 PM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.
Fri, Jul 12, 10:19 AM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a project to T227860: TLS certificates for Analytics origin servers: User-Elukey.
Fri, Jul 12, 9:52 AM · User-Elukey, Operations, Analytics, Traffic
elukey added a subtask for T226698: Allow all Analytics tools to work with Kerberos auth: T136858: Beeline does not print full stack traces when a query fails.
Fri, Jul 12, 9:42 AM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey added a parent task for T136858: Beeline does not print full stack traces when a query fails: T226698: Allow all Analytics tools to work with Kerberos auth.
Fri, Jul 12, 9:42 AM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.

Ok turns out this is only a problem that I have, Fran seems to get no errors.. Now I am really confused :D

Fri, Jul 12, 8:57 AM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.

No users were logged so I have also done apt-get purge hive; puppet agent -tv, to get a clean version of hive. Issue persists.

Fri, Jul 12, 8:14 AM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.

The hdfs-audit log error comes from this bit in /usr/lib/hive/bin/hive:

Fri, Jul 12, 8:09 AM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T222050: db1107 (eventlogging db master) possibly memory issues.

We can do it anytime with 10/15 mins of heads up Chris (I need to stop replication and traffic to db1107 before you can operate). Ping me on IRC! :)

Fri, Jul 12, 7:03 AM · Analytics, Operations, ops-eqiad, Analytics-EventLogging, DBA
elukey added a comment to T226467: Broken disk on analytics1072.

For some reason the disk doesn't show as failed by megacli but:

Fri, Jul 12, 5:57 AM · ops-eqiad, Operations, Analytics

Thu, Jul 11

elukey added projects to T227774: Add Jan Dittrich to the ldap/wmde and ldap/nda groups: LDAP-Access-Requests, Operations.
Thu, Jul 11, 2:19 PM · Operations, LDAP-Access-Requests, Analytics
elukey added a comment to T227093: Jan Dittrich would like to have access to superset.

@elukey I see – will you create the nda group task, or shall I do create it?

Thu, Jul 11, 1:04 PM · Analytics
elukey added a comment to T227093: Jan Dittrich would like to have access to superset.

Ah no sorry there is another issue, namely that you are not in the nda LDAP group. What it is currently failing is not superset, but the httpd's LDAP auth in front of it (that requires the user to be either in wmf or nda). We'll need to create a task like https://phabricator.wikimedia.org/T188105

Thu, Jul 11, 12:55 PM · Analytics
elukey added a comment to T227093: Jan Dittrich would like to have access to superset.

@Jan_Dittrich hi! The superset LDAP config requires the uid, that usually is not different but in your case is wmde-jand. I amended it now your account in superset, can you retry?

Thu, Jul 11, 12:38 PM · Analytics
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.

Error on stat1004 was due to an experiment that I was doing to nail down why the --verbose option leads to:

Thu, Jul 11, 12:31 PM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.

All right so ROCm 2.5 and tensorflow-rocm 1.13.3 seems to work. Other versions of TF (1.13.4 and 1.14.0) lead to the following error:

Thu, Jul 11, 11:58 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey moved T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created from In Progress to Done on the Analytics-Kanban board.
Thu, Jul 11, 10:44 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey set the point value for T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created to 8.
Thu, Jul 11, 10:43 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created.

https://grafana.wikimedia.org/d/ZAX3zaIWz/amd-rocm-gpu

Thu, Jul 11, 10:43 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports..

This is a pyspark2 session opened on stat1007:

Thu, Jul 11, 6:22 AM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey added a comment to T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports..

Very nice investigation, I was in fact trying to figure out the purpose of the last port and you solved it :) I'll make sure that port will get a range too!

Thu, Jul 11, 5:47 AM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics

Wed, Jul 10

elukey added a comment to T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports..

@EBernhardson should work now, let me know!

Wed, Jul 10, 5:07 PM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey added a comment to T170826: Enable base::firewall on stat boxes after restricting Spark REPL ports..

Not sure if it's related, but starting this week when i start a pyspark repl on stat1007 connected to yarn it shows up in yarn.wikimedia.org, but the ApplicationMaster link just times out. Seems plausibly related to firewalls

Wed, Jul 10, 4:12 PM · Analytics-Kanban, User-Elukey, Analytics-Cluster, Analytics
elukey added a project to T136858: Beeline does not print full stack traces when a query fails: Analytics.
Wed, Jul 10, 2:36 PM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T136858: Beeline does not print full stack traces when a query fails.

We currently are running Hive 1.1.0 (that should be 0.15 IIUC) and I can see this:

Wed, Jul 10, 2:36 PM · Analytics, Analytics-Kanban, Analytics-Cluster
elukey added a comment to T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN.

I am a little bit lost with LDAP config, since we use:

Wed, Jul 10, 10:19 AM · Analytics-Kanban, Analytics
elukey added a comment to T227164: Archive zookeeper puppet submodule.

There are some pull requests to close in https://github.com/wikimedia/puppet-zookeeper/pulls and also to set the mirror as read only, but I think that I don't have the necessary perms to do that.. @Krinkle can you help?

Wed, Jul 10, 9:29 AM · Patch-For-Review, Analytics-Kanban, Operations, Cleanup, Analytics
elukey updated the task description for T227164: Archive zookeeper puppet submodule.
Wed, Jul 10, 9:28 AM · Patch-For-Review, Analytics-Kanban, Operations, Cleanup, Analytics
elukey updated subscribers of T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN.

About 1.

Wed, Jul 10, 7:35 AM · Analytics-Kanban, Analytics
elukey moved T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created from Next Up to In Progress on the Analytics-Kanban board.
Wed, Jul 10, 7:21 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a project to T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created: Analytics-Kanban.
Wed, Jul 10, 7:21 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created.

I filed a code review to create the initial version of the node exporter, with the following metrics:

  • usage percent
  • power consumption (in watts)
  • fan usage percent
  • temperature (in celsius)
Wed, Jul 10, 7:20 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey changed the status of T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created, a subtask of T148843: Remove computational bottlenecks in stats machine via adding a GPU that can be used to train ML models, from Stalled to Open.
Wed, Jul 10, 7:18 AM · Patch-For-Review, User-Elukey, Operations, Analytics, Research-management
elukey changed the status of T220784: Investigate if a Prometheus exporter for the AMD GPU(s) can be easily created from Stalled to Open.
Wed, Jul 10, 7:18 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey moved T224723: Import AMD rocm packages in wikimedia-buster from Next Up to Done on the Analytics-Kanban board.
Wed, Jul 10, 6:28 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a project to T224723: Import AMD rocm packages in wikimedia-buster: Analytics-Kanban.
Wed, Jul 10, 6:27 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey set the point value for T224723: Import AMD rocm packages in wikimedia-buster to 13.
Wed, Jul 10, 6:27 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T227611: LDAP ldap-ro.eqiad.wikimedia.org not reachable from Analytics VLAN.

From puppet I can see that the change for ldap-ro was reverted:

Wed, Jul 10, 6:20 AM · Analytics-Kanban, Analytics

Tue, Jul 9

elukey updated subscribers of T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888.

Not really, I wish myself from the past added more info. I asked to @ayounsi and he didn't come up with a reason not to, so in theory we could try to modify the term on the firewall and see how it goes.

Tue, Jul 9, 3:31 PM · Traffic, Wikidata-Query-Service, Operations, WMDE-Analytics-Engineering, User-Addshore, Wikidata, Discovery
elukey added a comment to T224454: Create an alert for high memcached bw usage.

Makes sense, I am now wondering if we should create a generic and configurable alarm or not :)

Tue, Jul 9, 2:47 PM · observability, Performance-Team (Radar), User-Elukey, serviceops, Operations
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.

Added documentation in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU

Tue, Jul 9, 2:18 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.

Now the annoying part:

Tue, Jul 9, 9:53 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
jijiki awarded T225059: Consider adding per-shard metrics to the prometheus mcrouter exporter a Cookie token.
Tue, Jul 9, 8:48 AM · Patch-For-Review, Performance-Team (Radar), observability, User-Elukey, Operations
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.

New list:

Tue, Jul 9, 6:53 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.

Better now!

Tue, Jul 9, 6:12 AM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics

Mon, Jul 8

elukey changed the status of T227485: Decommission analytics10[28-41], a subtask of T211836: Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services, from Open to Stalled.
Mon, Jul 8, 2:34 PM · Analytics
elukey changed the status of T227485: Decommission analytics10[28-41] from Open to Stalled.

Not actionable yet.

Mon, Jul 8, 2:34 PM · decommission, Operations
elukey added a subtask for T211836: Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services: T227485: Decommission analytics10[28-41].
Mon, Jul 8, 2:34 PM · Analytics
elukey added a parent task for T227485: Decommission analytics10[28-41]: T211836: Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services.
Mon, Jul 8, 2:34 PM · decommission, Operations
elukey triaged T227485: Decommission analytics10[28-41] as Normal priority.
Mon, Jul 8, 2:33 PM · decommission, Operations
elukey added a comment to T224723: Import AMD rocm packages in wikimedia-buster.
root@install1002:~# reprepro --noskipold --component thirdparty/amd-rocm checkupdate buster-wikimedia
Calculating packages to get...
Mon, Jul 8, 2:01 PM · Analytics-Kanban, Patch-For-Review, User-Elukey, Operations, Analytics
elukey committed rLPRI0c5fc86a0408: Add analytics keytab for an-tool1006 (authored by elukey).
Add analytics keytab for an-tool1006
Mon, Jul 8, 1:53 PM
elukey added a subtask for T226089: Make the Kerberos infrastructure production ready: T227425: codfw: 1 misc node for the Kerberos KDC service.
Mon, Jul 8, 12:59 PM · User-Elukey, Analytics
elukey added a parent task for T227425: codfw: 1 misc node for the Kerberos KDC service: T226089: Make the Kerberos infrastructure production ready.
Mon, Jul 8, 12:59 PM · hardware-requests, Operations, User-Elukey, Analytics
elukey added a comment to T203963: Convert makevm to spicerack cookbook.

Ah, and one more thing: After typing "done" for confirmation, output stalls for about ten minutes while the Ganeti instance is created (but eventually all progress is printed), let's maybe add a short message like "Creating your instance, this will take some time", as initially I was worried it was failing.

Mon, Jul 8, 10:11 AM · serviceops-radar, Patch-For-Review, User-crusnov, Operations-Software-Development, User-jijiki, User-Joe, Operations
elukey added a comment to T203963: Convert makevm to spicerack cookbook.
elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_A test_not_existing.eqiad.wmnet --vcpus 2 --memory 4 --disk 150 --link analytics
START - Cookbook sre.ganeti.makevm
Exception raised while executing cookbook sre.ganeti.makevm:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/dns.py", line 137, in resolve
    response = self._resolver.query(qname, record_type)
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1051, in query
    raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)
dns.resolver.NXDOMAIN: None of DNS query names exist: test_not_existing.eqiad.wmnet., test_not_existing.eqiad.wmnet.eqiad.wmnet.
Mon, Jul 8, 9:55 AM · serviceops-radar, Patch-For-Review, User-crusnov, Operations-Software-Development, User-jijiki, User-Joe, Operations
elukey added a comment to T227288: eqiad: 1 misc node for the Kerberos KDC service.

Amended this task and created T227425 :)

Mon, Jul 8, 7:52 AM · hardware-requests, Operations, User-Elukey, Analytics
elukey created T227425: codfw: 1 misc node for the Kerberos KDC service.
Mon, Jul 8, 7:51 AM · hardware-requests, Operations, User-Elukey, Analytics
elukey renamed T227288: eqiad: 1 misc node for the Kerberos KDC service from eqiad: 2 misc nodes for the Kerberos KDC service to eqiad: 1 misc node for the Kerberos KDC service.
Mon, Jul 8, 7:49 AM · hardware-requests, Operations, User-Elukey, Analytics
elukey updated subscribers of T226983: Page summary endpoint in RESTBase not updated since about June 27.

Explicitly adding @Eevans @Pchelolo @WDoranWMF (@mobrovac is now OOTO).

Mon, Jul 8, 7:40 AM · Core Platform Team (Security, stability, performance and scalability (TEC1)), Core Platform Team Workboards (Team 2), Operations, RESTBase-API, Page-Previews
elukey moved T227265: mcrouter codfw proxies sometimes lead to TKOs from Backlog to Mcrouter on the User-Elukey board.
Mon, Jul 8, 7:37 AM · Performance-Team (Radar), User-Elukey, serviceops, Operations
elukey added a project to T227265: mcrouter codfw proxies sometimes lead to TKOs: User-Elukey.
Mon, Jul 8, 7:37 AM · Performance-Team (Radar), User-Elukey, serviceops, Operations