elukey (Luca Toscano)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Jan 5 2016, 9:54 PM (158 w, 2 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
LToscano (WMF) [ Global Accounts ]

Recent Activity

Yesterday

elukey merged T214008: Netbox Dies Mysteriously Sometimes into T212697: uwsgi's logsocket_plugin.so causes segfaults during log rotation.
Thu, Jan 17, 4:46 PM · Operations
elukey merged task T214008: Netbox Dies Mysteriously Sometimes into T212697: uwsgi's logsocket_plugin.so causes segfaults during log rotation.
Thu, Jan 17, 4:46 PM · netops, Operations
elukey updated the task description for T214057: Broken disk on analytics1056.
Thu, Jan 17, 4:22 PM · ops-eqiad, Operations, Patch-For-Review, Analytics
elukey triaged T214057: Broken disk on analytics1056 as Normal priority.
Thu, Jan 17, 4:22 PM · ops-eqiad, Operations, Patch-For-Review, Analytics
elukey added a comment to T214008: Netbox Dies Mysteriously Sometimes.

@crusnov hi! I think this is the same issue as T212697

Thu, Jan 17, 9:48 AM · netops, Operations
elukey updated the task description for T208844: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw.
Thu, Jan 17, 7:23 AM · User-jijiki, Patch-For-Review, Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

The alter tables to convert Aria tables to InnoDB are still in progress :)

Thu, Jan 17, 7:15 AM · Analytics-Kanban, User-Elukey, Analytics

Wed, Jan 16

elukey added a comment to T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..

Today I have made all the partitions on the old hosts (wiping the old content) so we are now ready to bootstrap the cluster merging https://gerrit.wikimedia.org/r/484374

Wed, Jan 16, 4:13 PM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey reassigned T211605: Upgrade Superset to 0.28.1 from elukey to fdans.
Wed, Jan 16, 4:11 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T212487: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup.

@Marostegui I have now everything in my home gzipped, I'll move it to stat1007 and HDFS asap:

Wed, Jan 16, 2:08 PM · User-Elukey, Analytics-Kanban, DBA, Analytics
elukey added a comment to T213859: eqiad: rack a3 pdu swap / failure / replacement.

Ok for me for the analytics nodes, but I'd need a bit of heads up to properly stop them if possible :)

Wed, Jan 16, 11:22 AM · Patch-For-Review, ops-eqiad, Operations
elukey added a comment to T212493: Clean up staging db.

@Marostegui I don't see any more "easy" way to ping people about those tables, do you think that this clean up was ok (and the task can be closed) or that we'd need something more?

Wed, Jan 16, 11:18 AM · Analytics-Kanban, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

Leaving also the following aside for a moment:

Wed, Jan 16, 9:33 AM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

Seems happening for .ores_classification tables:

Wed, Jan 16, 9:30 AM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

Skipping this table for the moment:

Wed, Jan 16, 9:22 AM · Analytics-Kanban, User-Elukey, Analytics
elukey updated the task description for T212493: Clean up staging db.
Wed, Jan 16, 8:27 AM · Analytics-Kanban, Analytics
elukey added a comment to T212493: Clean up staging db.

Thanks all for your clean up! So remaining tables to drop:

Wed, Jan 16, 7:55 AM · Analytics-Kanban, Analytics

Tue, Jan 15

elukey added a comment to T212493: Clean up staging db.

Thanks all!

Tue, Jan 15, 5:54 PM · Analytics-Kanban, Analytics
elukey updated subscribers of T212493: Clean up staging db.
Tue, Jan 15, 5:54 PM · Analytics-Kanban, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

Took a mysqldump of the staging database and moved it in two places:

  • on dbstore1002's /srv/elukey_backup
  • on stat1007's /home/elukey home dir (chmod root:root, 700)
Tue, Jan 15, 4:33 PM · Analytics-Kanban, User-Elukey, Analytics
elukey moved T212949: Allow the deployment of users without SSH access from In Progress to In Code Review on the Analytics-Kanban board.
Tue, Jan 15, 2:28 PM · Patch-For-Review, Operations, Analytics-Kanban, Analytics
elukey moved T213706: Convert Aria tables to InnoDB on dbstore1002 from Next Up to In Progress on the Analytics-Kanban board.
Tue, Jan 15, 2:28 PM · Analytics-Kanban, User-Elukey, Analytics
elukey added a project to T213706: Convert Aria tables to InnoDB on dbstore1002: Analytics-Kanban.
Tue, Jan 15, 2:28 PM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T213309: Reportupdater should alert if it fails over and over.

We decided to try a simple systemd timer for the moment, that will alarm if report updater will run and return a non zero code. This is currently tracked in T172532

Tue, Jan 15, 2:17 PM · Analytics
elukey added a comment to T213716: Alarms for virtualpageview should exist (probably in oozie) for jobs that have been idle too long.

As FYI I can see the following for Dec 17th in my inbox for analytics-alerts@:

Tue, Jan 15, 1:51 PM · Analytics
elukey raised the priority of T213706: Convert Aria tables to InnoDB on dbstore1002 from Normal to High.
Tue, Jan 15, 1:48 PM · Analytics-Kanban, User-Elukey, Analytics
elukey awarded T203786: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions a Love token.
Tue, Jan 15, 12:20 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Language-Team (Language-2019-January-March), MediaWiki Language Extension Bundle, Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error, User-Elukey, MediaWiki-Cache, Operations
elukey added a project to T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one.: User-Elukey.
Tue, Jan 15, 10:39 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

Another question - should we back up the staging database just in case something goes wrong?

Tue, Jan 15, 10:36 AM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

I have created /home/elukey/aria_tables_alter.sql on dbstore1002, if you can review them quickly as sanity check it would be great. Then I'd just execute mysql --skip-ssl < /home/elukey/aria_tables_alter.sql from a root tmux and monitor the status during the next hours.

Tue, Jan 15, 10:34 AM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T213706: Convert Aria tables to InnoDB on dbstore1002.

So as far as I can understand I'd need to grab the list of tables and produce a list of:

Tue, Jan 15, 10:24 AM · Analytics-Kanban, User-Elukey, Analytics
elukey claimed T213706: Convert Aria tables to InnoDB on dbstore1002.
Tue, Jan 15, 10:22 AM · Analytics-Kanban, User-Elukey, Analytics
elukey moved T213706: Convert Aria tables to InnoDB on dbstore1002 from Backlog to In Progress on the User-Elukey board.
Tue, Jan 15, 10:21 AM · Analytics-Kanban, User-Elukey, Analytics
elukey updated subscribers of T212493: Clean up staging db.
Tue, Jan 15, 9:46 AM · Analytics-Kanban, Analytics
elukey updated subscribers of T212493: Clean up staging db.

@Neil_P._Quinn_WMF @Milimetric @mforns @leila @nettrom_WMF @DarTar @Tbayer would you mind to review the tables in the description and see if anything is definitely not needed and can be dropped?

Tue, Jan 15, 9:46 AM · Analytics-Kanban, Analytics
elukey added a comment to T212493: Clean up staging db.

So the staging db is kinda problematic to clean up, since it is difficult to figure out owners and reach out to people. I have already started to ask to people to review/drop the old tables, but as precautionary step I'd migrate everything to one of the new dbstore hosts and then figure out later on if something else can be dropped.

Tue, Jan 15, 9:43 AM · Analytics-Kanban, Analytics
elukey updated the task description for T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..
Tue, Jan 15, 9:24 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey updated the task description for T213790: mgmt host interfaces down for rack D7 in codfw due to ge-0/0/30 on msw1-codfw down.
Tue, Jan 15, 8:51 AM · netops, Operations
elukey updated the task description for T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..
Tue, Jan 15, 8:47 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey updated the task description for T212256: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one..
Tue, Jan 15, 8:42 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey updated subscribers of T213790: mgmt host interfaces down for rack D7 in codfw due to ge-0/0/30 on msw1-codfw down.
Tue, Jan 15, 7:16 AM · netops, Operations
elukey triaged T213790: mgmt host interfaces down for rack D7 in codfw due to ge-0/0/30 on msw1-codfw down as High priority.
Tue, Jan 15, 7:13 AM · netops, Operations

Mon, Jan 14

elukey updated the task description for T213748: swap a2-eqiad PDU with on-site spare.
Mon, Jan 14, 7:03 PM · Patch-For-Review, DBA, Analytics, ops-eqiad, Operations
elukey updated the task description for T213748: swap a2-eqiad PDU with on-site spare.
Mon, Jan 14, 7:00 PM · Patch-For-Review, DBA, Analytics, ops-eqiad, Operations
elukey set the point value for T205846: Move users from stat1005 to stat1007 to 21.
Mon, Jan 14, 4:01 PM · Analytics-Kanban, Patch-For-Review, Analytics
elukey moved T205846: Move users from stat1005 to stat1007 from In Progress to Done on the Analytics-Kanban board.
Mon, Jan 14, 4:01 PM · Analytics-Kanban, Patch-For-Review, Analytics
elukey assigned T206965: Degraded RAID on dbstore1002 to Cmjohnson.
Mon, Jan 14, 3:52 PM · Product-Analytics, Analytics, ops-eqiad, Operations
elukey added a comment to T213670: dbstore1002 Mysql errors.

Update for all the users of dbstore1002:

Mon, Jan 14, 2:49 PM · Product-Analytics, Analytics-Kanban, Analytics
elukey added a project to T213706: Convert Aria tables to InnoDB on dbstore1002: User-Elukey.
Mon, Jan 14, 2:47 PM · Analytics-Kanban, User-Elukey, Analytics
elukey added a comment to T203786: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions.

@Nikerabbit looking forward to see it deployed, thanks to both you and Aaron for this work.

Mon, Jan 14, 2:35 PM · MW-1.33-notes (1.33.0-wmf.13; 2019-01-15), Language-Team (Language-2019-January-March), MediaWiki Language Extension Bundle, Patch-For-Review, Performance-Team (Radar), Wikimedia-production-error, User-Elukey, MediaWiki-Cache, Operations
elukey moved T213702: Check home leftovers of user imarlier (Ian Marlier) from Incoming to Ops Week on the Analytics board.
Mon, Jan 14, 2:11 PM · Analytics
elukey moved T212127: Clean up home dirs for users jamesur and nithum from Operational Excellence to Ops Week on the Analytics board.
Mon, Jan 14, 2:10 PM · Analytics
elukey triaged T213702: Check home leftovers of user imarlier (Ian Marlier) as Normal priority.
Mon, Jan 14, 2:03 PM · Analytics
elukey added a comment to T213670: dbstore1002 Mysql errors.

Update for all the users of dbstore1002:

Mon, Jan 14, 10:55 AM · Product-Analytics, Analytics-Kanban, Analytics
elukey renamed T213670: dbstore1002 Mysql errors from dbstore1002 crashed to dbstore1002 Mysql errors.
Mon, Jan 14, 8:13 AM · Product-Analytics, Analytics-Kanban, Analytics
elukey updated the task description for T213670: dbstore1002 Mysql errors.
Mon, Jan 14, 8:12 AM · Product-Analytics, Analytics-Kanban, Analytics
elukey added a comment to T213670: dbstore1002 Mysql errors.

racadm getsel for today (remember that one disk was already failed, we have a task about it):

Mon, Jan 14, 7:53 AM · Product-Analytics, Analytics-Kanban, Analytics
elukey added a comment to T213670: dbstore1002 Mysql errors.

Executed bmc-device --debug --cold-reset in localhost since the mgmt interface was not available ("No more sessions available").

Mon, Jan 14, 7:51 AM · Product-Analytics, Analytics-Kanban, Analytics

Fri, Jan 11

elukey moved T212487: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup from Backlog to In Progress on the User-Elukey board.
Fri, Jan 11, 3:01 PM · User-Elukey, Analytics-Kanban, DBA, Analytics
elukey moved T212386: Provide tools for querying MediaWiki replica databases without having to specify the shard from Backlog to In Progress on the User-Elukey board.
Fri, Jan 11, 3:01 PM · Analytics, WMDE-Analytics-Engineering, User-Addshore, User-Elukey, Research
elukey moved T208870: TCP connections between analytics and deployment-prep from Backlog to Stalled on the User-Elukey board.
Fri, Jan 11, 3:00 PM · Analytics, User-Elukey, Beta-Cluster-Infrastructure
elukey added a comment to T212386: Provide tools for querying MediaWiki replica databases without having to specify the shard.

@jcrespo it seems we should be able to deploy (out of the box with a new config) the tool existing in prod to the new and upcoming analytics replicas right? Am I missing something why this would not be possible?

I agree with you and I don't see why you couldn't do that. :-) My only petition at T212386#4862319 was to make maintenance easy for us, be it with a puppetized configuration or any other method.

Fri, Jan 11, 2:43 PM · Analytics, WMDE-Analytics-Engineering, User-Addshore, User-Elukey, Research
elukey added a project to T212172: Provide feature parity between the wiki replicas and the Analytics Data Lake: User-Elukey.
Fri, Jan 11, 2:35 PM · User-Elukey, Epic, Analytics, Contributors-Analysis, Product-Analytics
elukey updated subscribers of T213089: Upgrade memcached for Debian Stretch/Buster.
Fri, Jan 11, 9:50 AM · Performance-Team (Radar), Operations, User-Elukey
elukey moved T208844: Apply -R 200 to all the memcached mw object cache instances running in eqiad/codfw from In Progress to Stalled on the User-Elukey board.
Fri, Jan 11, 7:40 AM · User-jijiki, Patch-For-Review, Performance-Team (Radar), User-Elukey, MediaWiki-Cache, Operations

Thu, Jan 10

elukey updated the task description for T172532: Refactor analytics cronjobs to alarm on failure reliably.
Thu, Jan 10, 5:26 PM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey moved T172532: Refactor analytics cronjobs to alarm on failure reliably from Paused to In Progress on the Analytics-Kanban board.
Thu, Jan 10, 5:01 PM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey updated the task description for T172532: Refactor analytics cronjobs to alarm on failure reliably.
Thu, Jan 10, 11:13 AM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey updated the task description for T172532: Refactor analytics cronjobs to alarm on failure reliably.
Thu, Jan 10, 11:10 AM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey closed T210939: Varnishkafka error to investigate: Required feature not supported by broker as Resolved.

The issue seems a one time only, and the new alarms have been very reliable over the past months. Closing this task since there seems to be no action left, will re-open if necessary.

Thu, Jan 10, 9:07 AM · Patch-For-Review, User-Elukey, Analytics
elukey removed a project from T212011: migrate netinsights from rhenium to sulfur: User-Elukey.
Thu, Jan 10, 9:06 AM · netops, Operations
elukey moved T213038: Degraded RAID on analytics1054 from Backlog to Stalled on the User-Elukey board.
Thu, Jan 10, 9:05 AM · Patch-For-Review, Analytics, User-Elukey, ops-eqiad, Operations
elukey moved T204479: Heating alerts and broken RAM on kafka1014 from Backlog to Stalled on the User-Elukey board.
Thu, Jan 10, 9:05 AM · User-Elukey, Operations, ops-eqiad
elukey moved T209929: Decommission old Hadoop worker nodes and add newer ones from In Progress to Done on the User-Elukey board.
Thu, Jan 10, 9:05 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey moved T172532: Refactor analytics cronjobs to alarm on failure reliably from Stalled to In Progress on the User-Elukey board.
Thu, Jan 10, 8:43 AM · Patch-For-Review, Analytics-Kanban, User-Elukey, Analytics
elukey set the point value for T206943: JVM pauses cause Yarn master to failover to 5.
Thu, Jan 10, 8:43 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey moved T206943: JVM pauses cause Yarn master to failover from Stalled to Done on the User-Elukey board.
Thu, Jan 10, 8:43 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T204479: Heating alerts and broken RAM on kafka1014.

It is fine in here Daniel, thanks! In theory kafka1012->23 should be decommissioned when Event Gate (part of Modern Event Platform) will be up and running, since Mediawiki Avro Monolog will be migrated to it and at that point nothing will be pushing data to the old Kafka Analytics cluster anymore. So I wouldn't spend much time energy on this if possible, worst case scenario we can shrink down the cluster to 5/4 hosts and decom the ones (like this) that are not healthy anymore.

Thu, Jan 10, 7:01 AM · User-Elukey, Operations, ops-eqiad
elukey added a project to T204479: Heating alerts and broken RAM on kafka1014: User-Elukey.
Thu, Jan 10, 6:59 AM · User-Elukey, Operations, ops-eqiad

Wed, Jan 9

elukey added a comment to T213089: Upgrade memcached for Debian Stretch/Buster.

I'd prefer 1.5.6 too, we'd be really close to upstream (atm 1.5.12) and getting help from them would surely be easier if needed. I'd also love to be able to provide feedback to the memcached project about scalability and/or bottlenecks of running a recent version at scale (for example, LRU special use case, etc..).

Wed, Jan 9, 3:55 PM · Performance-Team (Radar), Operations, User-Elukey
elukey moved T209929: Decommission old Hadoop worker nodes and add newer ones from In Progress to Done on the Analytics-Kanban board.
Wed, Jan 9, 3:32 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey set the point value for T209929: Decommission old Hadoop worker nodes and add newer ones to 13.
Wed, Jan 9, 3:32 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T209929: Decommission old Hadoop worker nodes and add newer ones.

As mentioned before these nodes will become a new testing cluster, more info in T212256

Wed, Jan 9, 3:32 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T209929: Decommission old Hadoop worker nodes and add newer ones.

Nodes completely removed:

  • removed from the network topology and restarted namenodes
  • assigned role::spare:system and removed the datanode/nodemanager packages from the hosts to prevent the chance that a node comes back alive.
  • cleaned up the hosts.exclude list
Wed, Jan 9, 3:31 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T211250: Create a mediawiki::cronjob define.
  1. is done :)
Wed, Jan 9, 8:25 AM · Patch-For-Review, serviceops, User-jijiki, Operations

Tue, Jan 8

elukey added a comment to T206943: JVM pauses cause Yarn master to failover.

We bumped the Xmx/Xms settings of the HDFS namenode to 12G (was 8G) for unrelated changes and I haven't seen any more pauses since then. The increased zookeeper timeout has also helped. For the moment I don't see any more action items, so I think we are good to close.

Tue, Jan 8, 4:11 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey moved T206943: JVM pauses cause Yarn master to failover from Paused to Done on the Analytics-Kanban board.
Tue, Jan 8, 4:10 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T211605: Upgrade Superset to 0.28.1.

So I tried it today and I have a couple of notes:

Tue, Jan 8, 10:33 AM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T209929: Decommission old Hadoop worker nodes and add newer ones.
Tue, Jan 8, 7:31 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey updated the task description for T209929: Decommission old Hadoop worker nodes and add newer ones.
Tue, Jan 8, 7:31 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics

Mon, Jan 7

elukey added a project to T212958: Create staging domain for turnilo to test config changes: User-Elukey.
Mon, Jan 7, 6:59 PM · User-Elukey, Analytics-Kanban, Analytics
elukey lowered the priority of T212958: Create staging domain for turnilo to test config changes from High to Normal.

When we discussed this use case I was not aware (shame! shame!) about using SSH -L in the following way:

Mon, Jan 7, 6:58 PM · User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T213038: Degraded RAID on analytics1054.
Enclosure Device ID: 32
Slot Number: 1
Drive's position: DiskGroup: 2, Span: 0, Arm: 0
Enclosure position: 1
Device Id: 1
WWN: 500003964b700233
Sequence Number: 3
Media Error Count: 0
Other Error Count: 3
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Mon, Jan 7, 6:52 PM · Patch-For-Review, Analytics, User-Elukey, ops-eqiad, Operations
elukey added a comment to T213038: Degraded RAID on analytics1054.

@Cmjohnson so I got a different than usual output from:

Mon, Jan 7, 5:53 PM · Patch-For-Review, Analytics, User-Elukey, ops-eqiad, Operations
elukey assigned T213038: Degraded RAID on analytics1054 to Cmjohnson.
Mon, Jan 7, 5:14 PM · Patch-For-Review, Analytics, User-Elukey, ops-eqiad, Operations
elukey added a comment to T213038: Degraded RAID on analytics1054.

@elukey the disk still shows failed do you have to manually add it back?

Mon, Jan 7, 5:03 PM · Patch-For-Review, Analytics, User-Elukey, ops-eqiad, Operations
elukey changed the status of T205507: Decommission analytics100[1,2] from Stalled to Open.

Nevermind then, I can easily use only analytics1028->41, we are good to decom. Thanks!

Mon, Jan 7, 3:53 PM · Patch-For-Review, Operations, ops-eqiad, decommission, User-Elukey, Analytics
elukey changed the status of T205507: Decommission analytics100[1,2], a subtask of T203635: Replace the Analytics HDFS/Yarn masters (hardware refresh), from Stalled to Open.
Mon, Jan 7, 3:53 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey triaged T213089: Upgrade memcached for Debian Stretch/Buster as Normal priority.
Mon, Jan 7, 3:50 PM · Performance-Team (Radar), Operations, User-Elukey
elukey closed T129963: Update memcached package and configuration options as Resolved.

Going to close this task to open another one that tracks the upgrade to buster or stretch, this one is full of information and I wouldn't like to overload it (more than now).

Mon, Jan 7, 3:39 PM · Performance-Team (Radar), User-Elukey, Operations