Page MenuHomePhabricator

elukey (Luca Toscano)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Friday

  • Clear sailing ahead.

User Details

User Since
Jan 5 2016, 9:54 PM (201 w, 4 h)
Availability
Available
LDAP User
Unknown
MediaWiki User
LToscano (WMF) [ Global Accounts ]

Recent Activity

Yesterday

elukey added a project to T238113: Repurpose db1107 as a generic database: Analytics.
Tue, Nov 12, 6:37 PM · Analytics, DBA
elukey moved T231208: Version analytics meta mysql database backup from Backlog to Done on the User-Elukey board.
Tue, Nov 12, 1:25 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T231858: Archive data on eventlogging MySQL to analytics replica before decomisioning .

Sanitization done!

Tue, Nov 12, 11:43 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey moved T236818: Rerun sanitization before archiving eventlogging mysql data from In Progress to Done on the Analytics-Kanban board.
Tue, Nov 12, 11:42 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey set the point value for T236818: Rerun sanitization before archiving eventlogging mysql data to 5.
Tue, Nov 12, 11:42 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T213089: Upgrade memcached for Debian Stretch/Buster.

Not all new slab metrics are rendered, opened an issue upstream: https://github.com/prometheus/memcached_exporter/issues/75

Tue, Nov 12, 10:31 AM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
elukey added a comment to T213089: Upgrade memcached for Debian Stretch/Buster.

Build and deployed 0.6.0 and deployed on deployment-prep-memc08. Also created the following:

Tue, Nov 12, 9:31 AM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey

Mon, Nov 11

elukey created T237993: Create replacement for Varnishkafka.
Mon, Nov 11, 3:26 PM · Traffic, Operations, Analytics
elukey committed rLPRI7d16fd446316: Add fake kerberos keytabs for stat nodes (authored by elukey).
Add fake kerberos keytabs for stat nodes
Mon, Nov 11, 9:57 AM

Fri, Nov 8

elukey added a comment to T213089: Upgrade memcached for Debian Stretch/Buster.

stat items is something that changed a bit. For example, take slab 63:

Fri, Nov 8, 5:40 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
elukey added a comment to T212258: Create test Kerberos identities/accounts for some selected users in hadoop test cluster.

I think it is the suggested way, and keep in mind that you'll have to do it only once every 24h, not every time!

Gotcha - thanks! I suspect we don't have any control over the default display of JupyterHub, but if there's any way to include that as a note -- e.g., something like If you are having trouble connecting to Hive, make sure that you have executed the kinit command in the terminal -- that would be helpful for when we forget or are onboarding new users.

Fri, Nov 8, 7:42 AM · User-Elukey, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Fri, Nov 8, 7:17 AM · Patch-For-Review, Analytics-Kanban, Analytics

Thu, Nov 7

elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Thu, Nov 7, 4:59 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Thu, Nov 7, 4:46 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T212258: Create test Kerberos identities/accounts for some selected users in hadoop test cluster.

Thanks a lot for the tests!

Thu, Nov 7, 2:49 PM · User-Elukey, Analytics-Kanban, Analytics
elukey added a project to T230835: Investigate janitor, maintenance emails parser: User-Elukey.
Thu, Nov 7, 2:45 PM · User-Elukey, Operations
elukey added a comment to T234229: Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts .

it might mean that we'll have to stop rsyncs for a couple of days

Do we need to stop the real rsync jobs? The only one that wasn't working is the mw history rsync, because it is so large.

Thu, Nov 7, 2:13 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T234229: Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts .

After a chat with Moritz and my team, this is what we are planning to do:

Thu, Nov 7, 9:37 AM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
elukey added a comment to T220698: Check if a GPU fits in any of the remaining stat or notebook hosts.

@Cmjohnson we might need to add a new GPU next quarter (need to triple check with the Research team), is there any of the above hosts that can host a GPU in your opinion or should we open each one of them and measure to find one?

Thu, Nov 7, 9:15 AM · User-Elukey, Analytics, Operations, ops-eqiad
elukey moved T237269: Prepare the Hadoop Analytics cluster for Kerberos from Next Up to In Progress on the Analytics-Kanban board.
Thu, Nov 7, 6:57 AM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Thu, Nov 7, 6:50 AM · Patch-For-Review, Analytics-Kanban, Analytics
elukey created T237605: Create kerberos principals for users.
Thu, Nov 7, 6:50 AM · Analytics-Kanban, Analytics

Wed, Nov 6

elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Wed, Nov 6, 4:43 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey moved T231208: Version analytics meta mysql database backup from In Progress to Done on the Analytics-Kanban board.
Wed, Nov 6, 3:05 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T231208: Version analytics meta mysql database backup .
Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name
===================================================================
[..]
159737  Incr          2    3.596 G  OK       06-Nov-19 04:05 an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-hadoop-namenode-backup
159834  Full        781    19.16 G  OK       06-Nov-19 11:28 an-master1002.eqiad.wmnet-Monthly-1st-Mon-production-analytics-meta-mysql-lvm-backup  <<===
Wed, Nov 6, 3:00 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Wed, Nov 6, 2:17 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey moved T231208: Version analytics meta mysql database backup from Paused to In Progress on the Analytics-Kanban board.
Wed, Nov 6, 12:00 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey moved T233891: Drop Navigationtiming data entirely from mysql storage? from In Progress to Done on the Analytics-Kanban board.
Wed, Nov 6, 12:00 PM · Performance-Team (Radar), Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T236818: Rerun sanitization before archiving eventlogging mysql data .

Sanitization is still running on the two databases!

Wed, Nov 6, 11:43 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Wed, Nov 6, 8:59 AM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T235200: Create HDFS /tmp/ cleaner.

On an-coord1001 I can see:

Wed, Nov 6, 7:07 AM · Analytics-Kanban, Analytics-Cluster, Analytics

Tue, Nov 5

elukey claimed T236818: Rerun sanitization before archiving eventlogging mysql data .
Tue, Nov 5, 4:56 PM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey moved T233891: Drop Navigationtiming data entirely from mysql storage? from Next Up to In Progress on the Analytics-Kanban board.
Tue, Nov 5, 4:56 PM · Performance-Team (Radar), Analytics-Kanban, Analytics, Analytics-EventLogging
elukey moved T236818: Rerun sanitization before archiving eventlogging mysql data from Next Up to In Progress on the Analytics-Kanban board.
Tue, Nov 5, 4:56 PM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey set the point value for T233891: Drop Navigationtiming data entirely from mysql storage? to 5.
Tue, Nov 5, 4:55 PM · Performance-Team (Radar), Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T233891: Drop Navigationtiming data entirely from mysql storage? .

To recap:

  • Drop table
NavigationTiming_17703215
NavigationTiming_17729222
NavigationTiming_17740037
NavigationTiming_17854363
NavigationTiming_17957960
NavigationTiming_17975587
NavigationTiming_18156125
NavigationTiming_18358840
NavigationTiming_18435897
NavigationTiming_18636169
NavigationTiming_18813782
NavigationTiming_18917904
NavigationTiming_18988839
NavigationTiming_19147339
NavigationTiming_19157653
Tue, Nov 5, 4:55 PM · Performance-Team (Radar), Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T220387: Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task).

One thing that we didn't discuss for this goal is Zookeeper. At the moment multiple things are using conf100[4-6] hosts:

  • Hadoop Yarn (leader election + rmstore)
  • Hadoop HDFS (leader election)
  • Kafka Main eqiad
  • Kafka Jumbo
  • Kafka logging
  • Kafka Burrow (consumer lag metrics)

Zookeeper also holds the Kafka ACLs for topic, only used by Kafka Jumbo afaik at the moment. The best thing to do in my opinion is to request new conf10XX-like hosts only for analytics and move away the Hadoop use case. We can make it optional to avoid blocking this task, but if everybody agrees I'll add a subtask and start working on it.

Tue, Nov 5, 4:12 PM · User-herron, Operations
elukey added a comment to T233661: Publish tls related info to webrequest via varnish.

Agreed, let's not go down that road right here (because we have a burning need for this data pronto), but side note to keep in mind: "one day" is really really soon (like, we'll probably be migrating varnishkafka stuff to ATS next quarter).

Probably this is not the best place to talk about this, but next quarter seems really close :) Who is going to replace Varnishkafka? Is there any plan from Traffic or should we (as Analytics) schedule time for it? I know that with fifo-log-demux the job of the new "atskafka" should be relatively easy (read from a socket and push to kafka) but we really need metrics (like we have now) to build monitoring on top of them. If the migration is so close, can we open a task (if there is not one) and start the discussion? :)

Yes, next quarter is closer every day :) We probably should at least start talking informally about it and deciding who can line up time and making a task (if you'd like to work on it and have time that'd be awesome, but otherwise we will because it will be our next blocker on that front).

Tue, Nov 5, 3:20 PM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Tue, Nov 5, 3:10 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Tue, Nov 5, 2:42 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Tue, Nov 5, 2:41 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T213089: Upgrade memcached for Debian Stretch/Buster.

Added metrics to http://beta-prometheus.wmflabs.org about memcached:

Tue, Nov 5, 2:25 PM · Patch-For-Review, User-jijiki, serviceops, Performance-Team (Radar), Operations, User-Elukey
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Tue, Nov 5, 2:04 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Tue, Nov 5, 10:27 AM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T222253: Upgrade Spark to 2.4.x.
Tue, Nov 5, 9:25 AM · Patch-For-Review, Analytics-Kanban, Analytics, Analytics-Cluster
elukey added a comment to T236818: Rerun sanitization before archiving eventlogging mysql data .

Remaining steps:

Tue, Nov 5, 7:52 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T233661: Publish tls related info to webrequest via varnish.

Agreed, let's not go down that road right here (because we have a burning need for this data pronto), but side note to keep in mind: "one day" is really really soon (like, we'll probably be migrating varnishkafka stuff to ATS next quarter).

Tue, Nov 5, 7:46 AM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic

Mon, Nov 4

elukey added a comment to T231208: Version analytics meta mysql database backup .

'the meta backup's files'? You mean the mysqldumps?
In general yes if we have another way of storing versioned backups (mysqldump+bacula) then we should remove the custom HDFS backup upload cron.

Mon, Nov 4, 4:06 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T231208: Version analytics meta mysql database backup .

@Ottomata I am wondering if we could simply configure bacula to copy the meta backup's files as we do for Archiva. This will allow us to remove the cron that uploads to HDFS.. Need to check with Jaime first if this is possible, but if so what do you think about it?

Mon, Nov 4, 3:36 PM · User-Elukey, Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Mon, Nov 4, 3:26 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey created T237271: Create a script to ease the Oozie work while enabling kerberos in Hadoop.
Mon, Nov 4, 3:26 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey updated the task description for T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Mon, Nov 4, 3:21 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey created T237269: Prepare the Hadoop Analytics cluster for Kerberos.
Mon, Nov 4, 3:19 PM · Patch-For-Review, Analytics-Kanban, Analytics
elukey added a comment to T236406: Switchover backup director service from helium to backup1001.

Nice thanks! Just pushed the new rules to the routers, so in theory an-master1002 and analytics1029 should go away now! Let me know :)

Mon, Nov 4, 1:41 PM · Patch-For-Review, Goal, DBA, serviceops, Operations
elukey added a comment to T237016: Update router ACLs for newer bacula hosts.

To keep archives happy:

Mon, Nov 4, 1:38 PM · Operations, netops
elukey added a comment to T236406: Switchover backup director service from helium to backup1001.

I am all for simplifying and standardizing confs, so no opposition about incremental. Only one question - what would it change when trying to restore the database? This is basically my only concern at the moment.. If it is as simple as doing a recovery via Bacula I am all for it!

Mon, Nov 4, 11:55 AM · Patch-For-Review, Goal, DBA, serviceops, Operations
elukey added a comment to T236406: Switchover backup director service from helium to backup1001.

@elukey @Ottomata Re: matomo1001, is there a reason not to have daily incrementals? If the reason is that it generates a full backup each time, maybe I can suggest using the Monthly policy (like we use for databases, and in reality it generates weekly backups)? Or alternatively, do daily incrementals if daily granularity is ok and efficient.
This is not a big deal, but having similar policies for all backed up host unless necessary would be a huge simplification and also simplifies monitoring and debugging future issues/backup and recovery expectations.

Mon, Nov 4, 11:42 AM · Patch-For-Review, Goal, DBA, serviceops, Operations
elukey added a comment to T237016: Update router ACLs for newer bacula hosts.

Thanks a lot Jaime!

Mon, Nov 4, 11:29 AM · Operations, netops
elukey updated subscribers of T236818: Rerun sanitization before archiving eventlogging mysql data .

High level plan that I have in mind:

Mon, Nov 4, 9:02 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey updated subscribers of T233891: Drop Navigationtiming data entirely from mysql storage? .

@mforns When you are online can you ping me? I'd like to drop the above tables but with somebody triple checking what I am doing :)

Mon, Nov 4, 8:52 AM · Performance-Team (Radar), Analytics-Kanban, Analytics, Analytics-EventLogging
elukey moved T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster from In Progress to Done on the Analytics-Kanban board.
Mon, Nov 4, 8:39 AM · Analytics-Kanban, Analytics
elukey updated the task description for T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster.
Mon, Nov 4, 8:39 AM · Analytics-Kanban, Analytics
elukey updated subscribers of T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster.

Added documentation to https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#New_Worker_Installation_(12_disk,_2_flex_bay_drives_-_analytics1028-analytics1077) Cc: @Ottomata

Mon, Nov 4, 8:38 AM · Analytics-Kanban, Analytics
elukey added a comment to T212258: Create test Kerberos identities/accounts for some selected users in hadoop test cluster.

@Isaac @Neil_P._Quinn_WMF checking in, any issue with kerberos? Doubts/fears/etc.. ? :)

Mon, Nov 4, 8:30 AM · User-Elukey, Analytics-Kanban, Analytics

Thu, Oct 31

elukey added a comment to T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster.

Summary of things done if a rollback is needed:

Thu, Oct 31, 5:02 PM · Analytics-Kanban, Analytics
elukey added a comment to T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster.

The procedure should be as simple as:

Thu, Oct 31, 2:27 PM · Analytics-Kanban, Analytics
elukey moved T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster from Next Up to In Progress on the Analytics-Kanban board.
Thu, Oct 31, 2:16 PM · Analytics-Kanban, Analytics
elukey added a project to T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster: Analytics-Kanban.
Thu, Oct 31, 2:16 PM · Analytics-Kanban, Analytics
elukey created T236995: Enable TLS encryption for the MapReduce Shufflers in the Hadoop Analytics cluster.
Thu, Oct 31, 10:47 AM · Analytics-Kanban, Analytics
elukey committed rLPRI1c2c29de5ba1: Fix typo in hiera config for the Hadoop Cluster (authored by elukey).
Fix typo in hiera config for the Hadoop Cluster
Thu, Oct 31, 10:03 AM
elukey committed rLPRIffa45d335a16: Add secrets for the Hadoop Analytics TLS config in hiera (authored by elukey).
Add secrets for the Hadoop Analytics TLS config in hiera
Thu, Oct 31, 9:59 AM
elukey committed rLPRI5792c859337e: Add TLS fake cert files for analytics1042 (authored by elukey).
Add TLS fake cert files for analytics1042
Thu, Oct 31, 9:55 AM
elukey added a comment to T236818: Rerun sanitization before archiving eventlogging mysql data .

As part of this task I'd also clean up profile::mariadb::misc::eventlogging::sanitization from the db110[78] hosts :)

Thu, Oct 31, 6:53 AM · Analytics-Kanban, Analytics, Analytics-EventLogging
elukey added a comment to T235405: Build cergen for buster.

On puppetmaster2001 I cannot see /etc/apt/sources.list.d/buster-cergen.list, hence the new package version seems not available.. expected?

Thu, Oct 31, 6:30 AM · Patch-For-Review, Operations

Wed, Oct 30

elukey added a comment to T235405: Build cergen for buster.

Huh, can you add that to the debian/README?

Wed, Oct 30, 5:56 PM · Patch-For-Review, Operations
elukey added a comment to T235405: Build cergen for buster.

CERGEN=yes DIST=buster-wikimedia pdebuild is needed to build the package in case anybody needs it :)

Wed, Oct 30, 5:00 PM · Patch-For-Review, Operations
elukey added a comment to T236895: ArticlePlaceholder dashboard stopped tracking page views.

Hello! Do you have a coordinator that I can check in Hue? https://hue.wikimedia.org/oozie/list_oozie_coordinator/0015017-190822093211873-oozie-oozi-C/ seems the closest one and it works afaics..

Wed, Oct 30, 2:27 PM · Analytics, wikidata-tech-focus, Wikidata-Campsite, Wikidata, ArticlePlaceholder
elukey added a comment to T233661: Publish tls related info to webrequest via varnish.

I set up a test webrequest.conf on cp2001, and confirmed that the solution works!

Wed, Oct 30, 11:03 AM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
elukey added a comment to T233661: Publish tls related info to webrequest via varnish.

Sure! The JSON format of what we collect from Varnish for webrequest is in profile::cache::kafka::webrequest:

Wed, Oct 30, 10:43 AM · Patch-For-Review, Analytics-Kanban, observability, Operations, Analytics, Traffic
elukey closed T236876: git-sync on deployment-puppetmaster03 fails to rebase since days ago as Resolved.
Wed, Oct 30, 9:46 AM · Release-Engineering-Team, Beta-Cluster-Infrastructure
elukey added a comment to T236876: git-sync on deployment-puppetmaster03 fails to rebase since days ago.

Sure, even if I might not resolve the conflict in the best way and break Beta :)

Wed, Oct 30, 9:43 AM · Release-Engineering-Team, Beta-Cluster-Infrastructure
elukey triaged T236876: git-sync on deployment-puppetmaster03 fails to rebase since days ago as High priority.
Wed, Oct 30, 7:38 AM · Release-Engineering-Team, Beta-Cluster-Infrastructure

Tue, Oct 29

elukey added a comment to T204736: Move Analytics Report Updater to Python 3.

+1 it does look like a bug. The solution would be as simple as .encode('utf-8') (I ran that locally and it works, at least) however I haven't been able to verify which encoding the Graphite plaintext protocol expects, so a non-ascii character might cause meltdown.

Tue, Oct 29, 2:59 PM · Analytics-Kanban, Analytics
elukey reopened T204736: Move Analytics Report Updater to Python 3, a subtask of T204734: Deprecate Python 2 software from the Analytics infrastructure, as Open.
Tue, Oct 29, 2:50 PM · Analytics-Kanban
elukey reopened T204736: Move Analytics Report Updater to Python 3 as "Open".

@awight this is probably something that we didn't test it, as far as I know we don't use the graphite writer.. the stacktrace makes sense, a text string should not be passed to sock.send().. Will check the code asap!

Tue, Oct 29, 2:50 PM · Analytics-Kanban, Analytics
elukey claimed T234808: Upgrade python-kafka to 1.4.7.
Tue, Oct 29, 2:45 PM · Performance-Team (Radar), Analytics-Kanban, Analytics
elukey moved T234808: Upgrade python-kafka to 1.4.7 from Next Up to Done on the Analytics-Kanban board.
Tue, Oct 29, 2:42 PM · Performance-Team (Radar), Analytics-Kanban, Analytics
elukey set the point value for T234808: Upgrade python-kafka to 1.4.7 to 8.
Tue, Oct 29, 2:42 PM · Performance-Team (Radar), Analytics-Kanban, Analytics
elukey added a comment to T234808: Upgrade python-kafka to 1.4.7.

Ack. @elukey Let me know what day/time works best next week. I'll be on US Eastern Time that week.

Tue, Oct 29, 8:40 AM · Performance-Team (Radar), Analytics-Kanban, Analytics
elukey moved T219928: Move AQS logging to new logging pipeline from In Code Review to Paused on the Analytics-Kanban board.
Tue, Oct 29, 8:37 AM · Analytics-Kanban, User-Elukey, Patch-For-Review, observability, Analytics, Core Platform Team Legacy (Watching / External), Services (watching), service-runner, Wikimedia-Logstash, Operations
elukey moved T233231: Upgrade eventlogging to Python 3 from In Progress to Done on the Analytics-Kanban board.
Tue, Oct 29, 8:37 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey set the point value for T233231: Upgrade eventlogging to Python 3 to 13.
Tue, Oct 29, 8:37 AM · User-Elukey, Analytics-Kanban, Patch-For-Review, Analytics
elukey changed the status of T219928: Move AQS logging to new logging pipeline from Open to Stalled.

Pending T236757

Tue, Oct 29, 7:48 AM · Analytics-Kanban, User-Elukey, Patch-For-Review, observability, Analytics, Core Platform Team Legacy (Watching / External), Services (watching), service-runner, Wikimedia-Logstash, Operations
elukey changed the status of T219928: Move AQS logging to new logging pipeline, a subtask of T211125: Move service-runner to new logging infrastructure, from Open to Stalled.
Tue, Oct 29, 7:48 AM · observability, Core Platform Team Legacy (Watching / External), Patch-For-Review, service-runner, Wikimedia-Logstash, Operations
elukey created T236757: Upgrade service-runner on AQS to unblock rsyslog logging.
Tue, Oct 29, 7:47 AM · User-Elukey, Analytics, service-runner
elukey added a comment to T236180: Deploy search platform airflow service.

About the database - we could use a mariadb on the VM, but my experience with that is that if the db usage is a bit above the norm then the overall performance of the VM will suffer a lot (like it used to happen for Matomo). I'd start using an-coord1001, and then we may move the db out if we see that the usage is too high (I don't expect that but it might happen). We'll also create a user that is able to modify only one database, so there shouldn't be risks of hitting the rest inadvertently. @Ottomata what do you think?

Tue, Oct 29, 7:04 AM · Patch-For-Review, Discovery-Search
elukey updated the task description for T236180: Deploy search platform airflow service.
Tue, Oct 29, 7:01 AM · Patch-For-Review, Discovery-Search
elukey closed T236181: setup/install an-airflow1001.eqiad.wmnet on ganeti, a subtask of T236180: Deploy search platform airflow service, as Resolved.
Tue, Oct 29, 6:52 AM · Patch-For-Review, Discovery-Search
elukey closed T236181: setup/install an-airflow1001.eqiad.wmnet on ganeti as Resolved.

All right then, let's close this and re-open if necessary!

Tue, Oct 29, 6:52 AM · vm-requests, Operations, Discovery-Search

Mon, Oct 28

elukey added a comment to T231858: Archive data on eventlogging MySQL to analytics replica before decomisioning .

Ahhh ok right. Right.

sanitize data in the log databases

This shouldn't be necessary, right? IIUC, data should be sanitized already?

Mon, Oct 28, 7:23 PM · Analytics-Kanban, Analytics, Analytics-EventLogging