Ottomata (Andrew Otto)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 9 2014, 4:50 PM (206 w, 1 d)
Availability
Available
IRC Nick
ottomata
LDAP User
Ottomata
MediaWiki User
Unknown

Recent Activity

Yesterday

Ottomata added a comment to T188136: Migrate Mediawiki Monolog Kafka producer to new Stream Intake Service (as part of Modern Event Platform).

Just edited the task description. If possible, we will try to get these logs off of Kafka analytics by eventually moving them to new Modern Event Platform components.

Thu, Sep 20, 2:17 PM · Discovery, Analytics, Patch-For-Review, Analytics-Cluster
Ottomata renamed T188136: Migrate Mediawiki Monolog Kafka producer to new Stream Intake Service (as part of Modern Event Platform) from Migrate Mediawiki Monolog Kafka producer to Kafka Jumbo to Migrate Mediawiki Monolog Kafka producer to new Stream Intake Service (as part of Modern Event Platform).
Thu, Sep 20, 2:17 PM · Discovery, Analytics, Patch-For-Review, Analytics-Cluster
Ottomata merged T204952: Migrate mediawiki current usage of monolog to log to EventBus to use the new scalable event intake service into T188136: Migrate Mediawiki Monolog Kafka producer to new Stream Intake Service (as part of Modern Event Platform).
Thu, Sep 20, 2:15 PM · Discovery, Analytics, Patch-For-Review, Analytics-Cluster
Ottomata merged task T204952: Migrate mediawiki current usage of monolog to log to EventBus to use the new scalable event intake service into T188136: Migrate Mediawiki Monolog Kafka producer to new Stream Intake Service (as part of Modern Event Platform).
Thu, Sep 20, 2:15 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata updated the task description for T204952: Migrate mediawiki current usage of monolog to log to EventBus to use the new scalable event intake service .
Thu, Sep 20, 2:14 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201963: RFC: Modern Event Platform: Scalable Event Intake Service.

Any thoughts from @Pchelolo @mobrovac ?

Thu, Sep 20, 1:54 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201963: RFC: Modern Event Platform: Scalable Event Intake Service.

I'm assuming we'd have to keep maintaining our fork which I don't think is desirable.

Yeah we'd likely be maintaining a fork, but hopefully most of the changes would be in an extended API class. It does require some modification to the http routes, but that logic is minimal.

Thu, Sep 20, 1:36 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T203814: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError).

Imperative we get a count - using statsd

Thu, Sep 20, 1:26 PM · Patch-For-Review, Wikimedia-Site-requests, Analytics, MobileFrontend (MobileFrontend.js), Readers-Web-Backlog
Ottomata added a comment to T204088: exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster.

@Krenair oh ya? Maybe I just didn't know that! So query_resources will work?! If so I can probably just enable the prometheus based kafka monitoring in deployment-prep.

Thu, Sep 20, 1:24 PM · Beta-Cluster-Infrastructure, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1), Operations, Puppet
Ottomata added a comment to T166351: Revision create events are made for all history of a page when it is undeleted.

THANKS PETRRRRR!

Thu, Sep 20, 1:20 PM · Services (done), MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), ChangeProp

Tue, Sep 18

Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

Sure! Can you invite me to your the Product Analytics team meeting next Tuesday the 25th?

Oh and invite @Milimetric too plz :)

Tue, Sep 18, 2:05 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201963: RFC: Modern Event Platform: Scalable Event Intake Service.

Heya @fgiunchedi, could you take a look at this (and the parent) task and tell me what you think? We are considering scheduling this work for Q2. My current hope/suggestion is to use Kafka REST Proxy for this, which would mean eventually (probably not in next quarter) a publicly exposed Kafka REST Proxy (JVM based) HTTP produce endpoint. Thoughts?

Tue, Sep 18, 1:52 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T204685: cloudvps: reportcard project trusty deprecation.

I betcha we can just delete this whole project. reportcard.wmflabs.org now redirects to https://analytics.wikimedia.org/dashboards/reportcard/#new-editors. Ping @Milimetric @Nuria .

Tue, Sep 18, 1:05 PM · Cloud-VPS
Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

As a data analyst or product manager, I want a canonical place where I can easily draft schema definitions and implementation details in collaboration with product engineers during implementation (example), document and access them once a schema is live, and correct and amend them later as needed.

Tue, Sep 18, 1:01 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

@Ottomata could commit to presenting the plans to the Product Analytics team before they're final?

Sure! Can you invite me to your the Product Analytics team meeting next Tuesday the 25th?

Tue, Sep 18, 12:38 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban

Thu, Sep 13

Ottomata added a comment to T200822: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete.

It's already been broken for a few weeks. We don't need to delete the data at all, but the change Luca is working on will cause this job to run with the webrequest_text data partition, which is a lot more than webrequest_misc. We can do it, but we'd rather not if we don't have to!

Thu, Sep 13, 4:22 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata added a comment to T204183: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached".

Luca did some restarts of the main Kafka clusters recently, I wonder if these correlate to those restarts.

Thu, Sep 13, 1:38 PM · EventBus, WMF-JobQueue, Analytics, Wikimedia-production-error
Ottomata added a comment to T203132: Streamline Superset signup and authentication.

When superset (and Flask) auto create the user account in superset on first LDAP login, the only information they have from ldap is the username. Superset's MySQL database has a unique key constraint on email address. The first user who signed up this way got a blank email address of '-'. The next person who tried also got an email address of '-', but the login failed because superset couldn't create another user record with the same '-' email address.

Thu, Sep 13, 1:32 PM · Analytics, Contributors-Analysis, Product-Analytics

Wed, Sep 12

Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

What happens with metadata will probably interest TechComm (e.g. allowing remote clients to configure their sampling rates via a service), but we've removed that discussion from this RFC. I think this one can be moved along too.

Wed, Sep 12, 8:21 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T203132: Streamline Superset signup and authentication.

Ok, this kinda works, but then it doesn't. We need to find a way to get the user's email address from ldap when they first sign up. I think this will need to be upstreamed into flask appbuilder.

Wed, Sep 12, 7:36 PM · Analytics, Contributors-Analysis, Product-Analytics
Ottomata updated the task description for T201963: RFC: Modern Event Platform: Scalable Event Intake Service.
Wed, Sep 12, 5:34 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas .

Naw, i'll just do it as part of this one, thanks.

Wed, Sep 12, 4:50 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata updated subscribers of T204088: exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster.
Wed, Sep 12, 3:51 PM · Beta-Cluster-Infrastructure, Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1), Operations, Puppet
Ottomata moved T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas from In Progress to Done on the Analytics-Kanban board.
Wed, Sep 12, 1:32 PM · Patch-For-Review, Analytics-Kanban, Analytics

Tue, Sep 11

Ottomata set the point value for T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas to 5.
Tue, Sep 11, 4:35 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas from Next Up to In Progress on the Analytics-Kanban board.
Tue, Sep 11, 4:35 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T203804: Refactor Refine job scalaopt to use property files and CLI overrides from In Progress to In Code Review on the Analytics-Kanban board.
Tue, Sep 11, 3:10 PM · Patch-For-Review, Analytics-Kanban
Ottomata added a comment to T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas .

Looking in the consumer logs as you suggest was very easy (but less fun :p). Between Aug 26 06:50:58 (our oldest consumer log) and now, there were 110 unique schemas inserted into mysql:

Tue, Sep 11, 2:17 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T203814: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError).

EL is really not the best tool to do error logging

Tue, Sep 11, 2:02 PM · Patch-For-Review, Wikimedia-Site-requests, Analytics, MobileFrontend (MobileFrontend.js), Readers-Web-Backlog
Ottomata added a comment to T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas .

That's true it would be! :o It would also be in the eventlogging-valid-mixed files too.

Tue, Sep 11, 2:00 PM · Patch-For-Review, Analytics-Kanban, Analytics

Mon, Sep 10

Ottomata added a comment to T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas .

Alright! First we need a list of active schemas that are not blacklisted. Those will all go to the eventlogging-valid-mixed topic.

Mon, Sep 10, 9:06 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a project to T203596: Flip blacklist for MySQL eventlogging consumer to be a whitelist of allowed schemas : Analytics-Kanban.
Mon, Sep 10, 8:51 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T203814: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError).

Let's please postpone any sampling rate changes until this schema is blacklisted in MySQL and its events only go to hadoop.

Mon, Sep 10, 4:26 PM · Patch-For-Review, Wikimedia-Site-requests, Analytics, MobileFrontend (MobileFrontend.js), Readers-Web-Backlog

Fri, Sep 7

Ottomata added a comment to T203814: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError).

I think blacklisting from MySQL might be sensible anyway, given a bug in MediaWiki:Mobile.js by an editor could spike the number of events.

Fri, Sep 7, 6:19 PM · Patch-For-Review, Wikimedia-Site-requests, Analytics, MobileFrontend (MobileFrontend.js), Readers-Web-Backlog
Ottomata closed T203561: Remove user "albe" from the wmde LDAP group as Resolved.

Looks like we can resolve, thanks.

Fri, Sep 7, 5:31 PM · Operations, LDAP-Access-Requests
Ottomata assigned T203561: Remove user "albe" from the wmde LDAP group to Dzahn.
Fri, Sep 7, 5:30 PM · Operations, LDAP-Access-Requests
Ottomata triaged T203494: Requesting access to Root for Giovanni Tirloni as Normal priority.
Fri, Sep 7, 5:29 PM · Patch-For-Review, SRE-Access-Requests, Operations
Ottomata assigned T203494: Requesting access to Root for Giovanni Tirloni to Andrew.
Fri, Sep 7, 5:28 PM · Patch-For-Review, SRE-Access-Requests, Operations
Ottomata added a comment to T203814: Turn on MinervaErrorLogSamplingRate (Schema:WebClientError).

Naw that should be fine! In general

Fri, Sep 7, 4:59 PM · Patch-For-Review, Wikimedia-Site-requests, Analytics, MobileFrontend (MobileFrontend.js), Readers-Web-Backlog
Ottomata moved T203804: Refactor Refine job scalaopt to use property files and CLI overrides from Next Up to In Progress on the Analytics-Kanban board.
Fri, Sep 7, 4:02 PM · Patch-For-Review, Analytics-Kanban
Ottomata created T203804: Refactor Refine job scalaopt to use property files and CLI overrides.
Fri, Sep 7, 3:53 PM · Patch-For-Review, Analytics-Kanban

Thu, Sep 6

Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

@daniel what's next for the RFC? How do I keep it moving?

Thu, Sep 6, 3:40 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T203635: Replace the Analytics HDFS/Yarn masters (hardware refresh).

greaaaattttttttt

Thu, Sep 6, 1:20 PM · Patch-For-Review, User-Elukey, Analytics-Kanban, Analytics
Ottomata updated the task description for T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.
Thu, Sep 6, 12:33 AM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

Ya makes a lot of sense. In addition to high level type documentation you are mentioning, the Better Use of Data Working group is calling out a need for what they are calling a 'data dictionary'. This might also end up being wiki documentation with good conventions, but more likely it will end up being something more queryable. I agree it makes sense to document these things in a canonical place separate from the schemas, especially since the schemas will no longer be mapped directly to their usages.

Thu, Sep 6, 12:33 AM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban

Wed, Sep 5

Ottomata added a project to T203597: CentralNoticeImpression occasionaly fails validation on device enum field: Fundraising-Analysis.
Wed, Sep 5, 6:39 PM · Analytics, Fundraising-Analysis
Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

Just had a shortish meeting with Tilman and Josh Minor. I don't think we resolved much, but Josh is going to work with Tilman and others via the Better Use of Data Working group to address some of Tilman's concerns. In the meantime, @Tbayer it'd be helpful if you were able to list your use cases that you are feel are not captured in T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service and/or ones that you think using git for schema storage would make things difficult for.

Wed, Sep 5, 6:39 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata created T203597: CentralNoticeImpression occasionaly fails validation on device enum field.
Wed, Sep 5, 6:22 PM · Analytics, Fundraising-Analysis
Ottomata moved T203592: CentralNoticeImpression event schema too large for MySQL from Next Up to Done on the Analytics-Kanban board.
Wed, Sep 5, 6:18 PM · Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging
Ottomata set the point value for T203592: CentralNoticeImpression event schema too large for MySQL to 1.
Wed, Sep 5, 6:18 PM · Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging
Ottomata created T203592: CentralNoticeImpression event schema too large for MySQL.
Wed, Sep 5, 6:08 PM · Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging
Ottomata updated the task description for T192642: Upgrade Analytics infrastructure to Debian Stretch.
Wed, Sep 5, 4:13 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T192641: Reimage thorium to Debian Stretch.

I think that was failing during the reinstall, it looks fine now.

Wed, Sep 5, 3:27 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T201963: RFC: Modern Event Platform: Scalable Event Intake Service.

For this component, I don't think it matters much. The event intake is just about a stanadlone HTTP API that accepts and validates events. Later on we'll think about stream processing platforms. That'll be much more relevant to dep tracking engine requirements.

Wed, Sep 5, 3:03 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201963: RFC: Modern Event Platform: Scalable Event Intake Service.

@daniel I heard you might have some questions/thoughts about this! Find me in IRC to chat?

Wed, Sep 5, 2:36 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata set the point value for T192641: Reimage thorium to Debian Stretch to 3.
Wed, Sep 5, 2:34 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T192641: Reimage thorium to Debian Stretch from In Progress to Done on the Analytics-Kanban board.
Wed, Sep 5, 2:34 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T192641: Reimage thorium to Debian Stretch.

Done!

Wed, Sep 5, 2:34 PM · Patch-For-Review, Analytics-Kanban, Analytics

Tue, Sep 4

Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

@kchapman I'm not totally familiar with the RFC process. What's next? Does this need to go through another RFC IRC meeting?

Tue, Sep 4, 5:36 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T192641: Reimage thorium to Debian Stretch.

FYI, this is scheduled to happen tomorrow Wed Sept 5 at about 13:30 UTC

Tue, Sep 4, 3:35 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T201409: Harmonise the identification of requests across our stack.

But I'm unsure whether we should also standardise the value format of the header (as UUID).

Tue, Sep 4, 2:52 PM · Performance-Team (Radar), Patch-For-Review, Operations, Services (designing), TechCom-RFC, User-mobrovac, Traffic
Ottomata added a comment to T203374: Request increased quota for analytics Cloud VPS project.

Waiting until after oct will be fine.

Tue, Sep 4, 2:51 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
Ottomata added a comment to T203374: Request increased quota for analytics Cloud VPS project.

I'd also add that we will eventually need some space to test the new data lake service (likely running apache presto) in the analytics project. This will also be 3ish medium-large instances. We'd like to keep these instances online for testing before making changes in production.

Tue, Sep 4, 2:50 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Thu, Aug 30

Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

the difference is that as envisioned on this system they are not being edited in git.

Not necessarily, they could still be in git, just not mw-config for SWAT deployment.

Thu, Aug 30, 7:18 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata updated subscribers of T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

In yesterdays meeting for T201643: RFC: Modern Event Platform: Schema Registry, we decided to split out the Schema Usage Configuration Service from the Schema Registry component, at least for RFC purposes. Before I make an RFC for a configuration service, I'd like to have a little more back and forth discussion here (perhaps a meeting?) to flush it out a bit more.

Thu, Aug 30, 6:50 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

1.) these don't necessarily correspond to the most important needs
2.) as discussed above, much of these just describe current practice and capabilities and don't support the need for a switch to a new platform
3.) they all seem to be relegated to the "Future version", with the MVP focusing on the engineering studies

Thu, Aug 30, 5:53 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata updated the task description for T201068: Modern Event Platform: Scalable Event Intake.
Thu, Aug 30, 5:48 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

I've edited the RFC description to account for:

  • add more info about schema usage metadata vs schema registry
  • modify RFC to be about git based storage for schemas
  • add info about file based versioning
Thu, Aug 30, 5:47 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata renamed T201643: RFC: Modern Event Platform: Schema Registry from RFC: Modern Event Platform: Schema Registry / Metadata Service to RFC: Modern Event Platform: Schema Registry.
Thu, Aug 30, 5:45 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata renamed T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service from Modern Event Platform: Schema Registry to Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.
Thu, Aug 30, 5:25 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata set the point value for T203115: Create .deb package for Presto to 5.
Thu, Aug 30, 3:57 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T203115: Create .deb package for Presto from Next Up to In Code Review on the Analytics-Kanban board.
Thu, Aug 30, 3:57 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T203132: Streamline Superset signup and authentication.

Make the choice between "wiki" and "shell" usernames (some LDAP services want one, some want another) less confusing. I imagine it's out of scope to standardize on one

Thu, Aug 30, 1:18 PM · Analytics, Contributors-Analysis, Product-Analytics
Ottomata updated subscribers of T203132: Streamline Superset signup and authentication.

BTW, there may be some interest from the newish security team in SRE to work on an analytics VPN. We aren't sure, but they just did an audit of our stack and they saw the need for one as well. That doesn't mean we'll get one, but at least there's maybe a way! Ping @chasemp :)

Thu, Aug 30, 1:16 PM · Analytics, Contributors-Analysis, Product-Analytics

Wed, Aug 29

Ottomata updated the task description for T201963: RFC: Modern Event Platform: Scalable Event Intake Service.
Wed, Aug 29, 8:55 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata updated the task description for T201963: RFC: Modern Event Platform: Scalable Event Intake Service.
Wed, Aug 29, 8:52 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T201643: RFC: Modern Event Platform: Schema Registry.

Copy/pasting this comment for visibility:

Wed, Aug 29, 8:49 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata created T203115: Create .deb package for Presto.
Wed, Aug 29, 8:32 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T201063: Modern Event Platform: Schema Registry + Schema Usage Metadata Configuration Service.

How is this blocked on implementing a (new) schema registry?

Wed, Aug 29, 8:23 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata added a comment to T202100: Intermittent git-fat failure during deploy.

Done, worked everywhere except:

Wed, Aug 29, 6:38 PM · Patch-For-Review, Operations, Scap
Ottomata added a comment to T201939: rack/setup/install an-master100[12].eqiad.wmnet.

@Cmjohnson Rob's latest comment is what I thought we agreed to:

Wed, Aug 29, 4:22 PM · Patch-For-Review, User-Elukey, Analytics, Operations
Ottomata changed the point value for T201963: RFC: Modern Event Platform: Scalable Event Intake Service from 0 to 5.
Wed, Aug 29, 2:18 PM · TechCom-RFC, Services (watching), Analytics-EventLogging, EventBus, Analytics, Analytics-Kanban
Ottomata moved T202011: Move internal sites hosted on thorium to ganeti instance(s) from In Code Review to Done on the Analytics-Kanban board.
Wed, Aug 29, 2:17 PM · Patch-For-Review, Operations, Analytics-Kanban, Analytics
Ottomata moved T201430: Update superset (we have 0.20.6, 0.26.3 is available) from In Code Review to Done on the Analytics-Kanban board.
Wed, Aug 29, 2:17 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata added a comment to T202812: Install pyArrow in Cluster.

@diego I GOT IT! Try it out, lemme know if it works.

Wed, Aug 29, 2:06 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T202812: Install pyArrow in Cluster from In Code Review to Done on the Analytics-Kanban board.
Wed, Aug 29, 2:06 PM · Patch-For-Review, Analytics-Kanban, Analytics

Tue, Aug 28

Ottomata set the point value for T202011: Move internal sites hosted on thorium to ganeti instance(s) to 13.
Tue, Aug 28, 9:27 PM · Patch-For-Review, Operations, Analytics-Kanban, Analytics
Ottomata moved T202812: Install pyArrow in Cluster from Next Up to In Code Review on the Analytics-Kanban board.
Tue, Aug 28, 9:27 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata set the point value for T202812: Install pyArrow in Cluster to 8.
Tue, Aug 28, 9:27 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T202812: Install pyArrow in Cluster.

Download the pyarrow (0.8.0) release and include it in the spark2 .deb package we make, and install it on all nodes in /usr/lib/spark2/python somewhere.

Tue, Aug 28, 9:26 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata changed the point value for T201430: Update superset (we have 0.20.6, 0.26.3 is available) from 3 to 5.
Tue, Aug 28, 5:51 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata claimed T192641: Reimage thorium to Debian Stretch.
Tue, Aug 28, 4:51 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T192641: Reimage thorium to Debian Stretch from Next Up to In Progress on the Analytics-Kanban board.
Tue, Aug 28, 4:51 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata moved T201430: Update superset (we have 0.20.6, 0.26.3 is available) from Done to In Code Review on the Analytics-Kanban board.
Tue, Aug 28, 4:48 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata moved T202011: Move internal sites hosted on thorium to ganeti instance(s) from In Progress to In Code Review on the Analytics-Kanban board.
Tue, Aug 28, 4:48 PM · Patch-For-Review, Operations, Analytics-Kanban, Analytics
Ottomata moved T201430: Update superset (we have 0.20.6, 0.26.3 is available) from In Progress to Done on the Analytics-Kanban board.
Tue, Aug 28, 4:48 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata added a comment to T201430: Update superset (we have 0.20.6, 0.26.3 is available).

This is done! Plz try it out :)

Tue, Aug 28, 4:47 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata added a comment to T202812: Install pyArrow in Cluster.

Alright, I spent a few hours trying to solve this for the distributed YARN case, and it is not easy! I'm not giving up yet, but I think the solution will be one of:

Tue, Aug 28, 12:46 AM · Patch-For-Review, Analytics-Kanban, Analytics

Mon, Aug 27

Ottomata claimed T202011: Move internal sites hosted on thorium to ganeti instance(s).
Mon, Aug 27, 6:57 PM · Patch-For-Review, Operations, Analytics-Kanban, Analytics
Ottomata added a comment to T170606: Add Accept header to webrequest logs.

@Pchelolo we discussed this in standup today.

Mon, Aug 27, 5:31 PM · Operations, Traffic, Services (blocked), Analytics