Ottomata (Andrew Otto)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 9 2014, 4:50 PM (223 w, 3 d)
Availability
Available
IRC Nick
ottomata
LDAP User
Ottomata
MediaWiki User
Ottomata [ Global Accounts ]

Recent Activity

Fri, Jan 18

Ottomata added a comment to T213566: Transferring data from Hadoop to production MySQL database.

I don't think writing from Hadoop directly to M2 master is a good idea. But it is not really my call.

Fri, Jan 18, 4:15 PM · User-Marostegui, Operations, Article-Recommendation, Analytics, Research

Thu, Jan 17

Ottomata updated subscribers of T214093: Modern Event Platform: Schema Guidelines and Conventions.

I'm opening this task now, as we are starting to design some new schemas as part of our goals for Q3 2018-2019. We plan to migrate the Mediawiki monolog avro schemas and events over to JSON Schema and use the new EventGate stream intake service. T214080: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate tracks this work.

Thu, Jan 17, 9:45 PM · Product-Analytics, Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata updated the task description for T185233: Modern Event Platform (TEC2).
Thu, Jan 17, 9:38 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata triaged T214093: Modern Event Platform: Schema Guidelines and Conventions as Normal priority.
Thu, Jan 17, 9:38 PM · Product-Analytics, Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata assigned T206889: Develop a library for JSON schema backwards incompatibility detection to Pchelolo.
Thu, Jan 17, 9:30 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata assigned T206812: Git Commit hook that adds a whole new file when a new version of schema is committed to Pchelolo.
Thu, Jan 17, 9:29 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata assigned T206814: CI Support for Schema Registry to Pchelolo.
Thu, Jan 17, 9:29 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T214080: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate from Backlog to Next Up on the EventBus board.
Thu, Jan 17, 9:28 PM · Services (watching), Discovery, Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T213566: Transferring data from Hadoop to production MySQL database.
to have a daemon on the mysql hosts

To clarify, it is unlikely these scripts would run on the mysql servers themselves. If not Hadoop, they'd run somewhere else, either a Ganeti instance as Dzahn suggests, or perhaps colocated with the recommendation-api service.

Thu, Jan 17, 8:24 PM · User-Marostegui, Operations, Article-Recommendation, Analytics, Research
Ottomata triaged T214080: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate as Normal priority.
Thu, Jan 17, 8:20 PM · Services (watching), Discovery, Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

FYI, the symlinked (or copied, or packaged .tgz) chart is necessary in the charts/ dir. Leaving it as a symlink since it is there for development.

Thu, Jan 17, 8:04 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T213566: Transferring data from Hadoop to production MySQL database.

We should have a clear separation of concerns and while the hadoop cluster is in charge of computing the data the task of updating the db needs to live in the service side.

Thu, Jan 17, 8:00 PM · User-Marostegui, Operations, Article-Recommendation, Analytics, Research
Ottomata added a comment to T213566: Transferring data from Hadoop to production MySQL database.

@bmansurov I think we can eventually figure out a way to get your dump files out of analytics to somewhere that can access mysql. Where and how to run your importer script is a different question that we probably need to talk with SRE about. Could we automate running it on the boxes where your service actually runs? Not sure.

Thu, Jan 17, 7:00 PM · User-Marostegui, Operations, Article-Recommendation, Analytics, Research
Ottomata added a comment to T213996: New MongoDB version is not DFSG-compatible, dropped by Debian.

nope! def not. must be some super legacy thang.

Thu, Jan 17, 6:50 PM · VisualEditor, Software-Licensing, Performance-Team, Operations
Ottomata committed rDEPLOYCHARTS26831754ce11: Helm chart for eventgate-analytics deployment (authored by Ottomata).
Helm chart for eventgate-analytics deployment
Thu, Jan 17, 5:57 PM
Ottomata updated subscribers of T213976: Workflow to be able to move data files computed in jobs from analytics cluster to production .

@fgiunchedi @mmodell got a sec sometime to discuss this? Here's a quick summary:

Thu, Jan 17, 5:26 PM · Discovery, Analytics
Ottomata committed rDEPLOYCHARTS5e5cdd29361b: Helm chart for eventgate-analytics deployment (authored by Ottomata).
Helm chart for eventgate-analytics deployment
Thu, Jan 17, 5:05 PM
Ottomata added a comment to T206789: Modern Event Platform: Schema Registry: Implementation.

@Pchelolo, so aside from the eventual HTTP based schema registry idea, we will still need (at least) one more git schema repository for analytics. This repo should use the same CI pipeline we build for event-schemas, but more people will have commit and merge access to it.

Thu, Jan 17, 4:47 PM · Core Platform Team Backlog (Watching / External), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata committed rDEPLOYCHARTS9b3ea751a24d: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Thu, Jan 17, 4:21 PM
Ottomata committed rDEPLOYCHARTSbff2f74cd30a: Add kafka-dev chart for local development (authored by Ottomata).
Add kafka-dev chart for local development
Thu, Jan 17, 4:15 PM
Ottomata added a comment to T213976: Workflow to be able to move data files computed in jobs from analytics cluster to production .

In our discussion yesterday, we mention both git lfs and swift as an option for this. I turned that idea down, but it seems it has been partially explored by RelEng. We should at least look into it.

Thu, Jan 17, 3:36 PM · Discovery, Analytics
Ottomata closed T213963: Include git in our alpine docker image on docker-registry.wikimedia.org, a subtask of T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline, as Declined.
Thu, Jan 17, 2:59 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata closed T213963: Include git in our alpine docker image on docker-registry.wikimedia.org as Declined.

For development phase, i'll use thee wmfdebug image. For prod deployment (outside of staging k8s), we'll build the schema repo clone into the app image itself.

Thu, Jan 17, 2:59 PM · serviceops, Analytics, EventBus
Ottomata added a comment to T213963: Include git in our alpine docker image on docker-registry.wikimedia.org.

Hm, well, I did consider doing this for final prod deployment.........yar ok. Nevermind. I'll DO IT!

Thu, Jan 17, 2:25 PM · serviceops, Analytics, EventBus
Ottomata added a comment to T213963: Include git in our alpine docker image on docker-registry.wikimedia.org.

Yar ok. I don't really want to put the schemas into EventGate, soooo I'll make a deploy repo after all! :)

Thu, Jan 17, 2:25 PM · serviceops, Analytics, EventBus

Wed, Jan 16

Ottomata updated subscribers of T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

@fselles, I'm not able to get the requirements.yaml repository to work. The only reason my symlink works is I think because the dependency will be looked for in charts/ by default. No matter what I put for repository, if I don't have the symlink in charts/, I get

Wed, Jan 16, 10:18 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata committed rDEPLOYCHARTS5df79943419e: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Wed, Jan 16, 10:16 PM
Ottomata committed rDEPLOYCHARTSd1eb4ab6fbdb: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Wed, Jan 16, 10:01 PM
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

That works for a java developer, but nobody else (including Quarry). Quarry is a Python app which is why I was looking for support in their python client.

Wed, Jan 16, 9:34 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata committed rDEPLOYCHARTS4078bcd2e75a: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Wed, Jan 16, 9:19 PM
Ottomata committed rDEPLOYCHARTSb28d3c4cea32: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Wed, Jan 16, 9:19 PM
Ottomata triaged T213963: Include git in our alpine docker image on docker-registry.wikimedia.org as Normal priority.
Wed, Jan 16, 8:11 PM · serviceops, Analytics, EventBus
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.
I was trying to figure out how this will actually look for the end user
Wed, Jan 16, 8:05 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Great meeting!

Wed, Jan 16, 5:02 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata committed rDEPLOYCHARTS4287754a04dc: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Wed, Jan 16, 2:21 PM
Ottomata committed rDEPLOYCHARTS114b97376ffe: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Wed, Jan 16, 2:21 PM
Ottomata committed rDEPLOYCHARTS137bd677a837: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Wed, Jan 16, 12:59 AM

Tue, Jan 15

Ottomata committed rDEPLOYCHARTSbd40b07522bc: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Tue, Jan 15, 9:35 PM
Ottomata updated subscribers of T213561: Discovery for Kafka cluster brokers.

Kafka doesn't support SRV. Hence my Round Robin DNS patch. After more discussion with @BBlack, I think I've decided to abandon this idea and just hardcode the Kafka brokers for now. @akosiaris mentioned that they have identified this problem (config management in helm charts) in other areas too, so I'll just hardcode for now and hope for a better future.

Tue, Jan 15, 8:23 PM · Patch-For-Review, Operations, Services (watching), EventBus, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Some links:

Tue, Jan 15, 6:59 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Tomorrow (Jan 15) we have a meeting with some SRE folks to revisit this. We've got the cloud-analytics Hadoop and Presto cluster up and running in Cloud VPS (thanks Andrew!). But as Brian says

Tue, Jan 15, 6:57 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata committed rDEPLOYCHARTS9e050fc3d65f: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Tue, Jan 15, 5:17 PM
Ottomata committed rDEPLOYCHARTSce69c52267bd: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Tue, Jan 15, 3:26 PM
Ottomata committed rDEPLOYCHARTS8538c2696b75: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Tue, Jan 15, 3:26 PM
Ottomata committed rDEPLOYCHARTSa0b6119f849e: Add kafka-single-node chart for local development (authored by Ottomata).
Add kafka-single-node chart for local development
Tue, Jan 15, 3:22 PM
Ottomata added a comment to T213561: Discovery for Kafka cluster brokers.

No no for me, all I want is an alias for the list of Kafka brokers in a given Kafka cluster. I don't need any DC failover stuff. Perhaps discovery is not the right word here. Round Robin DNS might be enough for me.

Tue, Jan 15, 3:13 PM · Patch-For-Review, Operations, Services (watching), EventBus, Analytics
Ottomata added a comment to T209732: Wire ORES recent_score events into Hadoop.

Yeah, this isn't the first time we've had this problem. It isn't actually that easy to solve, because the Kafka consumer doesn't advance if there are not messages in the topic, so we don't have a way of knowing if there are just no messages, or if the Kafka consumer is stalled/broken.

Tue, Jan 15, 2:44 PM · Patch-For-Review, Scoring-platform-team (Current), Analytics, ORES

Mon, Jan 14

Ottomata committed rDEPLOYCHARTS6b674db48323: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Mon, Jan 14, 9:44 PM
Ottomata committed rDEPLOYCHARTSa415c99ebe2b: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Mon, Jan 14, 8:23 PM
Ottomata committed rDEPLOYCHARTS7cb4eee2c39c: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Mon, Jan 14, 8:17 PM
Ottomata committed rDEPLOYCHARTS9cab02a51354: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Mon, Jan 14, 7:56 PM
Ottomata added a comment to T213081: Consider increasing kafka logging topic partitions.

Hm. In Kafka main clusters, we handle bursts of around 1000 msgs / sec per a single partition topic.

Mon, Jan 14, 7:28 PM · Patch-For-Review, Wikimedia-Logstash
Pchelolo awarded T213561: Discovery for Kafka cluster brokers a Love token.
Mon, Jan 14, 6:35 PM · Patch-For-Review, Operations, Services (watching), EventBus, Analytics
Ottomata added a comment to T208251: Modern Event Platform: Stream Intake Service: AJV usage security review.

Analytics is not planning on doing anything like this. If we were to do this, I think it should be a larger effort between RelEng and the SRE/Service Operations team.

Mon, Jan 14, 5:49 PM · Security-Team-Review-Active, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T213602: virtualpageview_hourly lacks data from December 17 on.

Thanks for filing Tilman, I'm refining this data now, and Oozie is scheduling the jobs now:
https://hue.wikimedia.org/oozie/list_oozie_coordinator/0001715-180905070129339-oozie-oozi-C/

Mon, Jan 14, 5:19 PM · Analytics

Fri, Jan 11

Ottomata added a comment to T210267: The continuous release pipeline should support more than one service per repo.

Seems like it would work, but it doesn't look like this provides much beyond the different variants in the blubber.config files. Could the stage and directory keys just be built into the variant config? Or, does that couple the blubber format to our CI pipeline in a way we don't want?

Fri, Jan 11, 8:36 PM · Release-Engineering-Team (Backlog), Release Pipeline (Blubber), Operations, ORES, Scoring-platform-team
Ottomata committed rDEPLOYCHARTSe22d52f70289: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Fri, Jan 11, 7:34 PM
Ottomata triaged T213564: Datacenter aware configs for EventGate topic prefixes as Normal priority.
Fri, Jan 11, 4:52 PM · serviceops, Services (watching), EventBus, Analytics
Ottomata updated subscribers of T213561: Discovery for Kafka cluster brokers.
Fri, Jan 11, 4:30 PM · Patch-For-Review, Operations, Services (watching), EventBus, Analytics
Ottomata triaged T213561: Discovery for Kafka cluster brokers as Normal priority.
Fri, Jan 11, 4:30 PM · Patch-For-Review, Operations, Services (watching), EventBus, Analytics

Wed, Jan 9

Ottomata closed T213344: unable to access Eventlogging via stat1006 as Resolved.

@Capt_Swing The issue is the presence of the .my.cnf file in your home dir on stat1006. It's being read by default and overriding the research-client.cnf conf file. Move or rename it out of the way and it will work!

Wed, Jan 9, 9:32 PM · SRE-Access-Requests, Operations
Ottomata updated subscribers of T202781: Determine then implement the proper way to log partial blocks.

OK cool, so then it sounds like at least for event schema stuff, adding partial block info to the event is not a blocker for partial blocks deployment; we can do that whenever we get around to it/need it.

Wed, Jan 9, 6:18 PM · Anti-Harassment, MediaWiki-Logging
Ottomata moved T212420: Move KafkaSSE development from Differential to Github from In Code Review to Done on the Analytics-Kanban board.
Wed, Jan 9, 5:11 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata committed rDEPLOYCHARTS2d4bb37fd544: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Wed, Jan 9, 3:43 PM
Ottomata updated subscribers of T202781: Determine then implement the proper way to log partial blocks.

Hi all! There are a lot of tickets about this stuff, and I'm just learning about it from Moriel's email so forgive me if I don't have a full understanding. Just an FYI, this will affect both eventbus and change-prop stuff. The EventBus extension is emitting user/blocks-change events on the BlockIpComplete hook. The user/blocks-change schema includes the blocks as they were before the change and as they are after.

Wed, Jan 9, 2:34 PM · Anti-Harassment, MediaWiki-Logging

Tue, Jan 8

Ottomata committed rDEPLOYCHARTSe9b2f3564d1d: [WIP] Helm chart for eventgate-analytics deployment (authored by Ottomata).
[WIP] Helm chart for eventgate-analytics deployment
Tue, Jan 8, 11:53 PM
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

I think kubectl describe pod is the most helpful. I'm onto something great here!

Tue, Jan 8, 11:48 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

Ok, I'm pretty close. I've got the charts deployed in minikube via helm. It seems my setup isn't quite right though, I think the image doesn't start properly. Got any tips for debugging? Not finding much help via helm / kubectl commands. (Tried both kubectl logs and attach)

Tue, Jan 8, 10:49 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

OH! Nevermind I see, that isn't an instruction...but a summary of what we are doing, never mind!

Tue, Jan 8, 9:51 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

@akosiaris In your doc , in the Start Up section, you mention 'Open grafana'. Can you elaborate here? :)

Tue, Jan 8, 9:49 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline.

@Pchelolo, hmmmm. eventgate in prod will need to have the event-schemas repo(s) available somehow. I'm working on getting the docker images and helm charts figured out. For the initial deployment prototype, I'm considering just making a blubber and CI based docker image that will be included in the eventgate docker image somewhere. This will work for a trial, but will be a bit inflexible, since it will mean that a new schema will require a rebuild and redeploy of eventgate.

Tue, Jan 8, 5:59 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata set the point value for T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline to 8.
Tue, Jan 8, 4:17 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users from In Code Review to Done on the Analytics-Kanban board.
Tue, Jan 8, 4:11 PM · Patch-For-Review, Analytics, Analytics-Kanban
Ottomata added a comment to T208251: Modern Event Platform: Stream Intake Service: AJV usage security review.

Ok thanks @sbasset. I've brought the form template over to this task and filled it out. @charlotteportero let me know if I'm missing anything.

Tue, Jan 8, 3:57 PM · Security-Team-Review-Active, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata renamed T208251: Modern Event Platform: Stream Intake Service: AJV usage security review from T206785: Modern Event Platform: Stream Intake Service: AJV usage security review to Modern Event Platform: Stream Intake Service: AJV usage security review.
Tue, Jan 8, 3:56 PM · Security-Team-Review-Active, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata reopened T208251: Modern Event Platform: Stream Intake Service: AJV usage security review as "Open".
Tue, Jan 8, 3:56 PM · Security-Team-Review-Active, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata reopened T208251: Modern Event Platform: Stream Intake Service: AJV usage security review, a subtask of T206785: Modern Event Platform: Stream Intake Service: Implementation, as Open.
Tue, Jan 8, 3:56 PM · Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T211605: Upgrade Superset to 0.28.1.

Hm, why was '0.26.3' in frozen-requirements.txt if you cherry picked it? It should be pointing at the github fork link. https://gerrit.wikimedia.org/r/#/c/analytics/superset/deploy/+/481054/1/frozen-requirements.txt

Tue, Jan 8, 2:22 PM · Patch-For-Review, Analytics-Kanban, Analytics
Ottomata added a comment to T208251: Modern Event Platform: Stream Intake Service: AJV usage security review.

@charlotteportero I don't think any of us knew there was a security review form. Can you link to it please?

Tue, Jan 8, 2:19 PM · Security-Team-Review-Active, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T37497: Implement a way to bring GitHub pull requests into gerrit.

the "Pull-request to Change"

Tue, Jan 8, 2:06 PM · Developer-Wishlist (2017), Developer-Advocacy, Gerrit

Mon, Jan 7

Ottomata added a comment to T212591: Provide Python 3.6 on SWAP.

Hmm, I'm not sure if it would work, but it might. We should be able to at least install the python 3.6 binary (we have deb packages for it now :) ). I don't think anyone would mind if we just did that, and you could try it on your own (since you are an ADVANCED SWAPPER :D )

Mon, Jan 7, 10:33 PM · Analytics, Analytics-SWAP, Contributors-Analysis, Product-Analytics
Ottomata added a comment to T212420: Move KafkaSSE development from Differential to Github.

If something like ^ worked nicely, my pro github arguments would be all moot and I'd be fine with gerrit all the way. :)

Mon, Jan 7, 9:30 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata added a comment to T212420: Move KafkaSSE development from Differential to Github.

Are you likely to get many contributors?

Mon, Jan 7, 8:59 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata added a comment to T210267: The continuous release pipeline should support more than one service per repo.

Q: would blubber's variants be enough to support the wsgi vs celery use case?

Mon, Jan 7, 8:54 PM · Release-Engineering-Team (Backlog), Release Pipeline (Blubber), Operations, ORES, Scoring-platform-team
Ottomata added a comment to T212420: Move KafkaSSE development from Differential to Github.

Most of the time I'm mostly unoppionated and am all for gerrit, especially when the target audience of the software is users that are (or should be) already used to using gerrit.

Mon, Jan 7, 8:48 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata moved T206815: Prototype in node intake service from In Progress to Done on the EventBus board.
Mon, Jan 7, 8:29 PM · Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Watching / External), Patch-For-Review, Analytics-Kanban, Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline from Next Up to In Progress on the EventBus board.
Mon, Jan 7, 8:29 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline from Next Up to In Progress on the Analytics-Kanban board.
Mon, Jan 7, 8:29 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a project to T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline: Analytics-Kanban.
Mon, Jan 7, 8:28 PM · Patch-For-Review, Analytics-Kanban, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T211833: [BUG] userAgent missing from all EventLogging analytics Hive tables between 2018-11-29 and 2018-11-14 from In Progress to Done on the Analytics-Kanban board.
Mon, Jan 7, 8:28 PM · Patch-For-Review, Analytics-Kanban, Product-Analytics, Analytics
Ottomata set the point value for T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users to 13.
Mon, Jan 7, 8:27 PM · Patch-For-Review, Analytics, Analytics-Kanban
Ottomata moved T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users from In Progress to In Code Review on the Analytics-Kanban board.
Mon, Jan 7, 8:27 PM · Patch-For-Review, Analytics, Analytics-Kanban
Ottomata moved T209453: Refine: Use Spark SQL instead of Hive JDBC from In Progress to In Code Review on the Analytics-Kanban board.
Mon, Jan 7, 8:27 PM · Analytics-Kanban, Analytics-EventLogging, Analytics
Ottomata moved T212420: Move KafkaSSE development from Differential to Github from Ready to Deploy to In Code Review on the Analytics-Kanban board.
Mon, Jan 7, 8:27 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata set the point value for T212420: Move KafkaSSE development from Differential to Github to 1.
Mon, Jan 7, 8:26 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata changed the point value for T206815: Prototype in node intake service from 0 to 21.
Mon, Jan 7, 8:26 PM · Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Watching / External), Patch-For-Review, Analytics-Kanban, Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T206815: Prototype in node intake service from In Progress to Done on the Analytics-Kanban board.
Mon, Jan 7, 8:25 PM · Core Platform Team (Modern Event Platform (TEC2)), Core Platform Team Backlog (Watching / External), Patch-For-Review, Analytics-Kanban, Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases from Next Up to In Code Review on the Analytics-Kanban board.
Mon, Jan 7, 8:25 PM · Patch-For-Review, Analytics-Kanban, Analytics, Analytics-Cluster, Operations, DBA
Ottomata moved T212420: Move KafkaSSE development from Differential to Github from Next Up to Ready to Deploy on the Analytics-Kanban board.
Mon, Jan 7, 8:25 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator
Ottomata added a comment to T212420: Move KafkaSSE development from Differential to Github.

The next time we build/deploy EventStreams, KafkaSSE from diffusion will no longer be used. Can we close this? Should we delete the repo from diffusion?

Mon, Jan 7, 8:25 PM · Patch-For-Review, Analytics-Kanban, Analytics, Wikimedia-Stream, Phabricator