Ottomata (Andrew Otto)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Oct 9 2014, 4:50 PM (214 w, 4 d)
Availability
Available
IRC Nick
ottomata
LDAP User
Ottomata
MediaWiki User
Ottomata [ Global Accounts ]

Recent Activity

Today

Ottomata updated subscribers of T206037: Consolidate articlequality and itemquality models into a "model family".

Since change-prop is responsible for emitting the revision-score event, we'll have to make sure that these fields are in the event schema, and that change-prop sets the properly. Also ping @Pchelolo.

Mon, Nov 19, 5:31 PM · Scoring-platform-team (Current), MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), MediaWiki-extensions-PageCuration, Growth-Team, Patch-For-Review, ORES
Ottomata added a comment to T197000: Modify revision-score schema so that model probabilities won't conflict.

We plan to deploy this Monday Nov 26th.

Mon, Nov 19, 4:29 PM · Scoring-platform-team, Analytics-Kanban, Patch-For-Review, User-Ladsgroup, Services (watching), ORES, Analytics, EventBus
Ottomata added a comment to T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet.

Ping on this! I know it is TG week so things might be slow, but I'm checking in anyway :)

Mon, Nov 19, 4:06 PM · ops-eqiad, Operations, User-Elukey, Analytics
Ottomata updated subscribers of T205437: Resurrect eventlogging_EventError logging to in logstash.

Sam, that'd be great! Find me and Marcel (mforns) on IRC in #wikimedia-analytics and lets discuss. Actually...@fgiunchedi or @herron might have something helpful to say, as they are currently implementing some new Kafka -> Logstash integration.

Mon, Nov 19, 2:34 PM · Analytics, Analytics-EventLogging

Thu, Nov 15

Ottomata added a comment to T206785: Modern Event Platform: Stream Intake Service: Implementation.

We need a name! Have been brainstorming over on https://etherpad.wikimedia.org/p/event-platform. The current three top contenders:

Thu, Nov 15, 2:50 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T208756: New Cloud VPS project 'cloud-analytics'.

PERFECT, thanks Andrew! We'll try to fit what we need in the stock VMs first. Sounds good.

Thu, Nov 15, 2:32 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)

Wed, Nov 14

Ottomata added a comment to T206824: Decide whether to use schema references in the schema registry.

If they work, I'm fine with them as long as...

Wed, Nov 14, 9:03 PM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T206889: Develop a library for JSON schema backwards incompatibility detection.

Oook thanks Petr!

Wed, Nov 14, 8:51 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
elukey awarded T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0 a Like token.
Wed, Nov 14, 6:42 AM · Patch-For-Review, Analytics-Kanban
Ottomata created T209453: Refine: Use Spark SQL instead of Hive JDBC .
Wed, Nov 14, 3:08 AM · Analytics-EventLogging, Analytics

Tue, Nov 13

Ottomata added a comment to T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0.
18/11/13 20:52:34 INFO RefineMonitor: No dataset targets in /wmf/data/raw/eventlogging between 2018-11-11T20:50:26.600Z and 2018-11-13T16:50:26.601Z need refinement to /wmf/data/event
Tue, Nov 13, 8:57 PM · Patch-For-Review, Analytics-Kanban
Ottomata set the point value for T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0 to 5.
Tue, Nov 13, 8:57 PM · Patch-For-Review, Analytics-Kanban
Ottomata moved T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0 from Next Up to Done on the Analytics-Kanban board.
Tue, Nov 13, 8:57 PM · Patch-For-Review, Analytics-Kanban
Ottomata updated subscribers of T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet.

IIUC it has to be row B for them to be used as Cloud Virts. @Andrew to confirm. If they can go any row, then they should be spread out as evenly amongst as many rows as possible.

Tue, Nov 13, 7:51 PM · ops-eqiad, Operations, User-Elukey, Analytics
Ottomata added a comment to T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0.

When ALTERing Hive tables, DataFrameToHive uses a manual JDBC connection to Hive,
rather than Spark SQL. This is a work around for https://issues.apache.org/jira/browse/SPARK-23890.
Spark doesn't allow issuing of ALTER statements via spark.sql().

Tue, Nov 13, 7:19 PM · Patch-For-Review, Analytics-Kanban
Ottomata created T209407: EventLogging Hive Refine broken after upgrade to CDH 5.15.0.
Tue, Nov 13, 7:18 PM · Patch-For-Review, Analytics-Kanban
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

@Cmjohnson I updated T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet to reflect the new naming. Please proceed and then assign to Cloud VPS folks for OS install / puppetization setup as Cloud Virt nodes.

Tue, Nov 13, 4:20 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata renamed T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet from rack/setup/install ca-worker100[1-5].eqiad.wmnet to rack/setup/install cloudvirtan100[1-5].eqiad.wmnet.
Tue, Nov 13, 4:19 PM · ops-eqiad, Operations, User-Elukey, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Ok!

Tue, Nov 13, 4:17 PM · Analytics-Kanban, netops, Operations, Analytics

Mon, Nov 12

Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Ok, @Cmjohnson your call then: we'd prefer cloudvirtanalytics1xxx, but if that is too long, then use cloudvirtan1xxx. How can should we now proceed?

Mon, Nov 12, 10:53 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

Ok +1

Mon, Nov 12, 8:19 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Let's make this happen! @Andrew are you ok with cloudvirt-anXXXX? @Cmjohnson would prefer to coordinate racking of these nodes on this ticket or on T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet (we'll have to rename the nodes).

Mon, Nov 12, 7:40 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata updated subscribers of T202489: Copy monthly XML files from public-dumps to HDFS.

@ArielGlenn, they need to be copied into HDFS inside of Hadoop, not just available on a regular filesystem.

Mon, Nov 12, 7:23 PM · Patch-For-Review, Analytics-Kanban, Research, Analytics
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

It has pros and cons: The downside of using backports is that it is a moving target, while we don't necessary want to follow up update. With a locally imported package we can more easily control when to upgrade (which is particularly important for fleet-wide installed packages where we try to keep oldstable and stable at the same version).

Mon, Nov 12, 6:03 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

The only one there that we should check on for sure is wdqs1009.eqiad.wmnet, but I'm pretty sure they use the Java client, not librdkafka for their updater service.

Mon, Nov 12, 5:47 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

Oh hm. There are no prod services running on the stat boxes. We can (and should) upgrade there anyway.

Mon, Nov 12, 5:46 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

I'm fairly certain there shouldn't be any streth hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

Mon, Nov 12, 5:40 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

Mon, Nov 12, 5:30 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
Ottomata added a comment to T208679: eventlogging Dockerfile doesn't work.

Could you get it up and running in just a python virtualenv instead of Docker? Or, mediawiki-vagrant has an eventlogging role!

Mon, Nov 12, 3:50 PM · Analytics, Analytics-EventLogging
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

I'd prefer if we used 'analytics' instead of 'data lake'. Can we do cloudvirtanXXXX? cloudvirt-anXXXX?

Mon, Nov 12, 3:45 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T208679: eventlogging Dockerfile doesn't work.

Ha yea Petr doesn't really have anything to do with this. Hm.

Mon, Nov 12, 3:42 PM · Analytics, Analytics-EventLogging

Mon, Nov 5

Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Cool, done: T208756: New Cloud VPS project 'cloud-analytics'

Mon, Nov 5, 6:03 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata updated the task description for T208756: New Cloud VPS project 'cloud-analytics'.
Mon, Nov 5, 6:02 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Ottomata closed T207205: Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster, a subtask of T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users, as Declined.
Mon, Nov 5, 6:02 PM · Patch-For-Review, Analytics, Analytics-Kanban
Ottomata closed T207205: Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster as Declined.

Declining in favor of T208756: New Cloud VPS project 'cloud-analytics'

Mon, Nov 5, 6:02 PM · Operations, vm-requests, Analytics, Analytics-Kanban
Ottomata created T208756: New Cloud VPS project 'cloud-analytics'.
Mon, Nov 5, 6:02 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests)
Ottomata updated subscribers of T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Ok, so plan:

Mon, Nov 5, 5:34 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207205: Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster.

Hold on this while we figure out T207321.

Mon, Nov 5, 5:33 PM · Operations, vm-requests, Analytics, Analytics-Kanban
Ottomata added a comment to T201068: Modern Event Platform: Stream Intake Service.

If this endpoint were to be added to your service, the EventLogging Core JavaScript library would need a lot (?) of work to make it capable of batch sending events after some offline period, which doesn't appear to be in scope for this project

Mon, Nov 5, 2:48 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban

Sun, Nov 4

Addshore awarded T185233: Modern Event Platform (TEC2) a Love token.
Sun, Nov 4, 4:33 PM · Core Platform Team Backlog (Epic), Core Platform Team (Modern Event Platform (TEC2)), Goal, Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban

Fri, Nov 2

Ottomata added a comment to T201068: Modern Event Platform: Stream Intake Service.

K here's our thoughts about this:

Fri, Nov 2, 8:31 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata added a comment to T208604: Ingest geocoded data from event database into druid .

But why?! It's a map on webrequest.

Fri, Nov 2, 4:31 PM · Analytics
Ottomata added a comment to T201068: Modern Event Platform: Stream Intake Service.

As an engineer, I want to batch produce many events at once so mobile apps can produce events after an offline period.

Fri, Nov 2, 3:13 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Interesting...I suppose this service isn't quite as critical as our prod ones. Maybe this is ok?

Fri, Nov 2, 2:59 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

1/ Will all those hosts need to be in the same vlan/row (eg. cloud-hosts1-b-eqiad)? Ideally they should be spread across multiple rows to avoid the scenario of one row (aka. failure domain) outage taking the whole service down

Fri, Nov 2, 2:43 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

OH duh, I forgot to account for the HDFS replication. Right. Ok in that case, let's go with option 2. Is there room on the switches for 10g? :D

Fri, Nov 2, 2:14 PM · Analytics-Kanban, netops, Operations, Analytics

Thu, Nov 1

Ottomata moved T208332: Add EditAttemptStep properties to the schema whitelist from Incoming to Data Quality on the Analytics board.
Thu, Nov 1, 4:35 PM · Analytics, Patch-For-Review, Growth-Team, Product-Analytics
Ottomata updated subscribers of T208332: Add EditAttemptStep properties to the schema whitelist.

@Nuria says check out https://meta.wikimedia.org/wiki/Data_retention_guidelines#How_long_do_we_retain_non-public_data. I think there is some problem with saving both page_id with user_id or any other user identifier.

Thu, Nov 1, 4:34 PM · Analytics, Patch-For-Review, Growth-Team, Product-Analytics
Ottomata reassigned T202592: Instrument the landing page from Ottomata to Nuria.
Thu, Nov 1, 4:32 PM · Patch-For-Review, Analytics-Kanban, Analytics, New-Readers
Ottomata moved T202592: Instrument the landing page from Incoming to Smart Tools for Better Data on the Analytics board.
Thu, Nov 1, 4:31 PM · Patch-For-Review, Analytics-Kanban, Analytics, New-Readers
Ottomata claimed T202592: Instrument the landing page.
Thu, Nov 1, 4:31 PM · Patch-For-Review, Analytics-Kanban, Analytics, New-Readers
Ottomata moved T207760: setup/install weblog1001/WMF4750 as oxygen replacement from Incoming to Radar on the Analytics board.
Thu, Nov 1, 4:30 PM · ops-eqiad, Operations, Analytics
Ottomata moved T208251: T206785: Modern Event Platform: Stream Intake Service: AJV usage security review from Incoming to Modern Event Platform on the Analytics board.
Thu, Nov 1, 4:30 PM · Security-Reviews, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata moved T208282: Increase EventLogging limit from 2K to 5K from Incoming to Radar on the Analytics board.
Thu, Nov 1, 4:29 PM · Performance-Team (Radar), Traffic, Analytics-EventLogging, Analytics, Operations
Ottomata added a comment to T208282: Increase EventLogging limit from 2K to 5K.

If this is possible we don't see any reason why not! I don't think there is anything on the analytics backend side that would need to be changed; just the VSL settings and the EL extension. If traffic team is cool with it please proceed!

Thu, Nov 1, 4:29 PM · Performance-Team (Radar), Traffic, Analytics-EventLogging, Analytics, Operations
Ottomata added a project to T208490: Please copy the wmde* and wdcm directories to stat1007 /srv/published-datasets: Analytics-Kanban.
Thu, Nov 1, 4:24 PM · Analytics-Kanban, WMDE-Analytics-Engineering, User-GoranSMilovanovic, Analytics
Ottomata lowered the priority of T208490: Please copy the wmde* and wdcm directories to stat1007 /srv/published-datasets from High to Normal.
Thu, Nov 1, 4:24 PM · Analytics-Kanban, WMDE-Analytics-Engineering, User-GoranSMilovanovic, Analytics
Ottomata triaged T208247: Wikistats Bug - Monthly overview's "Top editors" box links to mainspace instead of userspace as High priority.
Thu, Nov 1, 4:22 PM · Analytics-Kanban, Analytics, Analytics-Wikistats
Ottomata assigned T208247: Wikistats Bug - Monthly overview's "Top editors" box links to mainspace instead of userspace to fdans.
Thu, Nov 1, 4:21 PM · Analytics-Kanban, Analytics, Analytics-Wikistats
Ottomata merged task T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors into T207424: Many errors on "MobileWikiAppiOSSearch" and "MobileWikiAppiOSUserHistory".
Thu, Nov 1, 4:10 PM · Analytics-EventLogging, Analytics
Ottomata merged T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors into T207424: Many errors on "MobileWikiAppiOSSearch" and "MobileWikiAppiOSUserHistory".
Thu, Nov 1, 4:10 PM · iOS-app-v6.2-Beluga-On-A-Pogo-Stick, iOS-app-Bugs, iOS-app-feature-Analytics, Analytics, Wikipedia-iOS-App-Backlog
Ottomata added a comment to T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors.

Oh this is a duplicate, you already know over in T207424: Many errors on "MobileWikiAppiOSSearch" and "MobileWikiAppiOSUserHistory" ok thanks!

Thu, Nov 1, 4:10 PM · Analytics-EventLogging, Analytics
Ottomata added a comment to T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors.

@chelsyx I just assigned to you because you are listed as a maintainer of this schema. Please route to whomever is appropriate! :)

Thu, Nov 1, 1:45 PM · Analytics-EventLogging, Analytics
Ottomata updated subscribers of T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors.
Thu, Nov 1, 1:45 PM · Analytics-EventLogging, Analytics
Ottomata created T208498: MobileWikiAppiOSSearch EventLogging events causing lots of validation errors.
Thu, Nov 1, 1:44 PM · Analytics-EventLogging, Analytics

Wed, Oct 31

Ottomata added a comment to T208359: Allow (almost?) all EventLogging events to go into MySQL in beta.

Done https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster#In_MySQL

Wed, Oct 31, 8:27 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata moved T208359: Allow (almost?) all EventLogging events to go into MySQL in beta from Ready to Deploy to Done on the Analytics-Kanban board.
Wed, Oct 31, 7:48 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata added a comment to T208359: Allow (almost?) all EventLogging events to go into MySQL in beta.

hey hey look at that! :)

Wed, Oct 31, 7:47 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Let's hear what @JAllemandou thinks. The mediawiki_history dataset is under 1TB (snappy compressed parquet) per snapshots, and we want to keep a few snapshots around. We also may need some space to 'stage' the dataset copy while we load it into HDFS (not sure about this yet). I think ~15TB to start with should be ok.

Wed, Oct 31, 6:41 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

If possible, I think I slightly prefer option 1. We may need more storage in the future, but I think for the time being it should be fine. @JAllemandou can correct me if I'm wrong, but we might not need more than 10TB or so to host a few mediawiki_history snapshots at once.

Wed, Oct 31, 6:24 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Oh, I think we won't necessarily need so much storage. CPU and RAM more important. Faster disks might actually better than larger ones in this case.

Wed, Oct 31, 5:09 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Are these numbers per, or total?

Per worker. This number is also flexible, its just what we were aiming for with our bare metal hardware order.

Wed, Oct 31, 4:45 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T208359: Allow (almost?) all EventLogging events to go into MySQL in beta.

Ok! I just disabled the whitelist filter in beta. Produce some events and they *should* show up in the MySQL database on deployment-eventlog05.

Wed, Oct 31, 4:39 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata moved T208359: Allow (almost?) all EventLogging events to go into MySQL in beta from Next Up to Ready to Deploy on the Analytics-Kanban board.
Wed, Oct 31, 4:24 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata claimed T208359: Allow (almost?) all EventLogging events to go into MySQL in beta.
Wed, Oct 31, 4:24 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging
Ottomata updated subscribers of T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Just had a great meeting with @chasemp, @faidon, @JAllemandou and @Nuria. The main action item (after Nuria had to go) was to talk with Cloud VPS engineers to see if we could make this cluster on Cloud Virts instead of bare metal in prod. That would be totally fine with us, and actually even preferred. I think we thought this was not possible originally, but if it is, and we can do it within a couple of weeks, we'd like to proceed that way.

Wed, Oct 31, 3:50 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata renamed T208361: Make schemas use required $schema property with absolute path (not absolute URL) to the schema from Make schemas use required $schema property with absolute path to the schema to Make schemas use required $schema property with absolute path (not absolute URL) to the schema.
Wed, Oct 31, 1:16 PM · Core Platform Team Backlog (Later), Services (later), Analytics-EventLogging, EventBus, Analytics-Kanban
Ottomata added a comment to T208361: Make schemas use required $schema property with absolute path (not absolute URL) to the schema.

We need to be able to identify the schema for an event outside of any particular transport or api call. The schema will be used to dynamically create downstream data structures, e.g. Hive table schemas, etc.

Wed, Oct 31, 1:14 PM · Core Platform Team Backlog (Later), Services (later), Analytics-EventLogging, EventBus, Analytics-Kanban

Tue, Oct 30

Ottomata created T208359: Allow (almost?) all EventLogging events to go into MySQL in beta.
Tue, Oct 30, 9:17 PM · Patch-For-Review, Analytics-Kanban, Analytics-EventLogging

Mon, Oct 29

Ottomata triaged T208251: T206785: Modern Event Platform: Stream Intake Service: AJV usage security review as Normal priority.
Mon, Oct 29, 6:00 PM · Security-Reviews, Security-Team, Core Platform Team Backlog (Watching / External), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata added a project to T208208: Allow hadoop prod jobs to preempt resource over default queue: Analytics-Kanban.
Mon, Oct 29, 3:55 PM · Analytics-Kanban, Patch-For-Review, Analytics
Ottomata moved T207768: Make sure webrequest_text preferred partition leadership is balanced from Paused to Done on the Analytics-Kanban board.
Mon, Oct 29, 3:21 PM · Analytics-Kanban, Analytics
Ottomata moved T207768: Make sure webrequest_text preferred partition leadership is balanced from Incoming to Operational Excellence on the Analytics board.
Mon, Oct 29, 3:21 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T207768: Make sure webrequest_text preferred partition leadership is balanced.

I updated wikitech docs: https://wikitech.wikimedia.org/w/index.php?title=Kafka%2FAdministration&action=historysubmit&type=revision&diff=1807237&oldid=1798149

Mon, Oct 29, 3:03 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T207768: Make sure webrequest_text preferred partition leadership is balanced.

Oo, here's a plausible explanation. kafka-jumbo1006 was the only broker that was missing some of its leaders. It is usually the last nod e to be rebooted for a full cluster reboot. webrequest_text takes the longest to sync back up after a broker restart. I betcha that all but these two partitions had resynced into the ISR, and the auto leader rebalancer was triggered to run (after 300 seconds) and saw that the imbalance percentage was greater than 10%. It then triggered a leader election BEFORE these two webrequest_text replicas were back in the ISR. Most of the leaders would have then been elected appropriately, but not these partitions. Soon after these replicas would have resynced, but at this point the imbalance partition is less than 10%, so any future auto rebalancer runs wouldn't trigger an election.

Mon, Oct 29, 2:59 PM · Analytics-Kanban, Analytics
Ottomata moved T207768: Make sure webrequest_text preferred partition leadership is balanced from Next Up to Paused on the Analytics-Kanban board.
Mon, Oct 29, 2:47 PM · Analytics-Kanban, Analytics
Ottomata added a project to T207768: Make sure webrequest_text preferred partition leadership is balanced: Analytics-Kanban.
Mon, Oct 29, 2:46 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T207768: Make sure webrequest_text preferred partition leadership is balanced.

Interesting! Today Luca and I were about to move partition leadership using kafka reassign-partitions, but we noticed that the replica assignment actually looked correct, and what we were going to change it to. It was only the leadership that was out of whack. So we ran a kafka preferred-replica-election to see if it would rebalance the leadership. And it did! The partition leadership now looks good.

Mon, Oct 29, 2:46 PM · Analytics-Kanban, Analytics

Fri, Oct 26

Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

Ping @Nuria too

Fri, Oct 26, 5:44 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a comment to T206824: Decide whether to use schema references in the schema registry.

I think we can close this. How we actually use them hasn't been decided though. We'll need to find a schema $ref resolver library, or write one. AJV won't do it.

Fri, Oct 26, 5:03 PM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T208028: Geoip data archive repository cause puppet to run for minutes.

Am fine with /srv/geoip!

Fri, Oct 26, 4:57 PM · Patch-For-Review, Analytics-Kanban, Analytics

Thu, Oct 25

Ottomata added a comment to T206824: Decide whether to use schema references in the schema registry.

Hm a tricky bit about $refs and generating fully dereferenced schemas with AJV:

Thu, Oct 25, 7:17 PM · Core Platform Team Backlog (Designing), Services (designing), Core Platform Team (Modern Event Platform (TEC2)), Analytics-EventLogging, EventBus, Analytics
Ottomata added a comment to T207194: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet.

FYI, networking considerations being worked out in T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster

Thu, Oct 25, 7:02 PM · ops-eqiad, Operations, User-Elukey, Analytics
Ottomata added a comment to T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.

hey hey heyyy, the nodes are in! https://phabricator.wikimedia.org/T204177#4695147

Thu, Oct 25, 5:10 PM · Analytics-Kanban, netops, Operations, Analytics
Ottomata added a project to T206279: Hive join fails when using a HiveServer2 client: Analytics-Kanban.
Thu, Oct 25, 4:29 PM · Analytics-Kanban, Analytics-Cluster, Analytics, Contributors-Analysis, Product-Analytics
Ottomata added a comment to T206279: Hive join fails when using a HiveServer2 client.

Reopening because we have an idea:

Thu, Oct 25, 4:28 PM · Analytics-Kanban, Analytics-Cluster, Analytics, Contributors-Analysis, Product-Analytics
Ottomata updated subscribers of T207951: Requesting access to deployment, operational logs, and analytics cluster for jlinehan.

@Nuria needs to give the sign off from analytics, but from my POV this is all correct! Yeehaw!

Thu, Oct 25, 3:02 PM · Patch-For-Review, Operations, SRE-Access-Requests
Ottomata added a comment to T206789: Modern Event Platform: Schema Registry: Implementation.

@Pchelolo https://snowplowanalytics.com/blog/2014/05/15/introducing-self-describing-jsons/ is a really good ready if you haven't seen it.

Thu, Oct 25, 2:08 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata updated the task description for T206789: Modern Event Platform: Schema Registry: Implementation.
Thu, Oct 25, 1:50 PM · Core Platform Team Backlog (Later), Core Platform Team (Modern Event Platform (TEC2)), Services (watching), Analytics-EventLogging, EventBus, Analytics
Ottomata merged task T207869: Modern Event Platform: Schema Registry Implementation into T206789: Modern Event Platform: Schema Registry: Implementation.
Thu, Oct 25, 1:48 PM · Services (watching), Analytics-EventLogging, EventBus, Analytics-Kanban