Page MenuHomePhabricator

Ottomata (Andrew Otto)
User

Projects (9)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Oct 9 2014, 4:50 PM (295 w, 1 d)
Availability
Available
IRC Nick
ottomata
LDAP User
Ottomata
MediaWiki User
Ottomata [ Global Accounts ]

Recent Activity

Today

Ottomata added a comment to T254567: MobileWikiAppProtectedEditAttempt: 'protectionStatus' is a required property.
{
  "_index": "logstash-2020.06.05",
  "_type": "eventlogging",
  "_id": "AXKEJZP3Cmr1R-gxg7rU",
  "_score": 1,
  "_source": {
    "schema": "EventError",
    "event_revision": 8682497,
    "level": "ERROR",
    "wiki": "",
    "type": "eventlogging",
    "message": "'protectionStatus' is a required property",
    "uuid": "0f5c4b1ca71c11ea90e6fa163eaa9b9d",
    "normalized_message": "'protectionStatus' is a required property",
    "revision": 14035058,
    "tags": [
      "eventlogging_EventError",
      "kafka",
      "input-kafka-eventlogging",
      "es",
      "normalized_message_untrimmed"
    ],
    "raw_event": "{\"dt\": \"2020-06-05T11:02:34Z\", \"event\": {}, \"ip\": \"XXXXXX\", \"recvFrom\": \"deployment-cache-text06.deployment-prep.eqiad.wmflabs\", \"revision\": 8682497, \"schema\": \"MobileWikiAppProtectedEditAttempt\", \"seqId\": 10249742, \"userAgent\": {\"browser_family\": \"Android\", \"browser_major\": \"7\", \"browser_minor\": \"0\", \"device_family\": \"Phone\", \"is_bot\": false, \"is_mediawiki\": false, \"os_family\": \"Android\", \"os_major\": \"7\", \"os_minor\": \"0\", \"wmf_app_version\": \"33.0.33-alpha-2019-11-18\"}, \"uuid\": \"0471b47b943a5d8293ebde5dd27999a2\", \"wiki\": \"idwiki\"}",
    "@timestamp": "2020-06-05T11:02:34.690Z",
    "@version": "1",
    "host": "deployment-eventlog05.deployment-prep.eqiad.wmflabs",
    "error_code": "validation",
    "event": {},
    "event_schema": "MobileWikiAppProtectedEditAttempt",
    "timestamp": 1591354955
  },
  "fields": {
    "@timestamp": [
      1591354954690
    ]
  }
}
Fri, Jun 5, 5:57 PM · Wikipedia-Android-App-Backlog, Product-Analytics, Analytics, Analytics-EventLogging, Beta-Cluster-Infrastructure
Ottomata added a comment to T254275: HTML Dumps - June/2020.

Ya.

Fri, Jun 5, 2:47 PM · Analytics, Core Platform Team, Dumps-Generation
Ottomata updated subscribers of T254275: HTML Dumps - June/2020.

Our approach:

Fri, Jun 5, 2:20 PM · Analytics, Core Platform Team, Dumps-Generation
Ottomata added a project to T254275: HTML Dumps - June/2020: Analytics.
Fri, Jun 5, 2:10 PM · Analytics, Core Platform Team, Dumps-Generation
Ottomata updated subscribers of T254567: MobileWikiAppProtectedEditAttempt: 'protectionStatus' is a required property.

@mpopov

Fri, Jun 5, 1:47 PM · Wikipedia-Android-App-Backlog, Product-Analytics, Analytics, Analytics-EventLogging, Beta-Cluster-Infrastructure

Yesterday

Ottomata reassigned T254035: Resetting Kerberos access for sguebo from Ottomata to elukey.

Actually, I was going to do this but I'm not exactly sure how! Assigning to luca

Thu, Jun 4, 5:48 PM · Analytics
Ottomata added a comment to T234629: Move the Analytics infrastructure to Debian Buster.

Ah yes! https://phabricator.wikimedia.org/T229347#5439259

Thu, Jun 4, 2:16 PM · Patch-For-Review, Analytics-Kanban, Analytics

Wed, Jun 3

Ottomata added a comment to T116719: EventGate validation errors should be visible in logstash.

Well, no that's not entirely true. Logstash is consuming from Kafka, so if there are too many messages in Kafka the logstash consumer will just lag. Its Kafka that we should worry about here, but we have that same worry for valid events too.

Wed, Jun 3, 2:56 PM · Patch-For-Review, Event-Platform, Analytics-Kanban, Analytics, observability, Analytics-EventLogging
Ottomata added a comment to T116719: EventGate validation errors should be visible in logstash.

Nope, right now it is just error logs that eventgate logs anyway, so this would also be try of ANY http error logs that are sent to logstash, including those from MW API.

Wed, Jun 3, 2:55 PM · Patch-For-Review, Event-Platform, Analytics-Kanban, Analytics, observability, Analytics-EventLogging
Ottomata added a comment to T253396: Upgrade all TLS enabled charts to v0.2 tls_helper.

Thanks @JMeybohm yeah T242861 is complicated!

Wed, Jun 3, 1:29 PM · Patch-For-Review, serviceops, Prod-Kubernetes, Kubernetes

Tue, Jun 2

Ottomata added a comment to T254058: [beta] EventLogging trying to fetch wrong Schema title.

Hm, I can't see anything in the code that would cause this to happen, and it only happened for about a 20 minute period on May 30th. Let's leave this open for a bit and see if it happens again.

Tue, Jun 2, 1:23 PM · Analytics-Kanban, Analytics-EventLogging, Beta-Cluster-Infrastructure, Wikimedia-production-error, Analytics

Mon, Jun 1

Ottomata added a comment to T254058: [beta] EventLogging trying to fetch wrong Schema title.

Ah! I see. This is eventlogging-processor trying to parse an event that it shouldn't.
https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2020.05.30/eventlogging?id=AXJkY2rsCmr1R-gxWx1Q&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-60d,mode:quick,to:now))

Mon, Jun 1, 9:00 PM · Analytics-Kanban, Analytics-EventLogging, Beta-Cluster-Infrastructure, Wikimedia-production-error, Analytics
Ottomata added a comment to T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform.

test.wikipeda.org is now successfully sending SearchSatisfaction events via EventGate. I'll check to see that Refine handles them properly, and then perhaps enable this to other wikis throughout this week.

Mon, Jun 1, 6:09 PM · MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T254058: [beta] EventLogging trying to fetch wrong Schema title.

Although, I'm not sure what would be trying to reach searchsatisfaction on meta. The logstash-beta link you gave doesn't bring up any results for me. @MarcoAurelio, can you paste the full raw log message here, or paste a link to the json message object?

Mon, Jun 1, 4:45 PM · Analytics-Kanban, Analytics-EventLogging, Beta-Cluster-Infrastructure, Wikimedia-production-error, Analytics
Ottomata claimed T254058: [beta] EventLogging trying to fetch wrong Schema title.
Mon, Jun 1, 4:43 PM · Analytics-Kanban, Analytics-EventLogging, Beta-Cluster-Infrastructure, Wikimedia-production-error, Analytics
Ottomata added a comment to T254058: [beta] EventLogging trying to fetch wrong Schema title.

This is due to a change in T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform. I will look into it!

Mon, Jun 1, 4:35 PM · Analytics-Kanban, Analytics-EventLogging, Beta-Cluster-Infrastructure, Wikimedia-production-error, Analytics
Ottomata added a comment to T213561: Discovery for Kafka cluster brokers.

FYI, we've recently added a 'general.yaml' values support to our helm charts repo. This allows us to render values from puppet. I'd like to accomplish the intent of this task by just rendering the list of Kafka brokers there. That will be good enough, I really just want to DRY up that list.

Mon, Jun 1, 2:35 PM · Operations, Services (watching), Event-Platform, Analytics

Fri, May 29

Ottomata added a comment to T251935: All EventGate instances should use EventStreamConfig.

Hm. I don't love the name config, which would mean: Stream Config config setting. Also, a nested config might be hard to work with. I've justt added a 'constraints' parameter to the EventStreamConfig API setting, which allow you specify in the GET URI params key=values that must match in the Stream Config. With a non-nested destination_event_service, we'll be able to do

Fri, May 29, 11:18 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform

Thu, May 28

Ottomata added a comment to T253661: Web publication doesn't work.

How often do we auto sync now?

Every 15 minutes. Your files have been synced, but I think that varnish must just be caching the old ones. Let's check tomorrow.

Thu, May 28, 8:16 PM · Analytics
Ottomata moved T253157: eventgate-wikimedia should expose runtime stream configuration from Next Up to Done on the Analytics-Kanban board.
Thu, May 28, 3:28 PM · Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata moved T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster from Next Up to Done on the Analytics-Kanban board.
Thu, May 28, 3:28 PM · Analytics-Kanban, Analytics, Wikidata-Query-Service, Wikidata
Ottomata updated subscribers of T253661: Web publication doesn't work.

The /srv/published sync thing is complicated in that it is multi-source, and if there are files with the same path names, which one gets synced is not defined (but I'd guess that the latest alphabetically sorted hostname would win). In this case, /srv/published/notebooks/weekly_edits/weekly_edits_summary.html exists on both stat1005 and stat1007, and it looks like the stat1007 version wins (owned there by @SNowick_WMF).

Thu, May 28, 3:26 PM · Analytics
Ottomata added a comment to T251935: All EventGate instances should use EventStreamConfig.

Hm, we should settle on a config key for identifying the eventgate instance. The obvious one to use would be 'event_service_name', although that doesn't quite clarify what is being specified. I had suggested 'destination_event_service' (or '..._name'). We also may one day need stream config to have an explicit event service URL, to allow for dynamic configuration of clients...although we don't need that now. If we just set a name (defined in ProductionServices.php), MW php clients can easily map that to the URL, but remote ones can't. Perhaps we will need different config keys for both of these?

Thu, May 28, 2:03 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform

Wed, May 27

Ottomata added a comment to T251935: All EventGate instances should use EventStreamConfig.

Oh ho, if I do implement this as an API param in EventStreamConfig, then we don't need any extra eventgate configs; we can just set stream_config_uri to one that includes the settings constraints, e.g.

Wed, May 27, 9:53 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T251935: All EventGate instances should use EventStreamConfig.

We'd still need to figure out how to make eventgate-wikimedia generate static stream config from EventStreamConfig API at runtime

AH! eventgate-wikimedia already supports this. All I have to do is

Wed, May 27, 7:23 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T251935: All EventGate instances should use EventStreamConfig.

Hm, an interesting extension of this idea would be to make the eventgate-wikimedia's schema_title checking more flexible. Instead of explicitly setting schema_title in a stream config entry, we'd set something like

schema_shape:
  title: mediawiki/revision/create
  // and any other extra properties we'd want to match here, e.g.
  required:
    - cool_required_field
Wed, May 27, 6:39 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata reopened T251935: All EventGate instances should use EventStreamConfig, a subtask of T229863: Refactor EventBus mediawiki configuration, as Open.
Wed, May 27, 6:26 PM · Patch-For-Review, MW-1.34-notes (1.34.0-wmf.20; 2019-08-27), Analytics, CPT Initiatives (Modern Event Platform (TEC2)), Analytics-EventLogging, Event-Platform
Ottomata reopened T251935: All EventGate instances should use EventStreamConfig, a subtask of T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events, as Open.
Wed, May 27, 6:26 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata reopened T251935: All EventGate instances should use EventStreamConfig as "Open".

Had a discussion with @mpopov about this in IRC today, and then discussed it more with the Analytics team. I'd like to reopen this and reconsider.

Wed, May 27, 6:26 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T240460: Clients need to generate an ISO 8601 formatted timestamp.

Ok let's do it.

Wed, May 27, 5:11 PM · Patch-For-Review, Better Use Of Data
Ottomata added a comment to T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster.

Did 31 days:

Wed, May 27, 1:42 PM · Analytics-Kanban, Analytics, Wikidata-Query-Service, Wikidata

Tue, May 26

Ottomata added a project to T253634: Convert WikimediaEvents to use ResourceLoader packageFiles: Product-Infrastructure-Team-Backlog.
Tue, May 26, 5:19 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata added a comment to T233329: Write and update Event Platform instrumentation documentation for Product teams.

Today I joined the Tech Documentation office hours. Notes:

Tue, May 26, 4:35 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata created T253649: Write blog post(s) about MEP.
Tue, May 26, 4:31 PM · Analytics, Event-Platform
Ottomata added a comment to T253634: Convert WikimediaEvents to use ResourceLoader packageFiles.

Oh, I guess the docs should be updated then?

Tue, May 26, 3:24 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

FYI T242454: Add examples to all event schemas must be done for this to work.

Tue, May 26, 3:05 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a parent task for T242454: Add examples to all event schemas: T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.
Tue, May 26, 3:05 PM · Analytics-Kanban, Services (watching), Analytics-EventLogging, Event-Platform, Analytics
Ottomata added a subtask for T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events: T242454: Add examples to all event schemas.
Tue, May 26, 3:05 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

Ah right so if we don't have a stream config entry, we won't POST a canary event anyway, OR try to ingest. So we don't really need to delete Kafka topics. We just can't use the latest event timestamp to determine if we should POST canary or ingest. We should just always ingest and monitor if a stream config entry exists, and require that inactive streams be removed from stream config. When they are, they will be automatically ignored for ingestion and monitoring.

Tue, May 26, 2:53 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

If it is, then consume the latest message from kafka for that stream. If the timestamp is not too old (newer than 90 days) then that stream should be both ingested and monitored.
If the stream should be monitored, get the stream's event schema's examples and POST them as a canary event to that eventgate-wikimiedia instance.

Tue, May 26, 2:51 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

I'm going to make T250844 a subtask of this and continue implementation and discussion there, since we will implement emitting canary events first.

Changed my mind, I merged T250844 in here and updated this task description.

Tue, May 26, 2:29 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata renamed T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events from Automate ingestion and refinement into Hive of event data from Kafka to Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.
Tue, May 26, 2:28 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata merged T250844: MEP: canary events so we know events are flowing through pipeline into T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.
Tue, May 26, 2:23 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata merged task T250844: MEP: canary events so we know events are flowing through pipeline into T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.
Tue, May 26, 2:23 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T250844: MEP: canary events so we know events are flowing through pipeline .

Since the implementation of this and T251609 are so similar, I'm going to merge this task into T251609 and redescribe that one to mention canary events.

Tue, May 26, 2:23 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T253634: Convert WikimediaEvents to use ResourceLoader packageFiles.

@SBisson, do you think you could finish this patch? I think you have a lot more context here as to how WikimediaEvents is used.

Tue, May 26, 2:16 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata added a comment to T253634: Convert WikimediaEvents to use ResourceLoader packageFiles.

@Krinkle WikimediaEvents uses skinScripts, but the packageFiles docs say this isn't supported (yet). How to resolve?

Tue, May 26, 2:14 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata added a comment to T253634: Convert WikimediaEvents to use ResourceLoader packageFiles.

This will also allow us to finish https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/576145, where we don't use the deprecated onResourceLoaderGetConfigVars

Tue, May 26, 2:14 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata created T253634: Convert WikimediaEvents to use ResourceLoader packageFiles.
Tue, May 26, 2:13 PM · Better Use Of Data, Performance-Team (Radar), Analytics, Event-Platform
Ottomata added a comment to T253392: Document in-schema who sets which fields.

Hm, I'm not sure in the schema is quite the right place to do this. The fact that EventGate does anything at all is just an implementation detail, but doesn't really have anything to do with the schema. Also it wouldn't be accurate :). meta.dt IS set by 'client side' EventBus code, and http fields are set by MediaWiki core itself when it logs api requests mediawiki/api/request . eventgate-wikimedia is only setting meta.dt and http.request_headers['user-agent'] if they aren't provided by the client.

Tue, May 26, 1:22 PM · Better Use Of Data, Analytics, Event-Platform

Wed, May 20

Ottomata added a comment to T240460: Clients need to generate an ISO 8601 formatted timestamp.

@mpopov @jlinehan while working on some docs, I submitted https://gerrit.wikimedia.org/r/597617 so that my docs wouldn't be totally incorrect :p. Let me know what you think.

Wed, May 20, 8:47 PM · Patch-For-Review, Better Use Of Data
Ottomata added a comment to T240460: Clients need to generate an ISO 8601 formatted timestamp.

Although, on second thought, maybe client_dt is better here after all? Since this will be used only by analytics / instrumentation schemas, perhaps client makes sense? meta.dt is the default event time and will always be used for kafka timestamps and hive partitioning. Maybe client_dt is actually more descriptive for what it is being suggested for here.

Wed, May 20, 8:46 PM · Patch-For-Review, Better Use Of Data
Ottomata renamed T233329: Write and update Event Platform instrumentation documentation for Product teams from Review Event Platform instrumentation documentation for Product teams to Write and update Event Platform instrumentation documentation for Product teams.
Wed, May 20, 8:29 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata moved T233329: Write and update Event Platform instrumentation documentation for Product teams from In Progress to In Code Review on the Analytics-Kanban board.
Wed, May 20, 8:29 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata moved T233329: Write and update Event Platform instrumentation documentation for Product teams from Next Up to In Progress on the Analytics-Kanban board.
Wed, May 20, 8:28 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata added a project to T233329: Write and update Event Platform instrumentation documentation for Product teams: Analytics-Kanban.
Wed, May 20, 8:28 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata renamed T233329: Write and update Event Platform instrumentation documentation for Product teams from Review draft Modern Event Platform schema guidelines to Review Event Platform instrumentation documentation for Product teams.
Wed, May 20, 8:26 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata updated subscribers of T233329: Write and update Event Platform instrumentation documentation for Product teams.

Hm, that is a good idea. I think perhaps a 'analytics event platform tutorial' page will be better for this. How's that sound?

Ok @nshahquinn-wmf, I have done what I said I would do 8 months ago!

Wed, May 20, 8:25 PM · Analytics-Kanban, Analytics, Event-Platform, Product-Analytics
Ottomata updated the task description for T253239: Make it easy to debug eventlogging instrumentation, add ability to send client canary events..
Wed, May 20, 5:14 PM · Analytics
Ottomata added a comment to T250844: MEP: canary events so we know events are flowing through pipeline .

In https://phabricator.wikimedia.org/T251609#6152803 we figured out a mostly clean way to implement this. Am on it!

Wed, May 20, 4:12 PM · Analytics-Kanban, Analytics
Ottomata claimed T250844: MEP: canary events so we know events are flowing through pipeline .
Wed, May 20, 4:11 PM · Analytics-Kanban, Analytics
Ottomata added a subtask for T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events: T250844: MEP: canary events so we know events are flowing through pipeline .
Wed, May 20, 4:10 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a parent task for T250844: MEP: canary events so we know events are flowing through pipeline : T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.
Wed, May 20, 4:10 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

Had a great discussion with A-team today and got some great ideas.

Wed, May 20, 4:10 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T252203: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006.

OH RIGHT! Great. Will check that out and try to backfill it.

Wed, May 20, 3:12 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

Hue allows you to access Hive and files in HDFS, for which you need a shell account and membership in the analytics-privatedata-users group.

Wed, May 20, 1:24 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Ottomata added a comment to T250912: EventStreams socket stays connected without any traffic incoming.

Oops I did mean what you said. THANK YOU

Wed, May 20, 1:22 PM · Analytics, EventStreams

Tue, May 19

Ottomata added a comment to T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.

Ok, new idea. If we do T253157: eventgate-wikimedia should expose runtime stream configuration, I think we can use that to solve both this and T250844: MEP: canary events so we know events are flowing through pipeline .

Tue, May 19, 9:45 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata created T253157: eventgate-wikimedia should expose runtime stream configuration.
Tue, May 19, 8:06 PM · Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a comment to T250844: MEP: canary events so we know events are flowing through pipeline .

Could we re-use some of the EventGate kubernetes readinessProbe logic for this?

Tue, May 19, 6:02 PM · Analytics-Kanban, Analytics
Ottomata created T253121: EventLogging Server Side client should POST to EventGate.
Tue, May 19, 2:00 PM · Event-Platform, Analytics, Product-Infrastructure-Team-Backlog (Kanban), Better Use Of Data
Ottomata added a comment to T250844: MEP: canary events so we know events are flowing through pipeline .

We could, but how to know if that hour is absent because of a lack of data,
or due to loss?

Tue, May 19, 12:56 PM · Analytics-Kanban, Analytics

Mon, May 18

Ottomata added a comment to T240460: Clients need to generate an ISO 8601 formatted timestamp.

I agree though, having meta.dt sometimes be event time and sometimes the 'eventgate receive time' might get a bit confusing, but I don't think we can change meta.dt now. It is already being used as the event time for all of the EventBus sent mediawiki events. Also, if we can, we do want to use the event time as the Kafka timestamp and Hive partition timestamp, its just that in some cases where we don't trust the producers we can't be sure they will be setting event times that make sense.

Mon, May 18, 9:21 PM · Patch-For-Review, Better Use Of Data
Ottomata added a comment to T240460: Clients need to generate an ISO 8601 formatted timestamp.

@mforns I suggested event_dt in https://phabricator.wikimedia.org/T240460#6125463 mainly because that is what this is referred to in stream processing worlds. Also, client_dt might not make sense in other situations, like a syslog event or something. This field will be used to specify the time the event happened.

Mon, May 18, 9:18 PM · Patch-For-Review, Better Use Of Data
Ottomata created T253069: Set up an instance of EventStreams in beta that will allow for consuming any stream.
Mon, May 18, 9:03 PM · Patch-For-Review, Analytics-Kanban, MW-1.35-notes (1.35.0-wmf.27; 2020-04-07), Event-Platform, Analytics, Product-Infrastructure-Team-Backlog (Kanban), Better Use Of Data
Ottomata updated subscribers of T252203: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006.

In https://phabricator.wikimedia.org/T249261#6146372 @EBernhardson reported that he was missing event.mediawiki_cirrussearch_request/datacenter=eqiad/year=2020/month=5/day=8/hour=10. I just looked for this data in Camus raw, and indeed hour 10 is missing in raw data for that day. Once again, we've lost some data. I see the same hour missing for mediawiki_api_request too.

Mon, May 18, 8:54 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform.

Hm, not related to this ticket, but possibly to T252203. Looking

Mon, May 18, 8:41 PM · MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata created T253058: DRY kafka broker declaration into helmfiles from puppet .
Mon, May 18, 7:20 PM · Event-Platform, Analytics
Ottomata added a comment to T250912: EventStreams socket stays connected without any traffic incoming.

Hm, this is going to be hard to reproduce, especially given T179986. I'd expect this to be a client side problem, but it could be something on EventStreams side. Can we wait until T179986 is resolved before investigating further? Perhaps if that is fixed your problem will just disappear! :)

Mon, May 18, 5:25 PM · Analytics, EventStreams
Ottomata added a comment to T242767: EventStreams drops the connection after 15 minutes, which makes it unreliable.

Heh, just tried myself, I guess not; still got disconnected after 15 minutes.

Mon, May 18, 5:23 PM · Patch-For-Review, Traffic, Operations, Analytics-Kanban, Analytics, EventStreams
Ottomata added a comment to T250282: Build sidebar instrumentation.

Ok, makes sense! Thanks for considering! I hope to get all the EL docs updated very soon.

Mon, May 18, 5:20 PM · MediaWiki-extensions-WikimediaEvents, Readers-Web-Backlog (Kanbanana-2019-20-Q4), Desktop Improvements
Ottomata added a comment to T242767: EventStreams drops the connection after 15 minutes, which makes it unreliable.

Hio, I see some upgrades happened, should they have fixed this?

Mon, May 18, 4:42 PM · Patch-For-Review, Traffic, Operations, Analytics-Kanban, Analytics, EventStreams

Thu, May 14

Ottomata added a comment to T249752: Decomission notebook hosts .

Can you log out and log back in?

Thu, May 14, 9:09 PM · Analytics
Ottomata added a comment to T249752: Decomission notebook hosts .

OK done. FYI, all you have to do to reset your venv is to delete your ~/venv directory (or move it out of the way). SWAP will recreate the venv from scratch if it doesn't exist if you stop your jupyter server and then log out and log into jupyterhub.

Thu, May 14, 8:36 PM · Analytics
Ottomata updated the task description for T249752: Decomission notebook hosts .
Thu, May 14, 7:21 PM · Analytics
Ottomata added a comment to T249752: Decomission notebook hosts .

Yup, I think that will break stuff! Let me reset your venv on stat1005. I just stopped your Notebook Server too. Try to log back in and we'll see if you get a clean venv.

Thu, May 14, 7:15 PM · Analytics
Ottomata moved T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc. from Next Up to In Progress on the Analytics-Kanban board.
Thu, May 14, 7:10 PM · Analytics-Kanban, Event-Platform, Analytics
Ottomata added a comment to T249752: Decomission notebook hosts .

@MMiller_WMF pip install pandasql?

Thu, May 14, 7:08 PM · Analytics
Ottomata added a comment to T250250: Add TLS to Kafka Mirror Maker .

Ah! Since profile::kafka::mirror ssl.keystore.location, it will attempt to authenticate with Kafka. User:ANONYMOUS is allowed to do anything, but if a client authenticates and their principal does not have an ACL, it will fail. I just ran

Thu, May 14, 6:41 PM · Analytics-Kanban, Analytics
Ottomata added a comment to T248615: Capture and send back client-side errors.

Thank you!

Thu, May 14, 5:39 PM · Patch-For-Review, KaiOS-Wikipedia-app (MVP), Inuka-Team (Kanban), Event-Platform, Analytics
Ottomata added a comment to T248615: Capture and send back client-side errors.

@SBison can you add your code to the list of clients here? https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#eventgate-logging-external

Thu, May 14, 3:50 PM · Patch-For-Review, KaiOS-Wikipedia-app (MVP), Inuka-Team (Kanban), Event-Platform, Analytics
Ottomata added a comment to T252703: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue).

For Hue the user must also have shell access and be added to the analytics-privatedata-users group.

Thu, May 14, 2:58 PM · Analytics-Kanban, Analytics, LDAP-Access-Requests, Operations
Ottomata added a project to T252675: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster: Analytics-Kanban.
Thu, May 14, 2:35 PM · Analytics-Kanban, Analytics
Ottomata moved T116719: EventGate validation errors should be visible in logstash from In Progress to Done on the Event-Platform board.
Thu, May 14, 2:28 PM · Patch-For-Review, Event-Platform, Analytics-Kanban, Analytics, observability, Analytics-EventLogging
Ottomata moved T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events from In Progress to Next Up on the Event-Platform board.
Thu, May 14, 2:28 PM · MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), Patch-For-Review, Analytics-Kanban, Analytics, Analytics-EventLogging, Event-Platform
Ottomata added a subtask for T214430: Modern Event Platform: Stream Connectors: T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc..
Thu, May 14, 2:27 PM · Analytics, Goal, Services (watching), Analytics-EventLogging, Event-Platform
Ottomata added a parent task for T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc.: T214430: Modern Event Platform: Stream Connectors.
Thu, May 14, 2:27 PM · Analytics-Kanban, Event-Platform, Analytics
Ottomata merged T223626: Kafka Connect development work into T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc..
Thu, May 14, 2:26 PM · Analytics-Kanban, Event-Platform, Analytics
Ottomata merged task T223626: Kafka Connect development work into T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc..
Thu, May 14, 2:26 PM · Analytics, Event-Platform
Ottomata added a comment to T223626: Kafka Connect development work.

Merging this into the ingestion framework evaluation task.

Thu, May 14, 2:26 PM · Analytics, Event-Platform