With the migration to liftwing these settings are no longer applicable. cc @calbon

Approved!

Yes, that's correct! Approve x 2

Approved

Approved

Approved

Yes, approved

Approved

The annotations have been added to the graphs:

• https://stats.wikimedia.org/#/en.wikipedia.org/reading/total-page-views/normal%7Cline%7C2018-04-24~2024-02-22%7C~total%7Cmonthly
• https://stats.wikimedia.org/#/en.wikipedia.org/reading/unique-devices/normal%7Cline%7C2018-04-24~2024-02-22%7C(access-site)~mobile-site*desktop-site%7Cmonthly

Approved

Approved

Approved

Approved.

Approved.

Approved

Approved

Approved

Thank you @elukey!

@Milimetric What was the root cause of this issue (the cause of missing datasets)?

Decommissioning EventLogging would be EPIC!

Would the header be translated into an x-analytics value?

A few questions:

• While we ought to consider an upgrade for all 4 clusters, from what I understand Jumbo can be upgraded independently. Are there any concerns with that approach?
• What are the upgrade considerations for Kafka clients?
• Specifically are there clients that publish to Kafka Jumbo directly or do all Kafka topics get mirrored from main (possibly logging?)?

Approved

Approved

Approved

This was delivered as part of the "documentathon": https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/DataHub/Data_Catalog_Documentation_Guide

Approved!

approved

Approved

Approved!

@MGerlach does the pre-fetch traffic have headers that can identify it as such as it comes through as webrequests?

Approved

It looks like the request is also in PyHive with the following PR still open: https://github.com/dropbox/PyHive/pull/328

Bug closed because too old, and not fixed: https://github.com/apache/superset/issues/3243

@JAllemandou is the limitation in data formatting coming from Presto or Superset (or both :) ?

@BTullis we'll need the SRE team's help with the deployment of the event platform schema ingestion into Datahub. The deployment involves a) creating the event steams custom platform and
b) deploying the ingestion code/transformer

The failure of this job requires a manual rerun, and based on a recent assessment this happens with some frequency (on average once daily). Let's bring this into current sprint and continue to troubleshoot.

I approve

Approving group membership

Here are some considerations that we discussed, that we need to further explore and decide on:

• Explore creating a custom platform for Event Streams
• Add top level event schema description as the dataset documentation. TBD on how to accomplish this given import options.
• The schema import automatically adds subgroups under kafka based on the first dot segment of the schema name. In the production instance of DataHub there are also streams with the naming analytics/mediawiki/web_ab_test_enrollment. Can “/” be used as a separator to designate the top level category?
• Can we import goblin lineage to propagate lineage from kafka > hive?
• There would value to import hive event_raw database for completion of lineage events
• Can we add a link to the event platform schema/datahub documentation to hive tables in event and event_sanitized? Lineage would be one way to trace this. Another would be to add links in the documentation to datasets with equivalent schema both upstream and downstream. This falls into the larger consideration on how to propagate metadata between equivalent datasets stored across different platforms and refinements.
• Some of the kafka topics are remnants of tests and misconfiguration/misnamings. There is an option to add them to an exclusion list. Ideally we'd delete these in Kafka, otherwise there is an exclusion list.
• Given that the prod datahub has the event streams current Kafka metadata can we delete and reimport all the Kafka metadata? If a fresh backup is not available it would be have one handy
• Is there a way to add ownership data to event schema json and import it from there? This would benefit Metrics Platform work and allow alerting the right parties about event publishing errors. Some discussion about adding this data already happened https://phabricator.wikimedia.org/T201063#4546544
• What is the best way to ingest the metadata? Datahub transformer vs airflow vs TBD?

@tchin as discussed today, that sounds like a good approach. Before deploying to production, let's wipe out the kafka metadata given that the original POC was imported under the kafka platform. I'll add these to the acceptance criteria.

The work related to this has been done as part of standing up the DSE K8s cluster. I will go ahead and close the ticket.

@BTullis These are good to be removed

Done. Are there any recovery keys to be had in case I am not able to access
my phone for whatever reason?

Approved.

@Htriedman we are picking this work up again. Is the POC that you did available in a repository on gitlab?

Thank you @jbond!

Approved

@Mayakp.wiki @nshahquinn-wmf Is this still an issue?

This dataset is no longer subscribed to. We should remove the database from the download list.

Approved

@BTullis do the permissions need to be removed before closing the task?

@Antoine does this still need to be implemented?

So gratifying to be able to be closing this task!