Decommissioning EventLogging would be EPIC!

Can the header be translated into an x-analytics value?

A few questions:

• While we ought to consider an upgrade for all 4 clusters, from what I understand Jumbo can be upgraded independently. Are there any concerns with that approach?
• What are the upgrade considerations for Kafka clients?
• Specifically are there clients that publish to Kafka Jumbo directly or do all Kafka topics get mirrored from main (possibly logging?)?

Approved

Approved

Approved

This was delivered as part of the "documentathon": https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/DataHub/Data_Catalog_Documentation_Guide

Approved!

approved

Approved

Approved!

@MGerlach does the pre-fetch traffic have headers that can identify it as such as it comes through as webrequests?

Approved

It looks like the request is also in PyHive with the following PR still open: https://github.com/dropbox/PyHive/pull/328

Bug closed because too old, and not fixed: https://github.com/apache/superset/issues/3243

@JAllemandou is the limitation in data formatting coming from Presto or Superset (or both :) ?

@BTullis we'll need the SRE team's help with the deployment of the event platform schema ingestion into Datahub. The deployment involves a) creating the event steams custom platform and
b) deploying the ingestion code/transformer

The failure of this job requires a manual rerun, and based on a recent assessment this happens with some frequency (on average once daily). Let's bring this into current sprint and continue to troubleshoot.

I approve

Approving group membership

Here are some considerations that we discussed, that we need to further explore and decide on:

• Explore creating a custom platform for Event Streams
• Add top level event schema description as the dataset documentation. TBD on how to accomplish this given import options.
• The schema import automatically adds subgroups under kafka based on the first dot segment of the schema name. In the production instance of DataHub there are also streams with the naming analytics/mediawiki/web_ab_test_enrollment. Can “/” be used as a separator to designate the top level category?
• Can we import goblin lineage to propagate lineage from kafka > hive?
• There would value to import hive event_raw database for completion of lineage events
• Can we add a link to the event platform schema/datahub documentation to hive tables in event and event_sanitized? Lineage would be one way to trace this. Another would be to add links in the documentation to datasets with equivalent schema both upstream and downstream. This falls into the larger consideration on how to propagate metadata between equivalent datasets stored across different platforms and refinements.
• Some of the kafka topics are remnants of tests and misconfiguration/misnamings. There is an option to add them to an exclusion list. Ideally we'd delete these in Kafka, otherwise there is an exclusion list.
• Given that the prod datahub has the event streams current Kafka metadata can we delete and reimport all the Kafka metadata? If a fresh backup is not available it would be have one handy
• Is there a way to add ownership data to event schema json and import it from there? This would benefit Metrics Platform work and allow alerting the right parties about event publishing errors. Some discussion about adding this data already happened https://phabricator.wikimedia.org/T201063#4546544
• What is the best way to ingest the metadata? Datahub transformer vs airflow vs TBD?

@tchin as discussed today, that sounds like a good approach. Before deploying to production, let's wipe out the kafka metadata given that the original POC was imported under the kafka platform. I'll add these to the acceptance criteria.

The work related to this has been done as part of standing up the DSE K8s cluster. I will go ahead and close the ticket.

@BTullis These are good to be removed

Done. Are there any recovery keys to be had in case I am not able to access
my phone for whatever reason?

Approved.

@Htriedman we are picking this work up again. Is the POC that you did available in a repository on gitlab?

Thank you @jbond!

Approved

@Mayakp.wiki @nshahquinn-wmf Is this still an issue?

This dataset is no longer subscribed to. We should remove the database from the download list.

Approved

@BTullis do the permissions need to be removed before closing the task?

@Antoine does this still need to be implemented?

So gratifying to be able to be closing this task!

@lbowmaker now that we are on the other side of the Oozie migration can we prioritize for the next sprint?

Yes, confirming the above. I approve the request.

It turns out that the sql_labs role was missing. Unfortunately the error message did not indicate a permissions issue.

@HShaikh Can you please post the SQL statement that you are running.

Consider implementing together with https://phabricator.wikimedia.org/T329978

Consider implementing together with https://phabricator.wikimedia.org/T329310

Related conversation: https://phabricator.wikimedia.org/T332420

Approved

@xcollazo @Antoine_Quhen Does this apply to the airflow ci/cd that DE manages? Are there any improvements that we wish to adopt or expand?

Another ping to @Maryana and @MMiller_WMF . Do you have opinions on asking the TikTok team vs parsing the user-agent string as is?

Approved

Is this the same issue reported in https://phabricator.wikimedia.org/T328127?

unique_devices_project_wide_daily and unique_devices_project_wide_monthly have no data and have been marked as deprecated. Ticket to delete: https://phabricator.wikimedia.org/T329978

Consider doing this at the same time as https://phabricator.wikimedia.org/T329978

Currently only unique_devices_per_domain_monthly has the dataset description.
For unique devices, let's document all of:

• unique_devices_per_domain_daily
• unique_devices_per_domain_monthly
• unique_devices_per_project_family_daily
• unique_devices_per_project_family_monthly
• unique_devices_project_wide_daily
• unique_devices_project_wide_monthly

• The [[ media requestvapi entry | https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,event.mediawiki_api_request,PROD)/Schema?is_lineage_mode=false ]] is lacking information for the normalized host. Not sure why that's the case given that the rest of the structs are filled in. The other aspects look good. Who would be the best person to fill it in? Who should be assigned as the data owner.
• Are there links to external documentation that can be added?
• The field is_wmf_domain talks about how it is derived but not what the field means. Is it a boolean that indicates if the request came from a wmf domain vs externally e.g. bot or toolforge tool?
• Who should be the owner of this dataset?