We now have spark 3.5 on the cluster. If you try with spark35-sql, it works.

@Ahoelzl how will this data be used? example queries? target dashboards / graphs?

@Addshore Within the data-platform infrastructure, we have a large Kafka setup and a way to generate events through an API called EventGate. Some of our biggest datasets transit through it before being automatically ingested into our data lake, and adding new streams alongside the existing ones is straightforward.

I think we can push more config setup at execution time. This would be a big win and not huge work.

Root cause was demanding xcoms network loads for airflow PG and location of PG instance on a machine with 1GB nic.
Now solved by pushing big xcoms on s3.

Root cause was demanding xcoms network loads for airflow PG and location of PG instance on a machine with 1GB nic.
Now solved by pushing big xcoms on s3.

Root cause:

We've done some research to size the GCS > HDFS download.

👍 Dataset updated. Data looks good. Checked with hdfs dfs -count ...

The legacy pipeline has been extracted, and removed from analytics-refinery.

A patch is in preparation to modify the root cause here: https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/profile/files/benthos/instances/webrequest_live.yaml

Summary of the work done here.

One step higher in the problem is here: https://gerrit.wikimedia.org/g/operations/puppet/+/f0d57f3f75c39d96f0ea1480beacb2918d9b5f0d/modules/profile/files/benthos/instances/webrequest_live.yaml#58
Where x-analytics with multiple similar values could agglomerate into arrays.

Data cleaned with:

python
import json

We could do the same for file exporter jobs following same pattern as Refine: T408802

We can try to implement the same pattern as for Refine fix proposed here: https://phabricator.wikimedia.org/T418151

In between, what we may do is a custom Datahub import job which would read from ESC and overload the table schemas before import. This would solve the Datahub part, not the spark console part (which we also aimed for usability).

We are currently running Spark 3.1.2, which has a limitation: all Hive-Parquet tables are V1 tables.

Deployment process of db_cleanup dag patch:

• Today:
  • merge the db_cleanup patch
  • pause the db_cleanup dags on all instances
• On 3/3 we unpause them 1 by 1 with SREs and we monitor.

search and analytics_test are running on small DB instances with no free headroom.
During deletes on large tables, Airflow’s current strategy stages deleted rows into a temporary table before removing them. That briefly requires ~2x table storage, which causes the database to run out of space and crash.

On this ticket, we have consulted both SREs and our team. We have agreed on the following details:

• extract a single repo refinery-python to Gitlab from analytics/refinery
• multiple CI output for this repo with multiple pipeline depending on the need:
  • docker image (seems like the best solution for Airflow triggering)
  • conda package
• and the repo could be pip compatible to be eventually required from conda-analytics or airflow-dags

Closing. Next optimization could be splitting from main instance file_exporters job. Should be done in another ticket.

Following discussion with @Ahoelzl we can postpone that on Q4.

I've marked all failed dag run as success to clear the UI.

K8s execution deployed, but we are not observing the overall performance gain we would have expected. We later tweak those 2 Airflow configs:

• worker_pods_creation_batch_size
• worker_pods_queued_check_interval

At least each task is consuming less k8s resources.

Build creation has been moved here: https://gitlab.wikimedia.org/repos/data-engineering/datahub-cli

• git history is preserved (the repo is actually a fork of analytics/refinery)
• CI is from workflow utils
• To setup the repo & CI, important steps:
  • remove lfs support: https://gitlab.wikimedia.org/repos/data-engineering/datahub-cli/edit#js-shared-permissions
  • create a token which will be used within the CI: https://gitlab.wikimedia.org/repos/data-engineering/datahub-cli/-/settings/access_tokens
  • add variables https://gitlab.wikimedia.org/repos/data-engineering/datahub-cli/-/settings/ci_cd#js-cicd-variables-settings including one containing the token CI_PROJECT_PASSWORD

With T415357 I’ve already started extracting the Python conda environment build for analytics/refinery into GitLab CI.

• Option 1:
  • downstreaming xcoms (keeping only the necessary fields)
  • computing parameters for each task in a pre_execution function (1 for each task)

First deploy crashed because and reverted.
I was blocked by missing connection from k8s to eventgates.
It was fixed by SREs: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1229524
Now testing on dev-env before retrying a deploy.

Duplicate of T411999

Deployment of db_cleaner dag on Airflow instances went mostly well.

Last notebook is here:
https://gitlab.wikimedia.org/hghani/movement-insights-requests/-/blob/main/SDS%201.3/client-side/simple-client_analysis_summary.ipynb?ref_type=heads
I reviewed it.

As we are discussing the limits of the current system putting strains on Airflow. The idea of this now old refactoring seems not a priority.