On T386114 we fixed lock issues related to generating metrics.
However, after fixing those, two separate issues emerged:
and also:
The OOM issue has a simple fix, to be delivered as part of this ticket. However, we need to root cause and fix the silent failures.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| analytics: bump driver mem for mw_content_reconcile_mw_content_history_daily-compute_metrics. | repos/data-engineering/airflow-dags!1206 | xcollazo | bump-metrics-driver-mem | main | |
| analytics: Bump driver_memory to avoid OOM for mw content daily metrics. | repos/data-engineering/airflow-dags!1089 | xcollazo | fix-oom-on-metrics-v2 | main | |
| analytics: Fix OOM on compute_metrics from mw_content_reconcile_mw_content_history_daily. | repos/data-engineering/airflow-dags!1083 | xcollazo | fix-oom-on-metrics | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T366752 Dumps 2.0 Phase III: Production level dumps (SDS 1.2) | |||
| Resolved | xcollazo | T387021 Support downstream users in adopting mediawiki_content_history_v1 | |||
| Resolved | xcollazo | T387033 Figure root cause of silent failures when computing metrics for mediawiki_content_history_v1 |
xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1083
analytics: Fix OOM on compute_metrics from mw_content_reconcile_mw_content_history_daily.
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1083
analytics: Fix OOM on compute_metrics from mw_content_reconcile_mw_content_history_daily.
Mentioned in SAL (#wikimedia-operations) [2025-02-21T18:33:30Z] <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@60223e2]: Deploying latest DAGs for the analytics Airflow instance. T387033.
Mentioned in SAL (#wikimedia-operations) [2025-02-21T18:34:15Z] <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@60223e2]: Deploying latest DAGs for the analytics Airflow instance. T387033. (duration: 00m 45s)
Mentioned in SAL (#wikimedia-analytics) [2025-02-21T18:34:28Z] <xcollazo> Deployed latest DAGs for the analytics Airflow instance. T387033.
After further inspection, the error appears to be a skein application OOM:
sudo -u analytics yarn logs -appOwner analytics -applicationId application_1734703658237_1636365 | tail -n 100 ... 25/02/23 04:08:01 INFO skein.ApplicationMaster: Shutting down: Application driver failed with exit code 143. This is often due to the application master memory limit being exceeded. See the diagnostics for more information. 25/02/23 04:08:01 INFO skein.ApplicationMaster: Unregistering application with status FAILED 25/02/23 04:08:01 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 25/02/23 04:08:01 INFO skein.ApplicationMaster: Deleted application directory hdfs://analytics-hadoop/user/analytics/.skein/application_1734703658237_1636365 25/02/23 04:08:01 INFO skein.ApplicationMaster: WebUI server shut down 25/02/23 04:08:01 INFO skein.ApplicationMaster: gRPC server shut down ...
For some reasons the skein app is logging this as INFO. Add it to the reasons I love skein....
I have bumped metrics_driver_memory from 8GB to 16GB and this solves the issue. Will now make this the default via a MR.
xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1089
analytics: Bump driver_memory to avoid OOM for mw content daily metrics.
dr0ptp4kt merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1089
analytics: Bump driver_memory to avoid OOM for mw content daily metrics.
Reopening as we had another instance of skein OOM:
xcollazo@an-launcher1002:~$ sudo -u analytics yarn logs -appOwner analytics -applicationId application_1741864027385_383042 | grep "Application driver failed" -B 2 -A 2 ... 25/03/31 13:44:16 INFO skein.ApplicationMaster: Registering application with resource manager 25/03/31 13:44:16 INFO skein.ApplicationMaster: Starting application driver 25/03/31 18:00:10 INFO skein.ApplicationMaster: Shutting down: Application driver failed with exit code 143. This is often due to the application master memory limit being exceeded. See the diagnostics for more information. 25/03/31 18:00:10 INFO skein.ApplicationMaster: Unregistering application with status FAILED 25/03/31 18:00:10 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
Verified that driver has 16GB as per Airflow task details.
Bumping to 24GB manually to rerun, although 24GB for a driver for metrics seems silly. I wonder what dequee is doing in there...
xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1206
Draft: Bump metrics driver mem
xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1206
analytics: bump driver mem for mw_content_reconcile_mw_content_history_daily-compute_metrics.
I wonder if this line is a factor. withColumn copies the entire df so we're getting 3x the dataframes here.
I wonder if this line is a factor. withColumn copies the entire df so we're getting 3x the dataframes here.
Hmm.. a heavy df indeed but I wouldn't expect that to affect the driver. We can def try to make the whole job faster with the below:
# Analyzers can only be run on columns, so flatten revision_content_slots to support T382953
flattened_df = (df
.withColumn("revision_content_slots.main.content_body",
df["revision_content_slots"]["main"]["content_body"])
.withColumn("revision_content_slots.main.content_format",
df["revision_content_slots"]["main"]["content_format"])
.drop("revision_content_slots") <<<<<<<<<<<<<<<
)