User Details
- User Since
- Mar 30 2026, 1:14 PM (28 w, 5 d)
- Availability
- Available
- IRC Nick
- atsukoito
- LDAP User
- Atsuko
- MediaWiki User
- Atsuko-WMF [ Global Accounts ]
Fri, Oct 16
DO NOT UNPAUSE or TEST wdqs_wikidata_data_reload
We've upgraded one more instance, airflow-sre T435485, 12 to go.
We also found and fixed a regression in a SparkSqlOperator when used with Skein. It was impossible to test it in airflow-devenv as we use Kerberos there differently, and we ddin't have a test case in test_k8s instance, either.
webrequest_blocked_requests_requestctl DAG has finished without errors after fix and restart.
Thu, Oct 15
Now that we know that
- most of the code passing the tests correctly as CI is unblocked,
- Pools are upgraded,
- analytics-test runs on its own isolated Hadoop instance (has its own HDFS / Yarn / etc.) so anything you do there will not affect anything in production
Looks like it's still not transferring. after 10 minutes it requeues twice and then fails. lmk what other info would be helpful to provide. if you're able to- you're welcome to switch to my user for testing and getting past the hdfs permissions
To Do: check the results of webrequest_blocked_requests_requestctl DAG on 16.10.
Upgrade plan
Need to fix dag_run.start_date in test-time
Confirming that it works just fine my auth_manager code in Airflow.
Wed, Oct 14
@trueg @lerickson I'm planning to upgrade airfow-wikidata on Airlfow 3 soon. For the upgrade, we are generally want to test that DAGs are operating normally in airflow-devenv before actually upgrading the instance. Can you please advise me if there is the way to test wdqs_external_queries_rollup_daily and wdqs_wikidata_data_reload DAGs without modifying production data?
Tue, Oct 13
@AndrewTavis_WMDE Hi! We completed most of prerequisites to run Airflow 3 in production, and we are upgrading smaller instances now. We learned that WMDE instance is interdependent with other large instances (see previous message, airflow-wmde checks for the iceberg table porducers DAGs in airflow-main and airflow-search). With this, we are moving the upgrades of those instances to a later stage.
Sure, let's wait!
Fri, Oct 9
I'm looking thru the codebase now, and so far I can say that we have next dependencies between instances, meaning that main, platform_eng, search, and wmde instances will be upgraded in one go.
search -> platform_eng [via=search/dags/transfer_to_es.py, type=RestExternalTaskSensor] wmde -> search [via=wmde/dags/wd_query_segments/wd_query_segments_daily_dag.py, type=RestExternalTaskSensor] platform_eng -> main [to=iceberg_wmf,via=config/datasets.yaml, type=RestExternalTaskSensor] wmde -> main [to=iceberg_wmf,via=config/datasets.yaml, type=RestExternalTaskSensor]
Will need to double-check it with API/database lookup.
@elukey we are fairly confident that Airflow 3 works well. Considering that airflow-sre instance is quite small and independent, I'd like to upgrade it in a second wave. Gonna ping you on Monday about it.
airflow-fr-tech seems to have no dependencies on/to other instances, and only has 3 DAGs. Considering it as a next batch of upgrade.
Most of the DAGs are failed to import
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/opt/airflow/dags/airflow_dags/dumps/dags/mediawiki_pagetitles_dump.py", line 18, in <module>
from airflow.datasets import Dataset
ModuleNotFoundError: No module named 'airflow.datasets'Thu, Oct 8
Fixed high-load on dag-processor, now all DAGs are loaded in a reasonable time (under 5 minutes, need to tune CPU allocation vs number of threads a bit more).
Fixing the OOM on DB cleaner DAGs:
[Thu Oct 8 09:45:09 2026] memory: usage 524288kB, limit 524288kB, failcnt 131
Tue, Oct 6
We have two tests that verify SLAs, I'll start from those
tests/main/clickstream/clickstream_monthly_dag_test.py tests/main/wmf_contributors/test_editor_counts_per_page_daily_dag.py
The next popular use-case is running the DAG in test-time, and inspecting the XCOM. This requires reimplementing a big part of DAG.test()
tests/analytics_test/refine/refine_to_hive_hourly_dag_analytics_test_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/research/content_diff_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/research/revert_risk_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/research/top_pages_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/analytics_product/automoderator/test_automoderator_daily_snapshot_dag.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/canary_events/canary_events_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/google_search_console/corrections_test.py: pytest.skip("create_dagrun")
tests/main/google_search_console/corrections_test.py: pytest.skip("create_dagrun")
tests/main/google_search_console/corrections_test.py: pytest.skip("create_dagrun")
tests/main/google_search_console/google_search_console_to_hdfs_daily_dag_test.py: pytest.skip("create_dagrun")
tests/main/mediawiki/content/export/mw_content_xml_export_dags_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/mediawiki/content/export/mw_content_xml_export_dags_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/mediawiki/content/export/mw_content_xml_export_dags_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/mediawiki/content/history/mw_content_reconcile_mw_content_history_daily_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/mediawiki/content/history/mw_content_reconcile_mw_content_history_monthly_dag_test.py: pytest.skip("create_dagrun")
tests/main/refine/refine_to_hive_hourly_dag_test.py: pytest.skip("create_dagrun_xcom_pull")
tests/main/refine/refine_to_hive_hourly_dag_test.py: pytest.skip("create_dagrun")
tests/main/refine/refine_to_hive_hourly_dag_test.py: pytest.skip("create_dagrun_xcom_push")
tests/main/spur/spur_anonymous_residential_daily_dag_test.py: pytest.skip("create_dagrun")Mon, Oct 5
Almost all tests are back 56 failed, 2909 passed, 28 skipped, I'll check if it easier to fix it or skip it and create upgrade task.
The main problem with conftest is that the some of the Dag methods are now only available in SerialisedDag. Fortunately, the built-in test method (dag.test as described in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#concepts-debugging) is able to serialise it, so we might want to copy the behaviour.
Fri, Oct 2
Starting with
make test.interactive args=tests/test_on_disk_fixtures.py::test_kubernetes_pod_operator_against_fixtures
Thu, Oct 1
The library I'm using in Airflow 3 should work with RSA, so it should be fine.
Moving this two work items from the big research task. I need help fixing database access in these 2 files, or at least guidance on how to test related DAGs
Wed, Sep 30
Fix SMTP config
[2026-09-30T00:51:50.793808Z] INFO - ::group::Post Execute
[2026-09-30T00:51:50.942790Z] ERROR - Failed to send notification (sync): SMTP connection is not found.
[2026-09-30T00:51:50.942935Z] ERROR - Failed to send email notification via SmtpNotifierAirflowException: SMTP connection is not found.
File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/task_runner.py, line 2154 in _send_error_email_notification
File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/bases/notifier.py, line 138 in __call__
File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/notifications/smtp.py, line 155 in notify
File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 79 in __enter__
File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 119 in get_conn
,AirflowNotFoundException: The conn_id `smtp_default` isn't defined
File /home/app/.local/lib/python3.11/site-packages/airflow/providers/smtp/hooks/smtp.py, line 117 in get_conn
File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/bases/hook.py, line 61 in get_connection
File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/definitions/connection.py, line 296 in get
File /home/app/.local/lib/python3.11/site-packages/airflow/sdk/execution_time/context.py, line 231 in _get_connectionturns out it was 1) stale logs and 2) some unmerged changes to main_airflow_3.3. merging
we have one IDP in each core site and they share the sessions via Redis. and the currently active one is targeted via the idp.w.o CNAME
Tue, Sep 29
After applying scheduler.create_cron_data_intervals: true patch:
Run ID scheduled__2026-09-29T18:00:00+00:00 Data Interval Start 2026-09-29 17:00:00 Data Interval End 2026-09-29 18:00:00 [2026-09-29 18:00:26] INFO - Poking for wmf_raw.webrequest/webrequest_source=test_text/year=2026/month=9/day=29/hour=17
Run ID is now equals Data Interval End as opposed to Data Interval Start on Airflow 2, however DAGs are getting correct start time. Note that the wait_for_wmf_raw_webrequest_partitions skipped a run.
Out of scope for now
airflow-test-k8s.wikimedia.org done T433383, 13 to go.