Page MenuHomePhabricator

atsuko (Atsuko Ito)
SRE

Projects (5)

Today

  • No visible events.

Tomorrow

  • No visible events.

Wednesday

  • No visible events.

User Details

User Since
Mar 30 2026, 1:14 PM (31 w, 6 d)
Availability
Available
IRC Nick
atsukoito
LDAP User
Atsuko
MediaWiki User
Atsuko-WMF [ Global Accounts ]

Recent Activity

Fri, Nov 6

atsuko changed the status of T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline from Open to In Progress.
Fri, Nov 6, 5:37 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko moved T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline from Backlog - project to In Progress on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Fri, Nov 6, 5:37 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko claimed T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline.
Fri, Nov 6, 5:37 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a comment to T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline.

This happens because we have a resource with non-templated name (in this case, PersistentVolumeClaim/dev-metrics-git-pipeline), defined in helmfile.d/dse-k8s-services/airflow-main/values-production.yaml,

kubernetes_pod_operator:
  persistent_volume_claims:
    # CephFS disk for the dev-metrics git commit pipeline.
    # Treated as a rebuildable cache of the collected repo clones.
    # Mounted by the DAG.
    - claim_name: dev-metrics-git-pipeline
      size: 20Gi
      storage_class: ceph-cephfs-ssd
Fri, Nov 6, 5:34 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko moved T443947: Investigate why a Spark YARN app continued after Airflow ended its try from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Fri, Nov 6, 5:04 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Data-Engineering
atsuko added a project to T443947: Investigate why a Spark YARN app continued after Airflow ended its try: Data-Platform-SRE (2026-10-30 - 2026-11-20).

There was rolling restart of K8s nodes around the time operator pod has been reported missing,

2026-11-04, 16:52:46 UTC	1	heartbeat timeout		Task did not emit heartbeat within time limit (300 seconds) and will be terminated. See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks
Fri, Nov 6, 4:51 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Data-Engineering
atsuko added a comment to T423248: Upgrade to Airflow 3.3.x.

We have upgraded 6 instances and was ready to upgrade 7th (ml), but we are now held back with the investigation of performance problems T443252: airflow-dumps webserver performance, as well, and, also, whole SRE DPE are on SRE summit next week. We'll re-group and announce our next course of actions next week.

Fri, Nov 6, 3:52 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko updated the task description for T443252: airflow-dumps webserver performance.
Fri, Nov 6, 3:47 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work

Thu, Nov 5

atsuko added a comment to T435481: Upgrade airflow-ml to 3.3.x.

We hit the performance issues with airflow-dumps, I think we should take another look on the DAG structure for ML team to check if it is similar to the ones causing performance issues.

Thu, Nov 5, 1:43 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20), Machine-Learning-Team (Q2 FY2026-27), Essential-Work

Wed, Nov 4

atsuko changed the status of T443252: airflow-dumps webserver performance from In Progress to Stalled.

dumps UI operation is recovered

Wed, Nov 4, 11:15 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko changed the status of T443252: airflow-dumps webserver performance, a subtask of T435478: Upgrade airflow-dumps to 3.3.x, from In Progress to Stalled.
Wed, Nov 4, 11:15 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work
atsuko added a comment to T443252: airflow-dumps webserver performance.
Wed, Nov 4, 6:43 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko updated the task description for T443210: research the option to migrate airflow-devenv to a webservice.
Wed, Nov 4, 5:04 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko edited projects for T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline, added: Data-Platform-SRE (2026-10-30 - 2026-11-20); removed Data-Platform-SRE.
Wed, Nov 4, 4:58 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko created T443659: airflow-devenv: clash between devs on pvc/dev-metrics-git-pipeline.
Wed, Nov 4, 4:56 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko closed T442431: Upgrade airflow-experiment-platform to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, as Resolved.
Wed, Nov 4, 3:34 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko closed T442431: Upgrade airflow-experiment-platform to 3.3.x as Resolved.
Wed, Nov 4, 3:34 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Epic
atsuko moved T443252: airflow-dumps webserver performance from Backlog - project to In Progress on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Wed, Nov 4, 3:01 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko changed the status of T443252: airflow-dumps webserver performance, a subtask of T435478: Upgrade airflow-dumps to 3.3.x, from Open to In Progress.
Wed, Nov 4, 3:01 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work
atsuko changed the status of T443252: airflow-dumps webserver performance from Open to In Progress.
Wed, Nov 4, 3:01 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko added a comment to T443252: airflow-dumps webserver performance.

We now found that there are a few more hot URLs

  • /ui/structure/structure_data?dag_id=mediawiki_dumps_sql_xml_large_a_to_z_full&external_dependencies=false&version_number=2997
  • /ui/structure/structure_data?dag_id=mediawiki_dumps_sql_xml_large_a_to_z_full
Wed, Nov 4, 3:01 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko created P97276 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1362154/8 actual patch.
Wed, Nov 4, 2:48 PM
atsuko created P97259 Diff between pre-40f6ec1c6021f242e80e010043273d2a4cbd4887 and manual back-backport.
Wed, Nov 4, 12:01 PM
atsuko added a comment to T443252: airflow-dumps webserver performance.
Wed, Nov 4, 10:05 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work

Tue, Nov 3

atsuko added a comment to T443252: airflow-dumps webserver performance.

Collecting flamegraphs with pyinstrument

Tue, Nov 3, 1:41 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko created P97100 (An Untitled Masterwork).
Tue, Nov 3, 10:02 AM

Mon, Nov 2

atsuko added a comment to T443252: airflow-dumps webserver performance.

And I see a few recursions here on a smaller DAG https://phabricator.wikimedia.org/P97091

Mon, Nov 2, 7:11 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko created P97091 small.
Mon, Nov 2, 7:10 PM
atsuko added a comment to T443252: airflow-dumps webserver performance.

Tracing in-place, running in webserver container, with the token from the UI

python3 -m trace --count --file /tmp/report -C /tmp/cover /home/app/.local/bin/airflow api-server --port 8088 -l /tmp/log -w 1 --apps core &
time python3 -c "import requests; print(requests.get('http://localhost:8088/ui/grid/structure/mediawiki_dumps_sql_xml_large_a_to_z_full?include_upstream=false&include_downstream=false&limit=1&order_by=-run_after', headers={'Authorization': 'Bearer <token>'}).text)"
Mon, Nov 2, 6:01 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko added a comment to T443252: airflow-dumps webserver performance.

it seems that https://github.com/apache/airflow/issues/57776 is what we see

Mon, Nov 2, 4:39 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko created P97071 (An Untitled Masterwork).
Mon, Nov 2, 3:27 PM
atsuko added a comment to T443252: airflow-dumps webserver performance.

One of endpoints times out and takes about 4 minutes

runuser@airflow-webserver-98c9664b-lv7ml:/opt/airflow$ time python3 -c "import requests; requests.get('http://localhost:8080/ui/grid/structure/mediawiki_dumps_sql_xml_large_a_to_z_full?include_upstream=false&include_downstream=false&limit=1&order_by=-run_after', headers={'Authorization': 'Bearer ...'}).text"
Mon, Nov 2, 2:57 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko created T443252: airflow-dumps webserver performance.
Mon, Nov 2, 2:56 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Patch-For-Review, Essential-Work
atsuko updated the task description for T443226: topolvm doesn't remove stale annotations.
Mon, Nov 2, 12:33 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a parent task for T443045: topolvm fails to provision volume: T443226: topolvm doesn't remove stale annotations.
Mon, Nov 2, 12:32 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a subtask for T443226: topolvm doesn't remove stale annotations: T443045: topolvm fails to provision volume.
Mon, Nov 2, 12:32 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko created T443226: topolvm doesn't remove stale annotations.
Mon, Nov 2, 12:31 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko moved T443045: topolvm fails to provision volume from In Progress to Done on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Mon, Nov 2, 12:28 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko closed T443045: topolvm fails to provision volume as Resolved.

Note:
After changing volume group in topolvm

  1. I needed to restart lvmd with systemctl kill --signal=9 lvmd, otherwise it was stuck
  2. Annotations weren't removed from nodes, I needed to remove those manually and restart topolvm-node
Mon, Nov 2, 12:26 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko changed the status of T443045: topolvm fails to provision volume from Open to In Progress.
Mon, Nov 2, 12:08 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko moved T443045: topolvm fails to provision volume from Respond to Alerts to In Progress on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Mon, Nov 2, 12:08 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko claimed T443045: topolvm fails to provision volume.
Mon, Nov 2, 12:08 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko updated the task description for T443210: research the option to migrate airflow-devenv to a webservice.
Mon, Nov 2, 11:39 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko updated the task description for T443210: research the option to migrate airflow-devenv to a webservice.
Mon, Nov 2, 11:37 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko created T443210: research the option to migrate airflow-devenv to a webservice.
Mon, Nov 2, 10:43 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a comment to T443045: topolvm fails to provision volume.

Seems that topolvm didn't update annotations on the host even after restarting everything

kubectl --namespace=kube-system get node dse-k8s-worker1001.eqiad.wmnet -o yaml  grep topolvm
    capacity.topolvm.io/00default: "1989102534656"
    capacity.topolvm.io/balanced: "1989102534656"
    capacity.topolvm.io/capacity: "1989102534656"
    csi.volume.kubernetes.io/nodeid: '{"cephfs.csi.ceph.com":"dse-k8s-worker1001.eqiad.wmnet","rbd.csi.ceph.com":"dse-k8s-worker1001.eqiad.wmnet","topolvm.io":"dse-k8s-worker1001.eqiad.wmnet"}'
    topology.topolvm.io/node: dse-k8s-worker1001.eqiad.wmnet
Mon, Nov 2, 10:08 AM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)

Fri, Oct 30

atsuko added a comment to T443045: topolvm fails to provision volume.

I feel like WDQS should not be affected, as well as already scheduled Pods.

Fri, Oct 30, 5:46 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a comment to T443045: topolvm fails to provision volume.

This storage class was removed from those servers in T430654: Re-use the ml-cache* machines as dse-k8s-worker nodes supporting the topolvm-balanced storageclass, dse-k8s-eqiad: Reassign dse-k8s-worker1001-4 to the right storageclass and it seems to be applied or broke only after reboot on October 23rd. That explains why it started failing on October 24th.

Fri, Oct 30, 5:42 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a comment to T443045: topolvm fails to provision volume.

seems to be a misconfiguration, mentioned servers has device-class capacity, and balanced is available only in dse-k8s-wdqs1004.eqiad.wmnet,dse-k8s-wdqs-test1001.eqiad.wmnet,dse-k8s-worker[1042-1044].eqiad.wmnet

Fri, Oct 30, 5:28 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a comment to T443045: topolvm fails to provision volume.

The pods are corresponding to

  • dse-k8s-worker1001.eqiad.wmnet
  • dse-k8s-worker1004.eqiad.wmnet
Fri, Oct 30, 5:20 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko moved T443045: topolvm fails to provision volume from Backlog - project to Respond to Alerts on the Data-Platform-SRE (2026-10-30 - 2026-11-20) board.
Fri, Oct 30, 5:16 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko edited projects for T443045: topolvm fails to provision volume, added: Data-Platform-SRE (2026-10-30 - 2026-11-20); removed Data-Platform-SRE.
Fri, Oct 30, 5:16 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko created T443045: topolvm fails to provision volume.
Fri, Oct 30, 5:15 PM · Wikidata Platform Team, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko created P97037 json-lod from topolvm node.
Fri, Oct 30, 5:15 PM
atsuko renamed T442864: Airflow 3.3: Adjust "Generate API and CLI tokens from web interface" feature from Airflow 3.3: Generate API and CLI tokens from web interface feature to Airflow 3.3: Adjust "Generate API and CLI tokens from web interface" feature.
Fri, Oct 30, 4:35 PM · Essential-Work, Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko updated the task description for T423248: Upgrade to Airflow 3.3.x.
Fri, Oct 30, 4:33 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic

Thu, Oct 29

atsuko updated the task description for T434642: Migrate from SLA to Deadline.
Thu, Oct 29, 3:28 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko updated the task description for T434642: Migrate from SLA to Deadline.
Thu, Oct 29, 3:28 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko added a subtask for T423248: Upgrade to Airflow 3.3.x: T442864: Airflow 3.3: Adjust "Generate API and CLI tokens from web interface" feature.
Thu, Oct 29, 1:20 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko created T442864: Airflow 3.3: Adjust "Generate API and CLI tokens from web interface" feature.
Thu, Oct 29, 1:20 PM · Essential-Work, Data-Platform-SRE (2026-10-30 - 2026-11-20)

Wed, Oct 28

atsuko archived P97017 (An Untitled Masterwork).
Wed, Oct 28, 6:26 PM
atsuko added a comment to T442190: Blunderbuss: Have analytics_test artifacts go to analytics_test HDFS cluster.

Fixed permissions on analytics-test-hadoop,

sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chmod 755 /wmf/cache
Wed, Oct 28, 6:24 PM · Data-Engineering (Q2 FS26/27 October 1st - December 31th)), Patch-For-Review
atsuko created P97017 (An Untitled Masterwork).
Wed, Oct 28, 6:16 PM
atsuko added a comment to T434642: Migrate from SLA to Deadline.

I tried patching create_easy_dag and it breaks serialisation..

Wed, Oct 28, 4:30 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko renamed T434642: Migrate from SLA to Deadline from Create diffs for the DAG owners to migrate from SLA to Deadline to Migrate from SLA to Deadline.
Wed, Oct 28, 4:28 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko updated the task description for T434642: Migrate from SLA to Deadline.
Wed, Oct 28, 3:20 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20)
atsuko changed the status of T435476: Upgrade airflow-analytics-product to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, from Open to In Progress.
Wed, Oct 28, 12:22 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko changed the status of T435476: Upgrade airflow-analytics-product to 3.3.x from Open to In Progress.
Wed, Oct 28, 12:22 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Product-Analytics, Essential-Work
atsuko moved T435476: Upgrade airflow-analytics-product to 3.3.x from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Wed, Oct 28, 12:22 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Product-Analytics, Essential-Work
atsuko claimed T435476: Upgrade airflow-analytics-product to 3.3.x.
Wed, Oct 28, 12:22 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Product-Analytics, Essential-Work
atsuko added a comment to T435481: Upgrade airflow-ml to 3.3.x.

Following T435481#12476139, we added example_training_dag in the airflow-ml instance.

Wed, Oct 28, 12:11 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20), Machine-Learning-Team (Q2 FY2026-27), Essential-Work
atsuko updated the task description for T423248: Upgrade to Airflow 3.3.x.
Wed, Oct 28, 11:37 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko closed T435486: Upgrade airflow-wikidata to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, as Resolved.
Wed, Oct 28, 11:37 AM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko closed T435486: Upgrade airflow-wikidata to 3.3.x as Resolved.
Wed, Oct 28, 11:36 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Wikidata Platform Team
atsuko moved T435486: Upgrade airflow-wikidata to 3.3.x from In Progress to Done on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Wed, Oct 28, 11:36 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Wikidata Platform Team
atsuko added a comment to T435486: Upgrade airflow-wikidata to 3.3.x.
Wed, Oct 28, 11:07 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Wikidata Platform Team
atsuko closed T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures as Resolved.
Wed, Oct 28, 10:19 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko moved T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures from In Progress to Done on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Wed, Oct 28, 10:19 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko added a comment to T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures.

No more retries since you merged your change. I think you found the culprit :) thank you for this !

Wed, Oct 28, 10:19 AM · Data-Platform-SRE (2026-10-09 - 2026-10-30)

Tue, Oct 27

atsuko created P97009 (An Untitled Masterwork).
Tue, Oct 27, 5:35 PM
atsuko closed T435479: Upgrade airflow-fr-tech to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, as Resolved.
Tue, Oct 27, 5:07 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko closed T435479: Upgrade airflow-fr-tech to 3.3.x as Resolved.
Tue, Oct 27, 5:07 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Fundraising-Backlog, Fundraising-Analysis
atsuko moved T435479: Upgrade airflow-fr-tech to 3.3.x from Needs Review to Done on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Tue, Oct 27, 5:07 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Fundraising-Backlog, Fundraising-Analysis
atsuko updated the task description for T442431: Upgrade airflow-experiment-platform to 3.3.x.
Tue, Oct 27, 4:41 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Epic
atsuko closed T435478: Upgrade airflow-dumps to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, as Resolved.
Tue, Oct 27, 4:28 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko closed T435478: Upgrade airflow-dumps to 3.3.x as Resolved.
Tue, Oct 27, 4:28 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work
atsuko moved T435478: Upgrade airflow-dumps to 3.3.x from Needs Review to Done on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Tue, Oct 27, 4:28 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work
atsuko added a comment to T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures.

Merged, looking at https://airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test/tasks/webrequest_frontend_test/task_instances for a day

Tue, Oct 27, 4:14 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko updated the task description for T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures.
Tue, Oct 27, 2:34 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko updated the task description for T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures.
Tue, Oct 27, 2:19 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko created P97006 skein/gobblin.jar.
Tue, Oct 27, 2:18 PM
atsuko added a comment to T442524: Investigate airflow-analytics-test.wikimedia.org/dags/gobblin_webrequest_frontend_test intermittent failures.

I found next lines that are suspicious:

Tue, Oct 27, 2:15 PM · Data-Platform-SRE (2026-10-09 - 2026-10-30)
atsuko claimed T435481: Upgrade airflow-ml to 3.3.x.
Tue, Oct 27, 1:47 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20), Machine-Learning-Team (Q2 FY2026-27), Essential-Work
atsuko claimed T435479: Upgrade airflow-fr-tech to 3.3.x.
Tue, Oct 27, 1:46 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Fundraising-Backlog, Fundraising-Analysis
atsuko changed the status of T435483: Upgrade airflow-research to 3.3.x, a subtask of T423248: Upgrade to Airflow 3.3.x, from Open to In Progress.
Tue, Oct 27, 1:46 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Essential-Work, Patch-For-Review, Epic
atsuko claimed T435483: Upgrade airflow-research to 3.3.x.
Tue, Oct 27, 1:46 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Research, Research-engineering, Essential-Work
atsuko changed the status of T435483: Upgrade airflow-research to 3.3.x from Open to In Progress.
Tue, Oct 27, 1:46 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Research, Research-engineering, Essential-Work
atsuko moved T435483: Upgrade airflow-research to 3.3.x from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Tue, Oct 27, 1:46 PM · Data-Platform-SRE (2026-10-30 - 2026-11-20), Research, Research-engineering, Essential-Work
atsuko moved T435481: Upgrade airflow-ml to 3.3.x from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Tue, Oct 27, 1:45 PM · Patch-For-Review, Data-Platform-SRE (2026-10-30 - 2026-11-20), Machine-Learning-Team (Q2 FY2026-27), Essential-Work
atsuko moved T435479: Upgrade airflow-fr-tech to 3.3.x from In Progress to Needs Review on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Tue, Oct 27, 1:39 PM · Patch-For-Review, Data-Platform-SRE (2026-10-09 - 2026-10-30), Essential-Work, Fundraising-Backlog, Fundraising-Analysis