@Zache Thanks for moving the dataset into the query string. Sparqlbridge is now on the allowlist as https://sparqlbridge.toolforge.org/sparql* (https://gerrit.wikimedia.org/r/1346704), and the WDQS hosts have been restarted to pick it up

kafka-configurator 0.0.1-1~wmf13u1 is available on apt now: https://apt-browser.toolforge.org/trixie-wikimedia/main/#:~:text=kafka-configurator

@pfischer I would rip out any SLIs/SLOs that don't map well to Search's actual service boundaries, so for example the client side rendering thing, that might be a good *alert* for search team (because plausibly we could accidentally break that, and also it's a real signal of degraded user experience even if it's not "our fault"), but would not be a good SLO.

In T434896#12353941, @atsuko wrote:

Simple up-for-grabs for rewriting Database access to REST API

• wmf_airflow_common/operators/dbt.py: updates TaskInstance
• wmf_airflow_common/plugins/pod_resources_grafana_link.py: getting data from TaskInstance
• analytics_test/dags/report_skein_oom.py: complex query for TaskInstance and then update

(and anoter example for how to use client)

All 3 hosts are in their desired pools; 1033 and 1034 are main, 1035 is internal-scholarly

In T435477#12353825, @atsuko wrote:

Pool patch merged and working in my local devenv. I also tried refine_to_hive_hourly_test which was previously blocked on the pool issue, and it gets past that step now. I did hit a separate LocalExecutor vs KubernetesExecutor mismatch in the devenv, but that occurs after the pool creation

@RKemper do you know what is the safe way to test the job? there are options how to go with it in T435172. We can try fixing the job options, or I maybe will be able to update the version for every job.

In T435172#12227100, @BTullis wrote:

However, fixing each and every job might not be the solution we need to move forward.

It would be great to know how many jobs use spark directly from any of our DAGs.
Maybe we could do some kind of analysis of how many tasks using different operators there are in total.

Alright currently we have only one host in eqiad wdqs-internal-scholarly; we should bump that to 2, and leave the rest of the hosts to public-main which actually needs the compute:

Willy followed up on this, I'll drive this forward

Pool patch merged and working in my local devenv. I also tried refine_to_hive_hourly_test which was previously blocked on the pool issue, and it gets past that step now. I did hit a separate LocalExecutor vs KubernetesExecutor mismatch in the devenv, but that occurs after the pool creation

Merged the above patches, and tested out the fresh .deb in pontoon.

Balthazar pointed out that 5 min cadence might generate a lot of metadata churn, so I've bumped it to 24h per his suggestion

@atsuko Yeah this local API method is way cleaner, thanks for figuring that out!

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

Almost got the patches ready, using my previous wdqs work from back in the day as a reference.

This would be a nice-to-have, but there isn't a strong enough need to justify prioritizing this. I'll close this for now and can re-open in the future if I have bandwidth and decide this is worth it

Traceback:

  File "/opt/airflow/dags/airflow_dags/wmf_airflow_common/sensors/rest_external_task.py", line 13, in <module>
    from wmf_airflow_common.sensors.rest_helper import (
  File "/opt/airflow/dags/airflow_dags/wmf_airflow_common/sensors/rest_helper.py", line 8, in <module>
    from airflow_client.client.model.dag import DAG
ModuleNotFoundError: No module named 'airflow_client.client.model'

Tried aqs_hourly_test yesterday, the shared sensor helpers still import airflow_client.client.model and airflow_client.client.model_utils

It works!

I've uploaded a patch that should remove 80+% of the noise in the original ticket desc screenshot.

Tested test_docker_image_fsspec in the Airflow 3.3.1 devenv. Both HDFS sensors and the DAG run succeeded

In T434896#12288844, @atsuko wrote:

Attempted to hack on wmf_airflow_common/pool.py, and got blocked on API access. We can potentially reduce the complexity of this, by moving the Pool creation in a maintenance DAG.

I've been trying to think of the least hacky way to do this. I think it's a 4 step chain. The above patch is step 1 of the chain:

I think we can sidestep the api issue entirely (at least wrt this pool issue) by moving pool creation into a setup step and keeping DAG parsing limited to just name validation

@dcausse I reverted the first fix before deployment, turns out using receipt time for merge ordering could let a delayed edit displace a delete of the same revision

While testing analytics-test I found a DAG analytics_test/dags/refine/refine_to_hive_hourly_dag.py that fails to parse with RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0. The UI only shows the bottom of the traceback (create_session hitting the sandbox's blocked session), but the only thing in that file that touches the db at import time is pool(...)

@dcausse I uploaded a reference patch, lmk if it looks sane

Following up here: the thanos swift object-expirer was enabled on July 15 in https://gerrit.wikimedia.org/r/1310971, and the subsequent account behavior shows that the WDQS backlog drained

I did stumble across one data-quality bug where lastBatchMedianLag on psi/omega occasionally reports multi-year lag values, so I'll get a separate ticket open for that.

The quick burn (2h/1d) on 2026-07-01 was real (T429844, T430843) and recovered on its own. The long-window (6h/3d) burn that wouldn't converge was an SLO bug: the search_sli_update_lag:bool recording rule had no aggregation, so every Flink restart left behind stale per-attempt series (task_attempt_num, job_id, pod, etc) that kept feeding old values into the burn-rate windows.

This is done.

Now that this incident is resolved, it's worth circling back on the root cause. I enlisted the help of a trusty clanker to trace the Presto source code to better understand the issue.

We did something very similar in https://wikimedia.slack.com/archives/C0B8DMX1ZGS/p1787758886054069 (not visible to community members unfortunately, thus why I'm schlepping relevant context here in the hope it's helpful in the future):

In T435477#12253291, @brouberol wrote:

annotate_oomed_skein_task_instances can be dropped I think. It was more a thought experiment than anything else.

Spot-checked some charts; LGTM following the rolling-restart. I'll leave to DE to move to Done and set Resolved if things look good on their end now

Spun up airflow-devenv like so: airflow-devenv create --branch main_airflow_3.3 --extra-values values-airflow3.yaml -d analytics_test

Met with Atsuko to get the rundown; I'll be handling the DPE SRE side of this

In T435555#12239691, @Scott_French wrote:

Dropping to Medium, as the trigger workload has been stopped.

@Snwachukwu was able to confirm that this was a one-off backfill job for wmf_content.mediawiki_html_content_current_v0 for commonswiki, and that there was a similar one for enwiki that completed yesterday. This aligns well with the two periods of rapid disk usage growth we saw on install1005 (webproxy).

The next step here is to figure out a better mechanism for proxying these requests outbound. That may look a lot like the discussion that's already happened in T429338, where it was decided that the url-downloaders were the right way to go. However, that might depend on the total number of requests and anticipated request rate.

Let's use this task to sort that out.

Puppet patch merged; the deployment-charts patch can be rolled out tomorrow, at which point I'll be able to close this out.

Patches rolled out; this is done

Can't find any more dangling references, this is done

PoC is working end-to-end in pontoon. Working on documentation for spin-up instructions, but in meantime the most important patchset to review is this one adding the basic required functionality to the kafka-configurator tool:

@VRiley-WMF We discussed this internally, you're free to proceed whenever. These hosts aren't lvs-backed so no depool concerns there, and the cluster should weather the single node loss without issue

an-worker[1178,1142,1219] are done (see the under-replicated block graph). That leaves two journal-nodes left:

• an-worker1151 still needs to be done and will move its VLAN
• an-worker1213 is blocked on missing disks, see T434494#12209021

Deployed 0.3.165 to prod 2026-08-07 evening

Okay, I think I figured out what's going on here.

Ah, we're missing the preseed entry. Working on patch

Back to working on getting wdqs1023 up via a fresh reimage with --new flag. Seeing this currently:

Okay killed it, and invoked with proper arg. (Ran from Brian's tmux just because it was a quick correction on an in-flight cookbook; will run any net-new stuff from my user)

I don't see any indication it's the patch; wdqs1023 is ssh unreachable so that would be the source of the existing failures. Separately I see https://sal.toolforge.org/log/cswOgZ8BffdvpiTrt6hO being ran with the wrong arg which is gonna transfer the full journal rather than the wikidata_main journal (or more accurately, it will get the right journal file but have its mutation topics set to that of the full graph's which will leave the data in an unknown state)

wdq1027 and wdqs1025 are done (ignore the failed transfer at the end, fat-finger) and I repooled dnsdisc for wdqs-scholarly-internal-eqiad (which has only 1 host thus required dnsdisc depool for reimage)

Carrying context over from slack so it doesn't get lost to time:

Ah, the druid_analytics point is good context, I wasn't aware of that.

We've got the third broker up now that some old instances have been cleaned up, so we're all set to start on the fun part (actually figuring out how to make config management work)