@JMeybohm do you know if we could stop this alert from creating new tasks?
I don't think there was a new failure since I closed the last one but somehow I still see the errors dangling in k8s jobs:

backfill done, sadly without filtering of automated traffic the data remains quite noisy.

In T441011#12426782, @EBernhardson wrote:

Poking at mediawiki, what gets used is SpecialPageFactory::resolveAlias. It applies a couple extra rules against the alias list:

• uses the content language object to casefold, which looks to result in calling php's mb_strtoupper. (ex: Special:SEARCH goes to Special:Search)
• Applies an extra check for turkish/azerbaijani which attempts to swap İ with I for T288402.

In T440913#12424123, @JMeybohm wrote:

FWIW I can get the logs with the command from the description:

 # kubectl logs -l cronjob=cirrus-build-completion-indices-codfw-s3 --timestamps=true --prefix=true -c mediawiki-main-app
[...]
 [pod/cirrus-build-completion-indices-codfw-s3-29862870-tc4rc/mediawiki-main-app] 2026-10-12T12:08:18.446309905Z extensions/CirrusSearch/maintenance/UpdateSuggesterIndex.php: Finished run

I see in SAL 2026-10-13T07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1153.eqiad.wmnet, worker1153 seems to be the machine that was hosting the pod that failed, going to assume that the depool caused the jobs to fail early and got killed by the depool cookbook.

Tagging ServiceOps for help.
According to logstash the job seems to stop mid-process without any error reported from the application.
The corresponding pod does not appear to be around and logs from kubectl are not viewable.

dcausse@deploy1003:~$ kubectl get jobs -l team=search-platform,cronjob=cirrus-build-completion-indices-eqiad-s3 --field-selector status.successful=0;
NAME                                                STATUS    COMPLETIONS   DURATION   AGE
cirrus-build-completion-indices-eqiad-s3-29857110   Failed    0/1           5d6h       5d6h
cirrus-build-completion-indices-eqiad-s3-29864310   Running   0/1           6h15m      6h15m

CirrusSearch has some support for a "common" index. This is used for NS_FILE with a config var named wgCirrusSearchExtraIndexes which maps a MW namespace to a "shared" index. This shared index is owned by another wiki (commons for NS_FILE). Deduplication is done at search time and thus only works if the number of duplicated pages is low. So I believe that the base support for a shared index is there, I would not surprised that it might need some adaptation because this feature has only been used for NS_FILE.
Couple questions:

• does "search integration" mean: autocompletion support in the main search box and Special:Search support?
• what wiki will own the "shared index"?
• are the produnto packages considered mediawiki titles and edited via the classic mediwiki editing workflows?

probably a transient failure, I can't find anything obvious in the logs except that the job stopped abruptly and the status has nothing obvious in it:

status:
  conditions:
  - lastProbeTime: "2026-10-08T06:44:28Z"
    lastTransitionTime: "2026-10-08T06:44:28Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: FailureTarget
  - lastProbeTime: "2026-10-08T06:44:38Z"
    lastTransitionTime: "2026-10-08T06:44:38Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: Failed
  failed: 1
  ready: 0
  startTime: "2026-10-08T02:30:00Z"
  terminating: 0
  uncountedTerminatedPods: {}

In T436736#12346270, @dcausse wrote:

In T436736#12341452, @RKemper wrote:

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

Oops my bad, I can re-run the extraction with this logic in mind, this should remove the noise regarding undeletes.
I doubt it'll change the conclusion of this, in that either dt or meta.dt for the dedup/merge window is fine (both can be wrong).

Re-ran the job on all events in September. over the 46M events, only 10 cases see a disagreement between event_time and meta.dt

Fix is deployed. Keeping this task open the time needed to backfill as much as we can (data between mid-may and june 14 is lost, will add a note to the dashboard).

config issue errors dropped to 0

Updated https://grafana-rw.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?from=now-7d&to=now&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search&viewPanel=panel-9 to include partial results

In T361246#12412440, @dancy wrote:

@dcausse: Do you happen to remember what deployment command you issued that resulted in this ticket?

Recording some limitations about the current schema evolution strategy:
Even if refine were to support nested structs with the current version of spark/iceberg we use there will be problems similar to the one found by Antoine at https://github.com/apache/iceberg/issues/14043.
This is apparently not limited to iceberg and affects spark with plain parquet as well. This is sadly not limited to map of struct but also array of struct and more annoyingly struct of struct too when adding a new field:

Started extraction at:

• jina https://yarn.wikimedia.org/proxy/application_1786527810139_1215765/
• qwen3 https://yarn.wikimedia.org/proxy/application_1786527810139_1215764/

Recorded metrics at P96592.
Observations:

• none of the features tested have led to a measurable impact on the load.
• we saw a decrease in net usage when reducing the function score window size, this is surprising as it suggests that the data transmitted between nodes depend on this size which contradict our current understanding of the rescore mechanism [TBD: ticket to investigate]
• interwiki search requests are captured in the main requests bucket, we might want to separate those [TBD: ticket to adjust metrics]
• enwiki served by codfw led to no big decrease in load in eqiad
• morelike cache hit rate dropped when reducing the function score window size, suggesting that non-negligible use-cases are using morelike with the default rescore profile [TBD: ticket to investigate such use-cases]

1. CirrusSearch perf eval, 2026-10-05 (UTC)

The fix should be live on all WMF wikis

Should now be fixed on all WMF wikis.

For posterity this is the outcome of deploying this optimization to all wikis:

In T440147#12394042, @neriah wrote:

Maybe T433224#12393862 caused this?

In T433224#12393862, @neriah wrote:

@dcausse, thanks for the fix!
Just one small problem, which I'm mentioning here because it seems related (it only happens for a special namespace): Search results for special pages appear twice, which is quite annoying. I checked other wikis where DWIM isn't enabled (enwiki), and the problem exists there too.

Actually looking closer I suspect a more likely cause is problems fetching the target wiki config during interwiki searches, in case of failures we fallback to a constructed minimal config. This artificial config was fine before we switched to a multi-cluster setup but now there's no way to actually construct this config reliably because we need to know what is the target cluster.

Suspecting cross cluster searches issued while fetching interwiki results having some race issues because we ccs_minimize_roundtrips is true by default. Trying to reproduce this might take time since the error rate is quite low. Possible fix to reduce the noise could be to relax the search query with ignore_unavailable=true & allow_no_indices=true for interwiki searches (these are generally best efforts and errors are not presented to users, plus I doubt that we want disable ccs_minimize_roundtrips).

In T436866#12370107, @EBernhardson wrote:

i wonder if we should keep deepcat as it's own pool, or perhaps fold it into the expensive-full-text pool that we generalized regex into?

In T411999#12373785, @JAllemandou wrote:

If I don't mistake, the repo you pointed don't have the refinery code you reference.

Would the existing refinery docker image built from https://gitlab.wikimedia.org/repos/data-engineering/refinery be suited for this task. The search airflow instance already uses this image to run /opt/refinery/bin/refinery-drop-older-than.

should be done, next Monday we'll see if the pipeline worked automatically.

In T436736#12341452, @RKemper wrote:

@dcausse Sorry for taking so long on this. I took a look; the notebook uses revision time for undeletes, but Erik's fix already made those use the event's dt (the same existing behavior I overlooked in my earlier patch). Does that change your leaning?

Tagging CirrusSearch we'll look into it, note that if you don't need private wikis you can also use https://global-search.toolforge.org

Should be done.

3.7 should be running there with opensearch-ml-extra required for semantic highlighting

Fixed for me, thanks for the quick fix!

Following-up here based on discussions on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1340539.
General guidelines:

• CirrusSearch should not depend on DiscussiontTools, since the data is "owned" by DiscussiontTools the mapping, the SearchDataForIndex2 hook and the search keyword implementation should be added to DiscussiontTools. This means that DiscussionTools will depend on CirrusSearch and probably something that needs agreement from the DiscussionTools maintainers.
• nested fields should be avoided, I understand the need for nested field if we want to perfectly match the comment text and its author but given the ticket description I am not clear if this is a strong requirement, for instance would filtering by author be already sufficient to narrow down the search space? Specific examples would greatly help I think here. If there are possible workarounds we should probably explore them first.

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseQualityConstraints/+/1341601 failed with the same error so suspecting something specific to WikibaseQualityConstraints.

Tentatively triaging as high because it breaks CI on unrelated projects.

Should be done

Adding DiscussionTools, my understanding is that this data is "owned" by this extension and thus should also own the search logic behind it.
There should be some estimate about the expected volume and cost before moving forward.

In T436736#12300123, @RKemper wrote:

@dcausse I reverted the first fix before deployment, turns out using receipt time for merge ordering could let a delayed edit displace a delete of the same revision

possibly related: T430588