Page MenuHomePhabricator
Create Task
Maniphest T400158

cirrussearch dumps have failed - 2025-07-21
Closed, ResolvedPublic
Actions

Assigned To
BTullis
Authored By
BTullis
Jul 22 2025, 1:23 PM
Tags
Referenced Files
F65689576: image.png
Jul 29 2025, 6:52 PM
F65686140: image.png
Jul 28 2025, 1:11 PM
F65685903: image.png
Jul 28 2025, 11:27 AM
F65588829: image.png
Jul 22 2025, 3:39 PM
F65588244: image.png
Jul 22 2025, 1:23 PM
Subscribers
Aklapper
bking
BTullis
dr0ptp4kt
EBernhardson
Kantakuziv
William_Avery
WMDE-leszek

Description

Our cirrussearch dumps have all failed with the same error at the start of the weekly dump.
https://airflow-test-k8s.wikimedia.org/dags/mediawiki_cirrussearch_dump/grid

image.png (810×754 px, 94 KB)

Note that the dumps are also still running on snapshot1016, although these are legacy dumps and are about to be switched off.

The logs of the failed jobs all say this:

[2025-07-21, 17:30:34 UTC] {pod_manager.py:520} INFO - [base] Named cluster (dnsdisc) is not configured for maintenance operations. Allowed clusters: eqiad, codfw, cloudelastic
[2025-07-21, 17:30:34 UTC] {pod_manager.py:520} INFO - [base] extensions/CirrusSearch/maintenance/DumpIndex.php failed for /mnt/dumpsdata/otherdumps/cirrussearch/20250721/enwiki-20250721-cirrussearch-content.json.gz

Here is an example.

Details

Other Assignee
BTullis
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Allow index dump from non-managed clustermediawiki/extensions/CirrusSearchwmf/1.45.0-wmf.11+28 -1
Allow index dump from non-managed clustermediawiki/extensions/CirrusSearchmaster+28 -1
Customize query in gerrit

Related Objects

Event Timeline

BTullis created this task.Jul 22 2025, 1:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2025, 1:23 PM
BTullis triaged this task as High priority.Jul 22 2025, 1:23 PM
gerritbot added a comment.Jul 22 2025, 1:43 PM

Change #1171581 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Allow index dump from non-managed cluster

https://gerrit.wikimedia.org/r/1171581

gerritbot added a project: Patch-For-Review.Jul 22 2025, 1:43 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 22 2025, 2:52 PM
gerritbot added a comment.Jul 22 2025, 3:06 PM

Change #1171581 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Allow index dump from non-managed cluster

https://gerrit.wikimedia.org/r/1171581

EBernhardson moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 22 2025, 3:26 PM
bking subscribed.Jul 22 2025, 3:31 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 22 2025, 3:31 PM
BTullis added a comment.Jul 22 2025, 3:39 PM

OK, so the patch is merged. When a new version of mediawiki has been deployed, what we need to do to trigger the re-run of the dumps is to clear this task.

image.png (696×1 px, 161 KB)

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.12; 2025-07-29).Jul 22 2025, 4:01 PM
dr0ptp4kt subscribed.Jul 22 2025, 4:56 PM
William_Avery subscribed.Jul 23 2025, 4:51 PM
WMDE-leszek subscribed.Jul 23 2025, 6:18 PM
BTullis claimed this task.Jul 28 2025, 11:24 AM
BTullis added a subscriber: EBernhardson.
BTullis added a comment.Jul 28 2025, 11:27 AM

This is still not working. Even though we are running on a mediawiki image generated today, the errors messages remains the same.

image.png (738×1 px, 183 KB)

I believe that @EBernhardson is out this week, so I'll reassign it to myself and try to make progress.

BTullis moved this task from To Be Deployed to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 28 2025, 11:30 AM
gerritbot added a comment.Jul 28 2025, 11:50 AM

Change #1173363 had a related patch set uploaded (by Reedy; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@wmf/1.45.0-wmf.11] Allow index dump from non-managed cluster

https://gerrit.wikimedia.org/r/1173363

gerritbot added a project: Patch-For-Review.Jul 28 2025, 11:50 AM
gerritbot added a comment.Jul 28 2025, 12:41 PM

Change #1173363 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.45.0-wmf.11] Allow index dump from non-managed cluster

https://gerrit.wikimedia.org/r/1173363

Stashbot added a comment.Jul 28 2025, 12:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T12:51:11Z] <reedy@deploy1003> Started scap sync-world: Backport for [[gerrit:1173363|Allow index dump from non-managed cluster (T400158)]]

Stashbot added a comment.Jul 28 2025, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T12:55:15Z] <reedy@deploy1003> reedy: Backport for [[gerrit:1173363|Allow index dump from non-managed cluster (T400158)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.11; 2025-07-22); removed MW-1.45-notes (1.45.0-wmf.12; 2025-07-29).Jul 28 2025, 1:02 PM
Stashbot added a comment.Jul 28 2025, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T13:04:50Z] <reedy@deploy1003> Finished scap sync-world: Backport for [[gerrit:1173363|Allow index dump from non-managed cluster (T400158)]] (duration: 13m 39s)

BTullis moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 28 2025, 1:09 PM

The backport has been deployed and I have restarted the cirrusseearch dumps.
https://airflow-test-k8s.wikimedia.org/dags/mediawiki_cirrussearch_dump/grid?dag_run_id=scheduled__2025-07-14T16%3A15%3A00%2B00%3A00&task_id=dump_cirrussearch_sections.execute_cirrussearch_dump_s8&tab=logs

Things look OK, so far.

BTullis added a comment.Jul 28 2025, 1:11 PM

image.png (864×1 px, 307 KB)

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 1:31 PM
BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:45 PM
BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis closed this task as Resolved.Jul 29 2025, 6:52 PM
BTullis moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.

I'll tentatively resove this ticket for now, as they seem to be proceeding well.
If we see futher issues, then we can reopen.

image.png (657×732 px, 73 KB)

EBernhardson removed a project: Discovery-Search.Aug 11 2025, 3:18 PM
Kantakuziv subscribed.EditedAug 14 2025, 2:03 PM

Hello, @BTullis
CirrusSearch dumps are still not available here - https://dumps.wikimedia.org/other/cirrussearch/ Looks like they are not published after being created. So I guess this ticket should be reopened.
Or maybe they can be accessed somewhere else. In this case could you please clarify how to access them now?

Kantakuziv added a comment.Aug 14 2025, 2:17 PM

I have created a new ticket for this issue - https://phabricator.wikimedia.org/T401923

Kantakuziv mentioned this in T401923: CirrusSearch dumps are not published .Aug 15 2025, 10:11 AM
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.Aug 20 2025, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits