Page MenuHomePhabricator

Update cirrus for mwscript-on-k8s
Closed, ResolvedPublic

Description

With MediaWiki at the WMF moving to Kubernetes, it's now time to start running manual maintenance scripts there. The old way will continue working for a little while, but it will be going away. Note that while the announcement is only for "manual maintenance scripts", it's probably safe to assume invoking mwscript from other scripts that invoke mwscript falls under a similar umbrella.

Notable changes:

  • mwscript-k8s does not, by default, wait for completion before returning
  • mwscript-k8s does not, by default, print the script output

Possible remediations::

  • mwscript-k8s includes a --follow flag which stream the logs. Does not propagate the return code. Note that this does provide output beyond what the script did, but it comes over stderr instead of stdout and should be ignorable.
  • Maybe ExpectedIndices.php could be provided via api? There is nothing secret in there, it simply seemed odd to expose to the public api since it's entirely operational in nature.

Problematic places include:

  • cirrus-reindex-orchestrator - in gitlab/repos/search-platform/cirrus-reindex-orchestrator
    • invokes ExpectedIndices for all wikis and expects to read the json formatted result
    • invokes UpdateOneSearchIndexConfig. Expects the call to delay until completion, and for the exit code to indicate one of three possible outcomes of the requested index update.
  • cleanup_weighted_tags.py - in CirrusSearch repo
    • invokes ExpectedIndices for all wikis and expects to read the json formatted result
  • check_indices.py - in CirrusSearch repo
    • invokes ExpectedIndices for all wikis and expects to read the json formatted result
  • dumpcirrussearch.sh - in puppet
    • Invokes DumpIndex.php and expects the return code to indicate success or failure. Additionally expects the call to not return until complete, as it will start another dump immediately after.
    • This is not the only dumps script using mwscript, we may be able to lean on whatever solution is found for the rest of the dumps and timers.

Event Timeline

Gehel triaged this task as High priority.Oct 14 2024, 2:47 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Note that while the announcement is only for "manual maintenance scripts", it's probably safe to assume invoking mwscript from other scripts that invoke mwscript falls under a similar umbrella.

Yes, sorry -- the intent of "manual" was "not launched on a schedule by a systemd timer." Any time mwscript is invoked from a script, as you've described here, that's also in scope. (And everything including systemd timers will be migrating too, eventually.) Thank you for pointing that out.

With the updates in T377292#10251796 you should be unblocked here.

Not estimating this ticket at the moment, most work will happen in T378382. We can come back to this one afterward if soemthing is left.

Moving back to our backlog. Some specific cases are addressed in T378382, the rest will be revisited once we have made more progress.

dcausse claimed this task.
dcausse subscribed.

All usage of mwscript for CirrusSearch have been migrated to mwscript-k8s:

The couple systemd timers for CirrusSearch will be migrated to mwcron as part of T388538