How can MW developers access the output of currently-runinng (or recently-completed) scheduled maintenance scripts that execute in Kubernetes?

Today this works via journalctl on mwmaint hosts. https://wikitech.wikimedia.org/wiki/Maintenance_server#Access_recent_runs

This question is related to T285896, which could potentially solve this by accessing it via Logstash. Ideally there'd still be a way to do it from the command-line as this would allow for real-time tailing, and easier grepping (Logstash is fairly hostile to simple search ops on small quantities of logs).

\cc @RLazarus

Script output is visible through kubectl logs, and mwscript-k8s can be invoked with -f to immediately start tailing the script output (under the hood, it just invokes that kubectl command). If you don't launch with -f it prints out the kubectl command so you can copy and paste it.

We'll eventually send it to logstash too, but that hasn't happened yet.

In T341560#9967304, @RLazarus wrote:

We'll eventually send it to logstash too, but that hasn't happened yet.

Everything a container produces on stdout and stderr should land in logstash automatically

I’m not sure if it should be a subtask of this or not, but if we’re planning to sunset the mwmaint servers, see also: T384764: mwgrep cannot be used from a deployment host

I recently stumbled upon instructions on Wikitech mentioning mwmaint1002 and since this wasn't the first time, I decided to take some time to update a bunch of these instructions. The replacement isn't always obvious among deployment host, mwdebug, mw-experimental, and TBD.

• Maintenance script ad-hoc runs: Deployment server via mwscript or mwscript-k8s.
  • Backport windows/Deployers
  • Reattach a global account
  • Regenerate cached special pages
  • Uploading large files
  • Generating CAPTCHAs
  • Password and 2FA reset
• Debugging in production § Debugging databases
• Debugging in production § Debugging in shell
• WikimediaDebug#Debug logging (CLI)
• foreachwiki command: Deployment server. (docs edit)
• sql command: Deployment server. (docs edit)
• MaxMind debugging: Deployment server. (docs edit)
• noc.wikimedia.org verification: WikimediaDebug (when mid-deploy with scap this works because k8s-mw-misc serves noc.wikimedia.org, and follows k8s-mw-debug rather than k8s-mw-web, and so receives staged changes).

TBD:

• Maintenance script tinkering: No documented replacement (ref T405688).
  • https://wikitech.wikimedia.org/wiki/Debugging_in_production#Debugging_a_maintenance_script
• noc.wikimedia.org tinkering: No documented replacement (ref T395682).

Thanks for the edits, but in the future, you can just flag them to us, and we'll coordinate rewriting the documentation to be up to date.
I've had a quick pass to replace some mwscript calls with mwscript-k8s, but some of these need an actual rewrite, some are redundant with already written documentation, some require consulting with other teams before rewrite, and I'm wary of having documentation that has been "updated" with outdated procedures (for instance using old-style mwscript from deployment servers in use-cases where mwscript-k8s works, which is everything but importImages.php)