In T404804#11231334, @Urbanecm_WMF wrote:

I spent some time looking into how we can get more data about this. First, I was thinking how to trigger profiling code from src/Profiler.php (in the config repo) in a maintenance script context. [..]

In production, mw-experimental is the environment for livehacking. This made me ask: how can I run maintenance scripts in mw-experimental?

It turns out that with the mw-experimental-deploy credentials, I can run kubectl exec, shelling into the mw-experimental pod, running /srv/mediawiki/multiversion/MWScript.php manually. So, I made the above-described changes in mw-experimental using the documented process, and then run the maintenance script as follows:

[urbanecm@deploy2002 ~]$ kubectl exec -c mediawiki-pinkllama-app mw-experimental.codfw.pinkllama-779d86d874-znl5n -it -- php /srv/mediawiki/multiversion/MWScript.php Version.php --wiki=dewiki

[..]

@Krinkle Please have a go running P84273 on the deployment host, and let me know if it helps

In T341197: Allow deployers to get a php REPL environment inside the mw-debug pods, @Joe added mw-debug-repl to get shell.php in the debug pods. Given this is the second pod in which running shell.php makes sense, I'm wondering, should we create a more generic wrapper?

(If we end up adding an extra wrapper, we should probably name it consistently, so either rename mw-debug-repl to mw-debug-shell or the other way around)

(Noting that for some reasons [historical ones, maybe?], mw-debug-repl requires root privileges. However, it seems like this task is now possible using deploy credentials. Maybe it should be rewritten in any case?)

In T405688#11302125, @jijiki wrote:

@Krinkle Please have a go running P84273 on the deployment host, and let me know if it helps

This is great. I've used it several times already!

Managing the file system separately from the shell is a bit confusing (one shell on the worker host with writable filesystem but without MW/PHP, another shell from the deploy host in the container with MW/PHP but without writable filesystem where you then use those files created/edited in the other shell). But, it works!

I wonder if the container could be read-write (i.e. ephemeral empty writable layer over top of /srv/mediawiki in the container?). That might pose some challenges with managing outdated state and forgotten/long-running containers, but would make this a bit closer to the level of ease it used to have on mwdebug hosts. Anyway, this is great!