Deployers should always run scap within a server side tmux/screen

I think if this is actually the best solution then some wrapper should be created for scap that makes the tmux session happen automagically instead of making it a nag workflow that folks have to remember to do manually.

jiji opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/266

Prevent scap from running outside of a screen or a tmux

There is a tmux/screen check for scap stage-train, but nothing else. This could be factored out to cover other scap subcommands.
Suggestions:

• scap backport
• scap deploy
• scap deploy-promote
• scap sync-*
• scap train
• scap stage-train
• scap lock

Applying the tmux/screen check for all scap subcommands (especially those unrelated to deployment) is definitely undesirable.

I might have used screen back in the old days (like in 2005 or so) and might have tried screen / tmux at some point in the early 2010's when I have started working for the WMF. Overall, my sole use case would have been for running deployment tools (Trebuchet, copy pasting commands, then scap). After 20+ years I have never felt like I should invest time in learning the magic keyboard shortcuts to navigate through them. I don't think I had more than a couple network interruptions while doing a deployment, and thus they are not in my habits.

Ok that is not an excuse :)

Would it be possible for scap to replace itself with the same invocation but wrapped in screen/tmux? Maybe something such as:

if 'TMUX' not in os.environ or 'STY' not in os.environ:
    multiplexer = shutil.which( 'tmux' ) or shutil.which( 'screen' )
    if multiplexer is None:
        log.info("Restarting with %s" % multiplexer)
        os.execvpe( sys.argv[0], sys.argv[1:], os.environ )

As a side note scap might well be interrupted with a ⌃+C which might leave the ongoing helm deployment in a weird state but that seems to be for scap to learn about such unexpected state which is T361720.

For this use case the only keybinding you would need to know is how to exit once your run is done, which you would do the same way you exit a shell, with exit or ^+d.

I agree that switching to the multiplexer should happen transparently for the deployer, though they would still need to learn how to reattach a session, which is still way better than learning how to unbreak helm for something as avoidable as a connection loss.

jiji closed https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/266

Prevent scap from running outside of a screen or a tmux

jiji opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/271

utils.py: add method to check if scap is running in a screen tmux

In T361724#9690204, @dancy wrote:

There is a tmux/screen check for scap stage-train, but nothing else. This could be factored out to cover other scap subcommands.
Suggestions:

• scap backport
• scap deploy
• scap deploy-promote
• scap sync-*
• scap train
• scap stage-train
• scap lock

Applying the tmux/screen check for all scap subcommands (especially those unrelated to deployment) is definitely undesirable.

I see your point, I rushed a bit into it. I updated the MR, and limited it to just introduce utils.is_shell_durable(). It would be great (and faster) if someone with better scap code knowledge than me, would inject it in the appropriate subcommands.

In T361724#9696779, @Clement_Goubert wrote:

I agree that switching to the multiplexer should happen transparently for the deployer, though they would still need to learn how to reattach a session, which is still way better than learning how to unbreak helm for something as avoidable as a connection loss.

While it would be convenient, I think it will be confusing overall, thus I reckon we should just warn users to restart what they were doing, via a screen or tmux. The learning curve is quite small, and picking up the habit of starting a screen or a tmux, can't be a terrible idea :)

@dancy it would be great if someone could finish this soon. While scap now does have an option to mitigate potential helm hiccups, I think we should add it to the mix nevertheless

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/412

cli: require_tty_multiplexer support

Change #1065271 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Enable require_tty_multiplexer

https://gerrit.wikimedia.org/r/1065271

thcipriani merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/412

cli: require_tty_multiplexer support

dancy closed https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/271

utils.py: add method to check if scap is running in a screen tmux

Change #1065271 merged by Dzahn:

[operations/puppet@production] scap.cfg.erb: Enable require_tty_multiplexer

https://gerrit.wikimedia.org/r/1065271

Change #1066810 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Enable require_terminal_multiplexer

https://gerrit.wikimedia.org/r/1066810

Change #1066810 merged by Dzahn:

[operations/puppet@production] scap.cfg.erb: Enable require_terminal_multiplexer

https://gerrit.wikimedia.org/r/1066810

Change #1066821 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Disable require_terminal_multiplexer

https://gerrit.wikimedia.org/r/1066821

Change #1066821 merged by Dzahn:

[operations/puppet@production] scap.cfg.erb: Disable require_terminal_multiplexer

https://gerrit.wikimedia.org/r/1066821

I just ran into this – no error, no warning, I just deployed away without screen (until a ssh disconnect broke things). Is this expected? Jayme thought it shouldn't happen any more. Ah, right, feature enforcement not enabled yet.

After discussion in a Scap (SpiderPig 🕸️) demo meeting, I'm noting here that this will become much less relevant over time as scap operations move into the web interface. (Not declining this task at the moment since there does seem to be some ongoing work.)

Just to confirm, the terminal-multiplexer check feature works as expected, and all that remains is docs updates and comms to ops@lists before it's reenabled, correct? (i.e., the description remains accurate)

In T361724#11604248, @Scott_French wrote:

Just to confirm, the terminal-multiplexer check feature works as expected, and all that remains is docs updates and comms to ops@lists before it's reenabled, correct? (i.e., the description remains accurate)

Correct.