@Anomie, can you run the scripts that need these Perl libraries from inside of a webservice perl5.32 shell container session? Or do they also need kubectl, toolforge-jobs, or similar software that is currently only available on the bastions?

Something that is probably under advertised related to my webservice perl5.32 shell question is that webservice passes extra cli args into the container's shell which allows things like:

$ webservice perl5.32 shell -- perl -v

This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-gnu-thread-multi
(with 48 registered patches, see perl -V for more detail)

Copyright 1987-2021, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

This might be handy if there are things like shell scripts that need to run on the bastion which need some data from a script that needs to run in a container, assuming that data can be passed as captured command output from a subshell.

webservice perl5.32 shell

Just noting what we talked about, that "toolforge-shell" would be a lot more discoverable as a command name for that. 🙂

Or do they also need kubectl, toolforge-jobs, or similar software that is currently only available on the bastions?

I think I could work around the places that use kubectl in the same way I already do in some other places to hit the API directly when in a container. See also T321919 on that topic.

One script does need toolforge-jobs. I don't know what APIs would be needed to work around that one, or if they'd need further access reconfiguration.

Also, Perl has this neat thing where in my local dev environment I can configure the DBI as dbi:Gofer:transport=stream;url=ssh:login.toolforge.org;dsn=DBI:mysql:mysql_read_default_file=/home/anomie/replica.my.cnf;host=$WIKI.$SVC.db.svc.wikimedia.cloud;database=$WIKI_p and it'll ssh into the bastion to connect to the replica DB from there, instead of having to develop directly in toolforge or constantly sync code to be able to test it. Possibly I could set up my ~/.ssh/config with an entry that would use ProxyCommand to ssh→become→webservice shell→sshd -i though...

Possibly I could set up my ~/.ssh/config with an entry that would use ProxyCommand to ssh→become→webservice shell→sshd -i though...

First attempt using ProxyCommand seemed like it almost worked, except that needs to end with connecting to an sshd somewhere (either one already listening on a port or run as sshd -i) and there isn't an sshd on the exec node for it to connect to.

In T360488#9677736, @bd808 wrote:

Dropping this into the "needs discussion" column for cloud-services-team as a blocker to decommissioning the remaining Buster bastions.

I see that, despite this, the main bastion at login.toolforge.org has been "updated" today and my scripts are now broken. 🙁

In T360488#9724584, @Anomie wrote:

In T360488#9677736, @bd808 wrote:

Dropping this into the "needs discussion" column for cloud-services-team as a blocker to decommissioning the remaining Buster bastions.

I see that, despite this, the main bastion at login.toolforge.org has been "updated" today and my scripts are now broken. 🙁

The announce email for switching login.toolforge.org to the new bastion style included a note that login-buster.toolforge.org can still be used by folks who need a "fat" bastion. The messaging could probably have been more proactive with an announce of the intention to change a day or so before actually changing to give more folks a chance to see the announcement, but time machines are in short supply at the moment.

I can't say that having to change various references to login.toolforge.org in my stuff to login-buster.toolforge.org (which seems like it will only work temporarily) seems like a very good solution.