Files created in a buildservice shell evaporate on logout
Closed, InvalidPublicBUG REPORT
Actions

Assigned To

None

Authored By

	Ijon
	Dec 5 2025, 1:07 AM

Description

Steps to replicate the issue (include links if applicable):

On toolforge, become gutensearch
run toolforge webservice buildservice shell
run touch xyzzy
run ls -l xyzzy
smile contentedly
exit
run toolforge webservice buildservice shell
run ls -l xyzzy
gasp!

What happens?:

tools.gutensearch@tools-bastion-15:~$ toolforge webservice buildservice shell
I have no name!@shell-1764896542:/workspace$ touch xyzzy
I have no name!@shell-1764896542:/workspace$ ls -l xyzzy
-rw-r--r-- 1 57211 57211 0 Dec 5 01:02 xyzzy
I have no name!@shell-1764896542:/workspace$ (ctrl+d)
logout
tools.gutensearch@tools-bastion-15:~$ toolforge webservice buildservice shell
I have no name!@shell-1764896572:/workspace$ ls -l xyzzy
ls: cannot access 'xyzzy': No such file or directory
I have no name!@shell-1764896572:/workspace$

What should have happened instead?:

xyzzy should have stuck around!

Other information (browser name/version, screenshots, etc.):

Event Timeline

Ijon created this task.Dec 5 2025, 1:07 AM

Restricted Application added a project: cloud-services-team. · View Herald TranscriptDec 5 2025, 1:07 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Ijon updated the task description. (Show Details)Dec 5 2025, 1:11 AM

This is totally confusing if you are used to working with the legacy containers rather than the build service created containers, but the default director is not the tool's NFS mounted $HOME. You need to cd $TOOL_DATA_DIR to get to the NFS backed directory.

bd808@tools-bastion-14.tools.eqiad1:~$ sudo become gutensearch
tools.gutensearch@tools-bastion-14:~$ toolforge webservice buildservice shell --mount=all
I have no name!@shell-1764897530:/workspace$ pwd
/workspace
I have no name!@shell-1764897530:/workspace$ ls
AGENTS.md             Procfile   config     log                     storage
API_DOCUMENTATION.md  README.md  config.ru  pg_catalog_example.csv  test
Dockerfile            Rakefile   db         public                  tmp
Gemfile               app        doc        script                  vendor
Gemfile.lock          bin        lib        spec
I have no name!@shell-1764897530:/workspace$ cd $TOOL_DATA_DIR
I have no name!@shell-1764897530:~$ pwd
/data/project/gutensearch
I have no name!@shell-1764897530:~$ ls
catalog      ingestion_shell.sh  replica.my.cnf    shell.sh
gutensearch  logs                service.manifest  update_and_deploy.sh

I realize that. But aren't files created in the /workspace also supposed to last past the current shell, so long as the image is around?

I tried to keep the scenario minimal, but if the above is expected behavior, then the even more confusing behavior I encountered was files disappearing from NFS when created from within the buildservice shell and through a symlink created in it, pointing to NFS. However, I was not able to reproduce that behavior now, so never mind. I'll re-open if I can reproduce it again.

In T411832#11435332, @Ijon wrote:

I realize that. But aren't files created in the /workspace also supposed to last past the current shell, so long as the image is around?

They stay in the ephemeral storage of the running instance of the image, but no they are not written back to the stored image itself. Each webservice shell invocation is a separate Kubernetes Pod containing an instance of the requested container. When Kubernetes reaps that Pod the ephemeral storage layer is discarded.

I tried to keep the scenario minimal, but if the above is expected behavior, then the even more confusing behavior I encountered was files disappearing from NFS when created from within the buildservice shell and through a symlink created in it, pointing to NFS. However, I was not able to reproduce that behavior now, so never mind. I'll re-open if I can reproduce it again.

This is along the lines of what I thought you were reporting on irc, and yes it sounds like unexpected behavior to me. There is some amount of caching and differed write synchronization in NFS itself, but that should not typically result in files not eventually (order of seconds at worst) being written to disk.

Files created in a buildservice shell evaporate on logoutClosed, InvalidPublicBUG REPORTActions

Description

Event Timeline

Files created in a buildservice shell evaporate on logout
Closed, InvalidPublicBUG REPORT
Actions