toolforge lima-kilo: toolforge-jobs fails to run because it can't create log files
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Jun 5 2023, 3:17 PM

Description

Using a fresh install of lima-kilo, this happens:

arturo@nostromo:~/git/wmf/cloud/toolforge/jobs-framework-cli $ toolforge-jobs run x --command ./test-cmd.sh --image bullseye
WARNING: not running as the tool account? Likely to fail. Perhaps you forgot `become <tool>`?
arturo@nostromo:~/git/wmf/cloud/toolforge/jobs-framework-cli $ toolforge-jobs show x
WARNING: not running as the tool account? Likely to fail. Perhaps you forgot `become <tool>`?
+-------------+------------------------------------------------------------------+
| Job name:   | x                                                                |
+-------------+------------------------------------------------------------------+
| Command:    | ./test-cmd.sh                                                    |
+-------------+------------------------------------------------------------------+
| Job type:   | normal                                                           |
+-------------+------------------------------------------------------------------+
| Image:      | bullseye                                                         |
+-------------+------------------------------------------------------------------+
| File log:   | yes                                                              |
+-------------+------------------------------------------------------------------+
| Output log: | x.out                                                            |
+-------------+------------------------------------------------------------------+
| Error log:  | x.err                                                            |
+-------------+------------------------------------------------------------------+
| Emails:     | none                                                             |
+-------------+------------------------------------------------------------------+
| Resources:  | default                                                          |
+-------------+------------------------------------------------------------------+
| Retry:      | no                                                               |
+-------------+------------------------------------------------------------------+
| Status:     | Failed                                                           |
+-------------+------------------------------------------------------------------+
| Hints:      | Last run at 2023-06-05T15:10:45Z. Pod in 'Failed' phase. State   |
|             | 'terminated'. Reason 'Error'. Started at '2023-06-05T15:10:56Z'. |
|             | Finished at '2023-06-05T15:10:56Z'. Exit code '2'.               |
+-------------+------------------------------------------------------------------+
arturo@nostromo:~/git/wmf/cloud/toolforge/jobs-framework-cli $ kubectl -n tool-tf-test get all
NAME          READY   STATUS   RESTARTS   AGE
pod/x-mkzv4   0/1     Error    0          19s

NAME          COMPLETIONS   DURATION   AGE
job.batch/x   0/1           19s        19s
arturo@nostromo:~/git/wmf/cloud/toolforge/jobs-framework-cli $ kubectl -n tool-tf-test logs pod/x-mkzv4
/bin/sh: 1: cannot create x.out: Permission denied

I'm pretty sure this is a regression. This worked before.

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T220020 Action items and work for retro 20190403
Resolved	dcaro	T220053 Build a dev/testing environment for webservice that would make it easier to get people involved in fixes
Resolved	• aborrero	T327254 WMCS FY22/23 Q3: next steps in grid engine deprecation
Resolved	• aborrero	T326789 Toolforge: improve local kubernetes development setup
Resolved	dcaro	T338153 toolforge lima-kilo: toolforge-jobs fails to run because it can't create log files

Event Timeline

• aborrero created this task.Jun 5 2023, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2023, 3:17 PM

• aborrero added a parent task: T326789: Toolforge: improve local kubernetes development setup.Jun 5 2023, 3:21 PM

taavi edited projects, added Toolforge Jobs framework; removed Cloud-Services.Jun 6 2023, 3:30 PM

• aborrero moved this task from Backlog to Next on the User-aborrero board.Jun 30 2023, 3:25 PM

• aborrero mentioned this in T346738: [lima-kilo,jobs-api,infra] tool jobs run as root user in lima-kilo environment.Sep 19 2023, 4:01 PM

• aborrero moved this task from Next to Backlog on the User-aborrero board.Feb 2 2024, 4:55 PM

This seems not to be an issue anymore, just tested:

local.tf-test@lima-lima-kilo:~$ toolforge jobs run x --command ./test-cmd.sh --image bullseye
...
local.tf-test@lima-lima-kilo:~$ ls -la
total 36
drwxrwxrwx 5 local.tf-test local.tf-test 4096 Mar 11 12:55 .
drwxr-xr-x 5 dcaro         dcaro         4096 Mar  8 10:26 ..
drwxr-xr-x 3 local.tf-test local.tf-test 4096 Mar 11 12:55 .kube
drwxr-xr-x 2 local.tf-test local.tf-test 4096 Mar  8 10:32 .toolskube
drwxrwxrwx 2 local.tf-test local.tf-test 4096 Mar  8 10:26 public_html
-rwxr-xr-x 1 local.tf-test local.tf-test   37 Mar  8 10:26 sleep40.sh
-rwxr-xr-x 1 local.tf-test local.tf-test   50 Mar  8 10:26 test-cmd.sh
-rw-r--r-- 1 root          root             7 Mar 11 12:55 x.err
-rw-r--r-- 1 root          root             7 Mar 11 12:55 x.out

It's interesting that they are created as root though, we might be missing some restrictions in lima-kilo, in toolforge per se it has the user's user.

dcaro closed this task as Resolved.Mar 11 2024, 1:20 PM

dcaro claimed this task.

In T338153#9619793, @dcaro wrote:

It's interesting that they are created as root though, we might be missing some restrictions in lima-kilo, in toolforge per se it has the user's user.

toolforge lima-kilo: toolforge-jobs fails to run because it can't create log filesClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

toolforge lima-kilo: toolforge-jobs fails to run because it can't create log files
Closed, ResolvedPublic
Actions

Related Objects
Search...