Page MenuHomePhabricator

[jobs-cli,jobs-api] Allow using file logs with build service images
Closed, ResolvedPublicFeature

Description

Running a test script on Toolforge as follows:

toolforge jobs run test1 --image tool-pywikibot/pywikibot-scripts-stable:latest --command "echo Hello, World!"

After a few seconds, the job shows as "Completed". I then expect to find a file test1.out in my tool's home directory containing the text "Hello, World!" and an empty file test1.err. Neither of these files is created. (They are created as expected when using a different --image).

Specifying the output files with --filelog-stdout and --filelog-stderr doesn't change the result.

Details

TitleReferenceAuthorSource BranchDest Branch
d/changelog: bump to 16.0.2repos/cloud/toolforge/jobs-cli!15dcarobump_to_16.0.2main
run: add filelog to buildservice if passedrepos/cloud/toolforge/jobs-cli!12dcarowrap_buildservicemain
[command] wrap buildservice with a shellrepos/cloud/toolforge/jobs-api!61dcarowrap_buildservicemain
jobs-api: bump to 0.0.254-20240108102443-2737a5d0repos/cloud/toolforge/toolforge-deploy!164project_1317_bot_df3177307bed93c3f34e421e26c86e38bump_jobs-apimain
Allow enabling filelog with buildpack imagesrepos/cloud/toolforge/jobs-cli!8taavimain-I50f4dcfac0ebc5ae708fb0d4e7e8131e8503aebdmain
api: allow file logging on build service jobs with all mountsrepos/cloud/toolforge/jobs-api!51taavimain-I581bae903659a41a746e2064646e3010cec324b5main
command: Always use absolute paths for file logsrepos/cloud/toolforge/jobs-api!50taavimain-I8fa677536982cd7134cdc4de01ac490207004269main
Customize query in GitLab

Event Timeline

taavi triaged this task as High priority.
taavi renamed this task from Pywikibot image for Toolforge jobs does not capture output to Allow using file logs with build service images.Dec 15 2023, 5:59 PM
taavi lowered the priority of this task from High to Medium.
taavi removed a project: Pywikibot.
taavi removed a subscriber: pywikibot-bugs-list.
taavi changed the subtype of this task from "Bug Report" to "Feature Request".Dec 15 2023, 7:49 PM

The internal log storage does function with the pywikibot image and other buildpack images, but I'll also add support for file logging on an opt-in basis until we get better logging systems available for all of the tools.

taavi changed the task status from Open to In Progress.Dec 15 2023, 7:56 PM
taavi moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 02) board.

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/164

jobs-api: bump to 0.0.254-20240108102443-2737a5d0

The shell wrapper thing still needs some thinking, so moving back to next up until I have some time to work on this.

taavi removed taavi as the assignee of this task.Feb 6 2024, 1:54 PM
taavi subscribed.

Not working on this ATM, feel free to take this.

bd808 changed the task status from In Progress to Open.Feb 12 2024, 11:08 PM
bd808 removed a project: Patch-For-Review.

I thought this was closed xd, that's why I opened a new one

dcaro changed the task status from Open to In Progress.Feb 19 2024, 4:10 PM
dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 05) board.
dcaro renamed this task from Allow using file logs with build service images to [jobs-cli,jobs-api] Allow using file logs with build service images.Mar 5 2024, 3:58 PM
dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 07) board.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Job_logs still says "Jobs using build service images (including the prebuilt Pywikibot image) can currently only use internal logging.". Is that no longer true?

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Job_logs still says "Jobs using build service images (including the prebuilt Pywikibot image) can currently only use internal logging.". Is that no longer true?

Updated, thanks for the notice.