Details

Maniphest Tasks

T187372: Blubber should implement a better file permissions convention

Reviewers

thcipriani
hashar
demon

Group Reviewers

Release-Engineering-Team

Commits

rGBLBR47526283fea7: Fix application files/runtime permissions scheme

Patch without arc

git checkout -b D984 && curl -L https://phabricator.wikimedia.org/D984?download=true | git apply

Summary

Introduces new lives configuration that provides the name/UID/GID of
the user that will own application files and installed dependencies.
This new configuration is distinct from runs in that the former
determines application file location ownership and the latter now only
determines runtime process ownership. Default configuration has also
been introduced for both config sections.

In addition to the new configuration, a new build.CopyAs instruction
has been introduced that ensures correct UID/GID ownership of files
copied into the container image, and all unqualified build.Copy
instructions are wrapped by the new build.CopyAs instruction using the
UID/GID appropriate for the current build phase. A new build.User
instruction is also introduced and injected into the build at the start
of certain phases to enforce ownership of build.Run processes.

This effective process/file ownership model is:

PhasePrivileged - "root"
PhasePrivilegedDropped - lives.as
PhasePreInstall - lives.as
PhaseInstall - lives.as
PhasePostInstall - runs.as

Fixes T187372

Test Plan

Run go test ./....

Diff Detail

Repository

rGBLBR Blubber

Branch

fix/permissions-scheme

Lint

Lint Passed

Unit

Tests Passed

Event Timeline

dduvall created this revision.Feb 22 2018, 12:20 AM

Restricted Application added a reviewer: Release-Engineering-Team. · View Herald TranscriptFeb 22 2018, 12:20 AM

Restricted Application added a project: Release-Engineering-Team. · View Herald Transcript

dduvall requested review of this revision.Feb 22 2018, 12:20 AM

Harbormaster completed remote builds in Restricted Buildable.Feb 22 2018, 12:21 AM

Fixed lint errors, default config, and wrote a more coherent commit message

Harbormaster completed remote builds in Restricted Buildable.Feb 22 2018, 8:32 PM

Fixed redundant USER root output

Harbormaster completed remote builds in Restricted Buildable.Feb 22 2018, 8:35 PM

PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestAptConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestAptConfigInstructions
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestAptConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestArtifactsConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestArtifactsConfigInstructions
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestArtifactsConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestCommonConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestCommonConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestFlagOverwrite
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfigInstructionsNoDependencies
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfigInstructionsNonProduction
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfigInstructionsProduction
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfigInstructionsEnvironmentOnly
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestNodeConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestResolveIncludesPreventsInfiniteRecursion
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestMultiLevelIncludes
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestMultiIncludes
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestRunsConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestRunsHomeWithUser
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestRunsHomeWithoutUser
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestRunsConfigInstructions
PASS   10ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestRunsConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestIsValidationError
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestVariantConfig
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestVariantDependencies
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestVariantLoops
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestVariantConfigInstructions
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::TestVariantConfigValidation
PASS   <1ms★  Go::Test::phabricator.wikimedia.org::source::blubber::config::ExampleResolveIncludes

dduvall added a child revision: D990: Simplify Docker instruction compilation.Feb 28 2018, 12:46 AM

Tests pass, 2 users are created, but I'm having a problem with dropped privileges using mathiod's blubber.yaml

FROM docker-registry.wikimedia.org/nodejs-devel
USER "root"
RUN apt-get update && apt-get install -y "librsvg2-2" "librsvg2-dev" "git" "python-pkgconfig" "build-essential" && rm -rf /var/lib/apt/lists/*
RUN groupadd -o -g "65533" -r "somebody" && useradd -o -m -d "/home/somebody" -r -g "somebody" -u "65533" "somebody" && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
RUN groupadd -o -g "666" -r "runuser" && useradd -o -m -d "/home/runuser" -r -g "runuser" -u "666" "runuser"
USER "somebody"
ENV HOME="/home/runuser"
ENV APP_BASE_PATH="/srv/service" LINK="g++"
WORKDIR /srv/app
COPY --chown=65533:65533 ["package.json", "/opt/lib"]
RUN cd "/opt/lib" && npm install
COPY --chown=65533:65533 [".", "."]
USER "runuser"
ENV NODE_ENV="" NODE_PATH="/opt/lib/node_modules" PATH="/opt/lib/node_modules/.bin:${PATH}"
ENTRYPOINT ["npm", "test"]
LABEL blubber.variant="test" blubber.version="0.2.0+c3fdb20"

Is the output dockerfile; however, there is a problem with the "somebody" user using the "/home/runuser" $HOME and then running npm install. I get:

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: librsvg@^0.7.0 (node_modules/librsvg):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Error: EACCES: permission denied, mkdir '/home/runuser/.npm'
npm ERR! Linux 4.14.0-3-amd64      
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install"                             
npm ERR! node v6.12.0  
npm ERR! npm  v3.10.10                        
npm ERR! path /home/runuser/.npm               
npm ERR! code EACCES
npm ERR! errno -13                                                                                                                            
npm ERR! syscall mkdir                                                                                                                                                                                                                    
npm ERR! Error: EACCES: permission denied, mkdir '/home/runuser/.npm'                                       
npm ERR!     at Error (native)
npm ERR!  { Error: EACCES: permission denied, mkdir '/home/runuser/.npm'
npm ERR!     at Error (native)             
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',                           
npm ERR!   syscall: 'mkdir',    
npm ERR!   path: '/home/runuser/.npm',
npm ERR!   parent: 'mathoid' }
npm ERR!                                                                                   
npm ERR! Please try running this command again as root/Administrator.
                                                            
npm ERR! Please include the following file with any support request:
npm ERR!     /opt/lib/npm-debug.log

In D984#19823, @thcipriani wrote:

Is the output dockerfile; however, there is a problem with the "somebody" user using the "/home/runuser" $HOME and then running npm install. I get:

Ah ha! Yes, the setting of HOME for the current user will probably have to be moved into config/variant.go, alongside the setting of USER.

Moved home directory setting into config/variant.go alongside build.User instruction

hashar mentioned this in T177499: On CI, upgrade docker-ce from 17.06.2 to 17.12.1.Mar 1 2018, 5:10 PM

Works well, I like the macros. Took me a while to get my head around what was wrong with the current mathoid config. I wish there was a way to warn if there were extra fields in the yaml file as in the case of including an in field for runs.

config/user.go
7	maybe should be uint16

This revision is now accepted and ready to land.Mar 5 2018, 4:46 PM

dduvall added inline comments.Mar 5 2018, 9:21 PM

config/user.go
7	I think technically 32bit uids are, though not recommended, allowed.

thcipriani added inline comments.Mar 5 2018, 9:24 PM

config/user.go
7	true enough, seems to work on my computer. Although uint64 is definitely too big: sudo useradd -o -m -d "/home/reallytoohigh" -u "18446744073709551615" "reallytoohigh" gives me: `useradd: invalid user ID '18446744073709551615'` Just a nitpick, feel free to ignore.

Closed by commit rGBLBR47526283fea7: Fix application files/runtime permissions scheme (authored by dduvall). · Explain WhyMar 5 2018, 9:25 PM

This revision was automatically updated to reflect the committed changes.

dduvall mentioned this in D1002: Fix ownership on artifact copies.Mar 9 2018, 11:56 PM

dduvall mentioned this in rGBLBR50c5793952a7: Fix ownership on artifact copies.Mar 22 2018, 6:00 PM

Fix application files/runtime permissions scheme
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 2603

blubber.example.yaml

build/instructions.go

build/instructions_test.go

build/macros.go

build/macros_test.go

config/common.go

config/lives.go

config/lives_test.go

config/reader.go

config/runs.go

config/runs_test.go

config/user.go

config/variant.go

config/variant_test.go

docker/compiler.go

docker/instructions.go

docker/instructions_test.go

Fix application files/runtime permissions schemeClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 2603

blubber.example.yaml

build/instructions.go

build/instructions_test.go

build/macros.go

build/macros_test.go

config/common.go

config/lives.go

config/lives_test.go

config/reader.go

config/runs.go

config/runs_test.go

config/user.go

config/variant.go

config/variant_test.go

docker/compiler.go

docker/instructions.go

docker/instructions_test.go

Fix application files/runtime permissions scheme
ClosedPublic
Actions

Revision Contents
Changeset List