The jwt-authorizer microservice now has debian/* files for building a package. The files need review and a package needs to be built and published to apt.wikimedia.org before we can move ahead with changes to the registry auth config.
Description
Description
Details
Details
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| require golang-any to be 1.16 or later in debian/control | repos/releng/jwt-authorizer!3 | dzahn | work/dzahn/build-golang | main | |
| debian: Omit sources from .deb | repos/releng/jwt-authorizer!2 | dduvall | debian/packaging | main | |
| Use 1.0 for debian/source/format | repos/releng/jwt-authorizer!1 | dduvall | debian/packaging | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | dduvall | T308271 Deploy buildkitd to trusted GitLab runners | |||
| Resolved | dduvall | T308501 Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token | |||
| Resolved | Dzahn | T309646 Debianize releng/jwt-authorizer |
Event Timeline
Comment Actions
We will need an upstream tarball that includes a ./vendor/ directory with all the needed artifacts.
"go mod vendor" created such a directory for me when I tried. It will contain a modules.txt with all these:
# github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.0-20210816181553-5444fa50b93d github.com/decred/dcrd/dcrec/secp256k1/v4 # github.com/goccy/go-json v0.9.7 github.com/goccy/go-json github.com/goccy/go-json/internal/decoder github.com/goccy/go-json/internal/encoder github.com/goccy/go-json/internal/encoder/vm github.com/goccy/go-json/internal/encoder/vm_color github.com/goccy/go-json/internal/encoder/vm_color_indent github.com/goccy/go-json/internal/encoder/vm_indent github.com/goccy/go-json/internal/errors github.com/goccy/go-json/internal/runtime # github.com/lestrrat-go/backoff/v2 v2.0.8 github.com/lestrrat-go/backoff/v2 # github.com/lestrrat-go/blackmagic v1.0.0 github.com/lestrrat-go/blackmagic # github.com/lestrrat-go/httpcc v1.0.1 github.com/lestrrat-go/httpcc # github.com/lestrrat-go/iter v1.0.1 github.com/lestrrat-go/iter/arrayiter github.com/lestrrat-go/iter/mapiter # github.com/lestrrat-go/jwx v1.2.24 github.com/lestrrat-go/jwx/jwk github.com/lestrrat-go/jwx/jwt github.com/lestrrat-go/jwx/internal/base64 github.com/lestrrat-go/jwx/internal/ecutil github.com/lestrrat-go/jwx/internal/iter github.com/lestrrat-go/jwx/internal/json github.com/lestrrat-go/jwx/internal/pool github.com/lestrrat-go/jwx/jwa github.com/lestrrat-go/jwx/x25519 github.com/lestrrat-go/jwx github.com/lestrrat-go/jwx/jwe github.com/lestrrat-go/jwx/jws github.com/lestrrat-go/jwx/jwt/internal/types github.com/lestrrat-go/jwx/internal/keyconv github.com/lestrrat-go/jwx/jwe/internal/cipher github.com/lestrrat-go/jwx/jwe/internal/content_crypt github.com/lestrrat-go/jwx/jwe/internal/keyenc github.com/lestrrat-go/jwx/jwe/internal/keygen github.com/lestrrat-go/jwx/jwe/internal/aescbc github.com/lestrrat-go/jwx/jwe/internal/concatkdf # github.com/lestrrat-go/option v1.0.0 github.com/lestrrat-go/option # github.com/pborman/getopt/v2 v2.1.0 github.com/pborman/getopt/v2 # github.com/pkg/errors v0.9.1 github.com/pkg/errors # golang.org/x/crypto v0.0.0-20220427172511-eb4f295cb31f golang.org/x/crypto/curve25519 golang.org/x/crypto/pbkdf2 golang.org/x/crypto/curve25519/internal/field golang.org/x/crypto/ed25519
and then a directory per source, github.com, and golang.org which contain the modules.
https://wikitech.wikimedia.org/wiki/ChartMuseum#Packaging says "The imported upstream tarballs should include the complete vendor directory".
Currently can't build it though even with the vendor dir because:
"dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../jwt-authorizer_1.0.0.orig.tar.{bz2,gz,lzma,xz}"
Comment Actions
Since we are vendoring a ton of Go libs, we don't really use the pristine original release tarball anyway. You can simply change debian/source/format to "1.0", then it will build fine.
Comment Actions
I originally had it doing go mod vendor in the debian/rules file but I've now simply committed the vendor directory to avoid downloading on the build server. Let me know if that's sufficient or if there's a better way.
Done.
Comment Actions
I also went ahead and added dh_auto_install -- --no-source to omit all the vendor files from the .deb.
Comment Actions
I did a fresh git clone of the repo and attempt a build on deneb (USENETWORK=yes DIST=bullseye pdebuild). It looks very promising, starting to set up all the go packages etc...
except for now it still fails at the end with this:
make[1]: Entering directory '/build/jwt-authorizer-1.0.0' dh_auto_build -O--buildsystem=golang -- -ldflags "-X main.Version=1.0.0" cd obj-x86_64-linux-gnu && go install -trimpath -v -p 8 -ldflags "-X main.Version=1.0.0" gitlab.wikimedia.org/repos/releng/jwt-authorizer/cmd/jwt-authorizer src/gitlab.wikimedia.org/repos/releng/jwt-authorizer/cmd/jwt-authorizer/main.go:8:2: cannot find package "io/fs" in any of: /build/jwt-authorizer-1.0.0/obj-x86_64-linux-gnu/src/gitlab.wikimedia.org/repos/releng/jwt-authorizer/vendor/io/fs (vendor tree) /usr/lib/go-1.15/src/io/fs (from $GOROOT) /build/jwt-authorizer-1.0.0/obj-x86_64-linux-gnu/src/io/fs (from $GOPATH) dh_auto_build: error: cd obj-x86_64-linux-gnu && go install -trimpath -v -p 8 -ldflags "-X main.Version=1.0.0" gitlab.wikimedia.org/repos/releng/jwt-authorizer/cmd/jwt-authorizer returned exit code 1 make[1]: *** [debian/rules:10: override_dh_auto_build] Error 25 make[1]: Leaving directory '/build/jwt-authorizer-1.0.0' make: *** [debian/rules:7: build] Error 2 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package
Comment Actions
Might be like this, where they needed golang 1.16. We have 1.18 in our go.mod but it seems to use 1.15.
go env
..
GOROOT="/usr/lib/go-1.15"
Comment Actions
Yea, indeed we needed golang >=1.16. Both on deneb (buster) and build2001 (bullseye) we only have 1.15 but as Moritz pointed out a newer version is in bullseye-backports.
It worked with DIST=bullseye-backports pdebuild which pulls in golang 1.17.
jwt-authorizer_1.0.0-1_amd64.deb
is in [build2001:/var/cache/pbuilder/result/bullseye-amd64
We need to put 1.16 in debian/control for the next build.
Comment Actions
uploaded and imported on apt1001 with:
[apt1001:/srv/wikimedia/incoming] $ sudo -i reprepro --ignore=wrongdistribution -C main include bullseye-wikimedia /home/dzahn/jwt-authorizer_1.0.0-1_amd64.changes
Have to force bullseye-wikimedia with --ignore=wrongdistribution because the .changes file has: Distribution: unstable.
After an apt-get update it's now available on a gitlab machine:
[gitlab-runner1001:~] $ apt-cache show jwt-authorizer Package: jwt-authorizer Version: 1.0.0-1 Architecture: amd64 Maintainer: Wikimedia Foundation Release Engineering <releng@wikimedia.org> Installed-Size: 7145 Depends: libc6 (>= 2.4) Built-Using: golang-1.17 (= 1.17.10-1~bpo11+1) Homepage: https://gitlab.wikimedia.org/repos/releng/jwt-authorizer Priority: optional Section: devel Filename: pool/main/j/jwt-authorizer/jwt-authorizer_1.0.0-1_amd64.deb Size: 2230456 SHA256: 956cf7701a62c6b8768a7fe28c2e2a130f0ed281cf9f9dfb8a0ce8611c716e0b SHA1: 9fd1ec508b2d431a413d9b2815f61da6c38dc527 MD5sum: dc290a76c5dea361c5eda459c2ea085a Description: A small stateless HTTP server that validates JSON Web Tokens for arbitrary service integration. Description-md5: 882c11f64c5a73bbe6574fb522eaf2ae
Comment Actions
my first merge request https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/3
Comment Actions
Thanks @Dzahn! Can you verify that the package is installable on the docker registry hosts?
Comment Actions
@dduvall It's not installable on production docker registry hosts because those are on buster.
Comment Actions
@dduvall But now it is.. after I imported it for buster-wikimedia as well, again using the "ignore wrong distro" option.
simulated (-s) install:
[registry1003:~] $ sudo apt-get install -s jwt-authorizer Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libnginx-mod-http-auth-pam libnginx-mod-http-dav-ext libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-subs-filter libnginx-mod-http-upstream-fair libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-image-4.19.0-14-amd64 linux-image-4.19.0-16-amd64 Use 'sudo apt autoremove' to remove them. The following NEW packages will be installed: jwt-authorizer 0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded. Inst jwt-authorizer (1.0.0-1 Wikimedia:10/buster-wikimedia [amd64]) Conf jwt-authorizer (1.0.0-1 Wikimedia:10/buster-wikimedia [amd64])
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2022-11-30T22:09:13Z] <mutante> registry2004 - apt-get update; apt-install jwt-authorizer (after apt1001: sudo -i reprepro --ignore=wrongdistribution -C main include buster-wikimedia /home/jelto/jwt-authorizer_1.1.0-1_amd64.changes) - T309646