I'm circling back to this after many months. TODOs:

In T434956#12385810, @Jdforrester-WMF wrote:

Is the release/deployment to prod/CI going to be tracked here, or will there be a sub-task?

In T438694#12370174, @elukey wrote:

I found https://github.com/anchore/grype/issues/3604 that seems more or less what grype is missing for our use case, but it doesn't seem a trivial endeavor judging from what the maintainers wrote on github.

What are our concrete use cases for reporting on CVEs from published SBOMs at the moment? They should inform our implementation, and specifically whether we need a service, a library, a CLI, etc. If we anticipate use cases that will need to consume CVE reports from a given image ref without the overhead of pulling down DBs, then a service definitely makes sense (we can cache CVE results quite reliably given the database version and image (index) digest.

@Dbrant I've fixed up and merged @hashar's patch. That should resolve the issue.

Noting that there are a number of images in operations/docker-images/production-images that depend on bullseye (directly and transitively) that cannot be (re)built ATM.

Yeah it appears that I did sneak it into that change https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/58/diffs#5bd615dff78ce55f6c20b15c924188a31bfedff7_81_85

Still seeing this error for newiki following group2 deployment of wmf.13 today, albeit at a low rate.

I don't see any errors following group1 deployment. Deescalating and I'll leave resolution status to y'all.

Thanks all! I will roll the train.

Given the un-readiness of the patch and the rate of errors, I am going to roll back.

Do you all want me to rollback to group0?

Sorry, yes! We have a working setup that uses CAPI and a custom SANs entry for haproxy to give us a publicly addressable API endpoint. We no longer have a requirement for the floating IP.

This is more a Kokkuri feature request than a Blubber one. Have you tried using a .kokkuri:deploy-image job by chance?

Possibly a duplicate of T428954: TypeError: MediaWiki\Extension\WikiLambda\Jobs\CacheTesterResultsJob::storeTestResult(): Argument #10 ($stashedResult) must be of type stdClass, array given, called in /srv/mediawiki/php-1.47.0-wmf.6/extensions/WikiLambda/inclu but filed just in case. @Jdforrester-WMF ?

In T428515#11996847, @bd808 wrote:

@dduvall The tls-server-name: 127.0.0.1 trick in the kubeconfig does not work for the executor access?

See https://phabricator.wikimedia.org/T394838#11058235 for past usage

I tested a full create/destroy cycle via tofu and it worked. Thanks, @Andrew

Thanks, @Andrew! I was able to delete the cluster template via openstack coe cluster template delete just now FWIW. I'll attempt to re-create the template and cluster via https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning today and let you know how it goes.

New error, this time on the zuul-executor (zuul-launcher is fulfilling its part now and creating a namespace and running pod in WMCS).

@Dzahn thanks for fixing the SSH private key issues. On to the next error:

Oh boy, there are many instances of old Blubber usage:

In T321316#11916376, @Jelto wrote:

Thank you very much for the detailed explanation!

In T321316#11911398, @Jelto wrote:

@dduvall blubbers buildkit frontend uses nativ llb directly with T345458: Refactor Blubber's BuildKit frontend gateway to use LLB directly. So my understanding is dockerfile-copy is no longer used/needed by blubber?

I tried to find traces of dockerfile-copyon our gitlab-runners, in buildkit, blubber and codesearch but without success.

@dduvall do you know where to look for dockerfile-copy? Are we still using the image somewhere?

@Dzahn and I debugged the TypeError: Object of type bytes is not JSON serializable error yesterday based on a hunch that the database table's default binary charset was the root cause. We looked at the zuul_buildset and it was indeed created with a DEFAULT CHARSET=binary:

The primary error seems due to a bug in the upstream kubernetes driver. I've submitted a patch.

An initial test in the test/gerrit-ping repo has surfaced a couple of issues.

The zuul-scheduler is now configured with a tenant config (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272970). It's managed via puppet for now meaning that subsequent changes to the project list will require a puppet patch. We can revisit this requirement down the road via the tenant_config_script if this workflow is not flexible enough.

The pipelinelib labels have all been renamed to reflect the capabilities that are required for each pipelinelib stage execution.