Page MenuHomePhabricator
Create Task
Maniphest T287130

Container image lifecycle management
Open, MediumPublic
Actions

Description

With our production software continuing to move to containers running on Kubernetes, we need to build out the tools to help us manage the lifecycle of container images. In particular, an image catalog service will record all versions of all images built and published in the registry.

The image catalog will record the build's dependencies (other Docker images as well as Debian packages) when the build is registered by docker-pkg or blubber, and will integrate with tools like Clair and Debmonitor for managing known security vulnerabilities.

The image catalog will also periodically check to see what images (and which versions) are currently running in production, in order to track images in need of an update (especially when a security update is available for one of its dependencies).

For more details see the original design doc.

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

RLazarus triaged this task as Medium priority.Jul 21 2021, 10:32 PM
RLazarus created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2021, 10:32 PM
gerritbot added a comment.Jul 21 2021, 10:37 PM

Change 706048 had a related patch set uploaded (by RLazarus; author: RLazarus):

[integration/config@master] zuul: Add a new project for operations/docker-images/imagecatalog

https://gerrit.wikimedia.org/r/706048

gerritbot added a project: Patch-For-Review.Jul 21 2021, 10:37 PM
gerritbot added a comment.Jul 22 2021, 12:40 PM

Change 706048 merged by jenkins-bot:

[integration/config@master] Zuul: [operations/docker-images/imagecatalog] add tox-docker

https://gerrit.wikimedia.org/r/706048

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2021, 1:10 PM
gerritbot added a comment.Sep 25 2021, 2:44 AM

Change 723663 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/imagecatalog@master] Minimal version of the image catalog

https://gerrit.wikimedia.org/r/723663

gerritbot added a project: Patch-For-Review.Sep 25 2021, 2:44 AM
gerritbot added a comment.Oct 21 2021, 7:01 AM

Change 723663 merged by jenkins-bot:

[operations/docker-images/imagecatalog@master] Minimal version of the image catalog

https://gerrit.wikimedia.org/r/723663

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2021, 7:10 AM
gerritbot added a comment.Dec 6 2021, 6:42 PM

Change 742574 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Install and configure OCI image catalog on deploy hosts

https://gerrit.wikimedia.org/r/742574

gerritbot added a project: Patch-For-Review.Dec 6 2021, 6:42 PM
gerritbot added a comment.Dec 8 2021, 9:12 AM

Change 745196 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] RBAC: Add ClusterRole and ClusterRoleBinding for imagecatalog

https://gerrit.wikimedia.org/r/745196

gerritbot added a comment.Dec 8 2021, 10:18 AM

Change 745208 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[labs/private@master] Add imagecatalog user to main and ml

https://gerrit.wikimedia.org/r/745208

gerritbot added a comment.Dec 8 2021, 10:18 AM

Change 745208 merged by JMeybohm:

[labs/private@master] Add imagecatalog user to main and ml

https://gerrit.wikimedia.org/r/745208

JMeybohm mentioned this in rLPRI8c7747a2238b: Add imagecatalog user to main and ml.Dec 8 2021, 10:25 AM
gerritbot added a comment.Dec 8 2021, 12:05 PM

Change 745202 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Use dedicated imagecatalog kubernetes user

https://gerrit.wikimedia.org/r/745202

gerritbot added a comment.Dec 15 2021, 5:30 PM

Change 745196 merged by jenkins-bot:

[operations/deployment-charts@master] RBAC: Add ClusterRole and ClusterRoleBinding for imagecatalog

https://gerrit.wikimedia.org/r/745196

Stashbot added a comment.Dec 15 2021, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-15T17:44:32Z] <jayme> deployed imagecatalog RBAC rules to all k8s clusters - T287130

gerritbot added a comment.Dec 15 2021, 5:45 PM

Change 745202 abandoned by JMeybohm:

[operations/puppet@production] Use dedicated imagecatalog kubernetes user

Reason:

Merged into I2ebf9c25d31334cbb6aad5e4de9293b6ab1d5cdc

https://gerrit.wikimedia.org/r/745202

gerritbot added a comment.Dec 15 2021, 6:30 PM

Change 742574 merged by RLazarus:

[operations/puppet@production] imagecatalog: Install and configure OCI image catalog on deploy hosts

https://gerrit.wikimedia.org/r/742574

gerritbot added a comment.Dec 15 2021, 6:33 PM

Change 747566 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Fix outdated TODO comment

https://gerrit.wikimedia.org/r/747566

gerritbot added a comment.Dec 15 2021, 6:36 PM

Change 747566 merged by RLazarus:

[operations/puppet@production] imagecatalog: Fix outdated TODO comment

https://gerrit.wikimedia.org/r/747566

gerritbot added a comment.Dec 15 2021, 9:02 PM

Change 747610 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: 0770, not 0440, so the init command can create a DB

https://gerrit.wikimedia.org/r/747610

gerritbot added a comment.Dec 15 2021, 9:04 PM

Change 747610 merged by RLazarus:

[operations/puppet@production] imagecatalog: 0770, not 0440, so the init command can create a DB

https://gerrit.wikimedia.org/r/747610

gerritbot added a comment.Dec 16 2021, 6:20 AM

Change 747683 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/imagecatalog@master] Use the Kubernetes config API as it was in v7.0.0 (buster)

https://gerrit.wikimedia.org/r/747683

gerritbot added a comment.Dec 16 2021, 6:43 AM

Change 747685 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Pass cluster names along with config paths

https://gerrit.wikimedia.org/r/747685

gerritbot added a comment.Dec 16 2021, 5:19 PM

Change 747881 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/imagecatalog@master] Add a pod_name column to ActiveContainerImage

https://gerrit.wikimedia.org/r/747881

gerritbot added a comment.Dec 17 2021, 6:41 PM

Change 747683 merged by jenkins-bot:

[operations/docker-images/imagecatalog@master] Use the Kubernetes config API as it was in v7.0.0 (buster)

https://gerrit.wikimedia.org/r/747683

gerritbot added a comment.Dec 18 2021, 4:02 AM

Change 748232 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/imagecatalog@master] Fix --cluster command line parsing and add tests

https://gerrit.wikimedia.org/r/748232

gerritbot added a comment.Dec 20 2021, 7:17 PM

Change 747881 merged by jenkins-bot:

[operations/docker-images/imagecatalog@master] Add a pod_name column to ActiveContainerImage

https://gerrit.wikimedia.org/r/747881

gerritbot added a comment.Dec 20 2021, 7:20 PM

Change 748232 merged by jenkins-bot:

[operations/docker-images/imagecatalog@master] Fix --clusters command line parsing and add tests

https://gerrit.wikimedia.org/r/748232

gerritbot added a comment.Dec 20 2021, 8:25 PM

Change 747685 merged by RLazarus:

[operations/puppet@production] imagecatalog: Pass cluster names along with config paths

https://gerrit.wikimedia.org/r/747685

gerritbot added a comment.Dec 20 2021, 8:30 PM

Change 748799 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Puppet spelling correction, s/str/String/

https://gerrit.wikimedia.org/r/748799

gerritbot added a comment.Dec 21 2021, 12:00 AM

Change 748799 merged by RLazarus:

[operations/puppet@production] imagecatalog: Pass cluster names along with config paths

https://gerrit.wikimedia.org/r/748799

gerritbot added a comment.Dec 21 2021, 1:28 AM

Change 748876 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Add an hourly systemd timer to scan for what's currently running

https://gerrit.wikimedia.org/r/748876

RLazarus added a comment.Dec 22 2021, 4:04 AM

Status update, leaving this for the end-of-year break: 748876 and 748873 are still to be merged, but then the hourly scan, and web API, should be done and ready.

Still to do:

  • Add an active/passive DNS discovery record (and read it from the Puppet class so that the imagecatalog record systemd timer is only present on the active host)
  • Regularly rsync the database from the active host to the passive one
  • Add code to blubber and docker-pkg to POST to /register_build for each new image build
  • Write it all up on Wikitech
  • Start thinking about integration with Clair and Debmonitor

I’ll pick that all back up after the break. @Joe: My recollection is you were going to take care of the blubber and docker-pkg parts (although it’s been a while since we talked about it) -- with the exception of adding a DNS record to talk to, that should be unblocked, so feel free to grab it while I’m still out, if you're so inclined.

Joe added a comment.Dec 22 2021, 6:51 AM
In T287130#7585276, @RLazarus wrote:

@Joe: My recollection is you were going to take care of the blubber and docker-pkg parts (although it’s been a while since we talked about it) -- with the exception of adding a DNS record to talk to, that should be unblocked, so feel free to grab it while I’m still out, if you're so inclined.

Your recollection is correct; I'll do what I can to actually start that work while you're out but not sure I'll get to it :)

Dzahn added a subscriber: Dzahn.EditedDec 22 2021, 6:51 PM
In T287130#7585276, @RLazarus wrote:
  • Regularly rsync the database from the active host to the passive one

Hello @RLazarus and @Joe,

I have good news. This part is already done :)

I was about to add code for this, then got reminded of class rsync::deployment and this has both the $deployment_server parameter to check whis is active and it syncs $deployment_path = '/srv/deployment'. And since you are under /srv/deployment/imagecatalog/ this is alerady included.

root@deploy1002:/srv/deployment/imagecatalog# ls -als catalog.sqlite 
64 -rw-r--r-- 1 imagecatalog imagecatalog 65536 Dec 21 00:10 catalog.sqlite


root@deploy2002:/srv/deployment/imagecatalog# ls -als catalog.sqlite 
64 -rw-r--r-- 1 helm helm 65536 Dec 21 00:10 catalog.sqlite



deploy1002:/etc/rsync.d] $ cat frag-deployment_module 
# This file is being maintained by Puppet.
# DO NOT EDIT

[ deployment_module ]
path            = /srv/deployment
read only       = yes


[deploy2002:~] $  sudo systemctl status sync_deployment_dir.timer
● sync_deployment_dir.timer - Periodic execution of sync_deployment_dir.service
   Loaded: loaded (/lib/systemd/system/sync_deployment_dir.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Wed 2021-02-24 23:27:14 UTC; 9 months 26 days ago
  Trigger: Wed 2021-12-22 19:00:00 UTC; 11min left


[deploy2002:~] $  sudo systemctl status sync_deployment_dir.service
● sync_deployment_dir.service - rsync the deployment server data directory /srv/deployment
   Loaded: loaded (/lib/systemd/system/sync_deployment_dir.service; static; vendor preset: enabled)
   Active: inactive (dead) since Wed 2021-12-22 18:00:34 UTC; 48min ago
  Process: 30285 ExecStart=/usr/bin/rsync -avz --delete deploy1002.eqiad.wmnet::deployment_module /srv/deployment (code=exited, status=0/SUCCESS)
 Main PID: 30285 (code=exited, status=0/SUCCESS)

Dec 22 18:00:04 deploy2002 systemd[1]: Started rsync the deployment server data directory /srv/deployment.
Dec 22 18:00:04 deploy2002 rsync[30285]: receiving incremental file list

passive host pulls from active host, with --delete

☑

gerritbot added a comment.Jan 25 2022, 9:28 PM

Change 748876 merged by RLazarus:

[operations/puppet@production] imagecatalog: Add an hourly systemd timer to scan for what's currently running

https://gerrit.wikimedia.org/r/748876

RLazarus added a comment.Jan 26 2022, 4:38 AM

The hourly imagecatalog record timer is working on deploy1002. It's failing on deploy2002, because something keeps overwriting the ownership of /srv/deployment/imagecatalog/catalog.sqlite -- it's supposed to be owned by imagecatalog, and that's how Puppet creates it, but in fact it's owned by helm. I haven't been immediately able to figure out why (the catalog is presumably getting blown away by that rsync, but I don't see where the helm user is coming from) but I don't think it needs to be figured out: that systemd timer is only going to run in the active DC anyway, so as soon as I set that up in the next Puppet patch, it'll be a non-issue.

I think we need to do something a little more complicated than just the overall /srv/deployment rsync, because we want to make sure the right thing happens when we switch DCs (regardless of when Puppet happens to run on each host) but we can address that in a followup.

gerritbot added a comment.Jan 26 2022, 9:45 PM

Change 757530 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] imagecatalog: Only run on the active deployment host

https://gerrit.wikimedia.org/r/757530

gerritbot added a comment.Feb 2 2022, 6:06 PM

Change 757530 merged by RLazarus:

[operations/puppet@production] imagecatalog: Only run on the active deployment host

https://gerrit.wikimedia.org/r/757530

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:23 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL