Change Details

I tried to check on images for T348647 and found that: * debmonitor is lacking most of our images (basically everything that should be handled by k8s_rules.ini IIUC) * docker-report is painfully slow, taking >5min to fetch the docker-registry-catalog (which probably is worth another task) * docker-report uses `docker-registry.wikimedia.org` as registry, so everything goes via CDN (the catalog is not cached IIRC) * debmonitor data is hard to read as there are so many images that we never ran and probably will never run in prod To work around the immediate issue I crated a list of all images currently running in prod, currently running `docker-report-debmonitor` for all of them on build2001

I tried to check on images for T348647 and found that: * debmonitor is lacking most of our images (basically everything that should be handled by k8s_rules.ini IIUC) * docker-report is painfully slow, taking >5min to fetch the docker-registry-catalog (which probably is worth another task) * docker-report uses `docker-registry.wikimedia.org` as registry, so everything goes via CDN (the catalog is not cached IIRC) * debmonitor data is hard to read as there are so many images that we never ran and probably will never run in prod To work around the immediate issue I crated a list of all images currently running in prod, currently running `docker-report-debmonitor` for all of them on build2001: ``` kubectl get pods --all-namespaces --field-selector=status.phase=Running -o jsonpath="{..image}" | tr ' ' '\n' | sed s/docker-registry.discovery.wmnet/docker-registry.wikimedia.org/ | sort -u > prod_images # on build2001 export http_proxy=http://webproxy.codfw.wmnet:8080 mkdir /tmp/tmpxczldyc8-docker-report chgrp debmonitor /tmp/tmpxczldyc8-docker-report chmod 0770 /tmp/tmpxczldyc8-docker-report cat prod_images | while read image; do docker-report-debmonitor "$image" /tmp/tmpxczldyc8-docker-report; done ``` While running that I realized it would be nice if docker-report-debmonitor could check if the image has already been submitted to debmonitor (to not do it again). It does not seem possible to GET or HEAD URLS like `debmonitor.discovery.wmnet/images/docker-registry.wikimedia.org/httpd-fcgi:2.4.38-10-u4-20231009` even when authenticating using the hosts client cert.

I tried to check on images for T348647 and found that: * debmonitor is lacking most of our images (basically everything that should be handled by k8s_rules.ini IIUC) * docker-report is painfully slow, taking >5min to fetch the docker-registry-catalog (which probably is worth another task) * docker-report uses `docker-registry.wikimedia.org` as registry, so everything goes via CDN (the catalog is not cached IIRC) * debmonitor data is hard to read as there are so many images that we never ran and probably will never run in prod To work around the immediate issue I crated a list of all images currently running in prod, currently running `docker-report-debmonitor` for all of them on build2001: ``` kubectl get pods --all-namespaces --field-selector=status.phase=Running -o jsonpath="{..image}" | tr ' ' '\n' | sed s/docker-registry.discovery.wmnet/docker-registry.wikimedia.org/ | sort -u > prod_images # on build2001 export http_proxy=http://webproxy.codfw.wmnet:8080 mkdir /tmp/tmpxczldyc8-docker-report chgrp debmonitor /tmp/tmpxczldyc8-docker-report chmod 0770 /tmp/tmpxczldyc8-docker-report cat prod_images | while read image; do docker-report-debmonitor "$image" /tmp/tmpxczldyc8-docker-report; done ``` While running that I realized it would be nice if docker-report-debmonitor could check if the image has already been submitted to debmonitor (to not do it again). It does not seem possible to GET or HEAD URLS like `debmonitor.discovery.wmnet/images/docker-registry.wikimedia.org/httpd-fcgi:2.4.38-10-u4-20231009` even when authenticating using the hosts client cert.