Looking at the numbers in logstash, the overall time for build-and-push-conainer-images appears to now be three times worse than when this was filed, approximately 11m 30s when doing a real build: https://logstash.wikimedia.org/goto/698d4cf72cf07a01723fc137a5d7820b

Is the capped upload speed likely to be the issue? There's also T366778: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts (for speeding up the step after this, AIUI), which might be affected?

Looks like something between the deployment server and the registry:
registry rx graph for the last long build-and-push

As is apparent, the registry can RX a lot higher, and looking at other times for the deployment server graph, it can TX a lot higher as well.

Creating a 8GB file (about the size of the mediawiki image) and using scp to transfer it between deploy1002 and registry2004 yields upwards of 60MB/s, and takes 2m30, so I'd wager it's something about how docker pushes the image that yields such bad results.

Just to point out that this is probably not from the network. We don't have networking rate limiting in either of these machines (nor actually anywhere) and 5MB/s is less than 5% of the capacity of a 1Gbps link, which is the lowest common denominator in our infrastructure.

It's possible it's to do with docker using single-threaded gzip for compression on push https://github.com/moby/moby/issues/41987

If compression / CPU is a bottleneck, and assume the images layers are already compressed, I imagine there would be no need to compress on push when there is enough network bandwidth. That is also 2013 bug https://github.com/moby/moby/issues/1266.

Self note: we have same issue when building and publishing Docker the CI images on contint.wikimedia.org.