Failing docker registry httpbb tests
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	DPogorzelski-WMF
	Jan 14 2026, 1:13 PM

Description

Running sudo httpbb /srv/deployment/httpbb-tests/docker-registry/* --hosts=registry1004.eqiad.wmnet

results in:

https://docker-registry.wikimedia.org/v2/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:28)
    Status code: expected 401, got 403.
    WWW-Authenticate header: expected 'Basic realm="docker-registry (push)"', was missing.
https://docker-registry.wikimedia.org/v2/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:33)
    Status code: expected 404, got 403.
https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-multiversion/manifests/sha256:bcb74d22d6fe40def32cbc39166889ac6114fcc2810e8a622a4c4983edeb89a7 (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:56)
    X-Cache-Status header: expected to match /^(MISS|BYPASS|EXPIRED|STALE|UPDATING|REVALIDATED|HIT)$/, was missing.
https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-multiversion/blobs/sha256:b7213227cfef387a937968df3b41e0938b593ca4353044adcfd2828ec0167206 (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:69)
    X-Cache-Status header: expected to match /^(MISS|BYPASS|EXPIRED|STALE|UPDATING|REVALIDATED|HIT)$/, was missing.
https://docker-registry.wikimedia.org/v2/restricted/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:85)
    Status code: expected 401, got 403.
    WWW-Authenticate header: expected 'Basic realm="docker-registry restricted (restricted-push)"', was missing.
https://docker-registry.wikimedia.org/v2/restricted/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:94)
    Status code: expected 404, got 403.
===

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	profile::httpbb::docker-registry: improve tests	operations/puppet	production	+32 -16
	httpbb: Remove assertions for X-Cache-Status	operations/puppet	production	+0 -4

Customize query in gerrit

Event Timeline

DPogorzelski-WMF created this task.Jan 14 2026, 1:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2026, 1:13 PM

The 403 vs. 401 or 404 are the result of the tests being run against a read-only (profile::docker_registry::read_only_mode) instance of the registry. I have updated the wikitech page accordingly.

With that out of the way, we still have four failing tests:

jayme@cumin1003:~$ sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2004.codfw.wmnet'
Sending to registry2004.codfw.wmnet...
https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-multiversion/manifests/sha256:bcb74d22d6fe40def32cbc39166889ac6114fcc2810e8a622a4c4983edeb89a7 (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:56)
    X-Cache-Status header: expected to match /^(MISS|BYPASS|EXPIRED|STALE|UPDATING|REVALIDATED|HIT)$/, was missing.
https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-multiversion/blobs/sha256:b7213227cfef387a937968df3b41e0938b593ca4353044adcfd2828ec0167206 (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:69)
    X-Cache-Status header: expected to match /^(MISS|BYPASS|EXPIRED|STALE|UPDATING|REVALIDATED|HIT)$/, was missing.

These seem like a behavior change in docker distribution we should double check since blobs and manifests should be cacheable.

And these which have been recently added by @DPogorzelski-WMF. Maybe you can take a look whats wrong here:

https://docker-registry.wikimedia.org/v2/ml/nonexistent/manifests/latest (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:110)
    Status code: expected 404, got 401.
https://docker-registry.wikimedia.org/v2/ml/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:114)
    Status code: expected 404, got 401.
===
FAIL: 22 requests sent to registry2004.codfw.wmnet. 4 requests with failed assertions

JMeybohm renamed this task from Failing docker registry tests to Failing docker registry httpbb tests.Jan 15 2026, 10:19 AM

Change #1227311 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] httpbb: Remove assertions for X-Cache-Status

https://gerrit.wikimedia.org/r/1227311

gerritbot added a project: Patch-For-Review.Jan 15 2026, 12:28 PM

Change #1227311 merged by JMeybohm:

[operations/puppet@production] httpbb: Remove assertions for X-Cache-Status

https://gerrit.wikimedia.org/r/1227311

The X-Cache-Status failures are gone now:

jayme@cumin1003:~$ sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2004.codfw.wmnet'
Sending to registry2004.codfw.wmnet...
https://docker-registry.wikimedia.org/v2/ml/nonexistent/manifests/latest (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:106)
    Status code: expected 404, got 401.
https://docker-registry.wikimedia.org/v2/ml/nonexistent/blobs/upload (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:110)
    Status code: expected 404, got 401.
===
FAIL: 22 requests sent to registry2004.codfw.wmnet. 2 requests with failed assertions.

@DPogorzelski-WMF could you please take care of fixing the new tests you added for /ml? Thanks!

JMeybohm moved this task from Inbox to Radar (Awareness) on the ServiceOps new board.Jan 15 2026, 1:24 PM

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2026, 1:31 PM

Change #1242452 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::httpbb::docker-registry: improve tests

https://gerrit.wikimedia.org/r/1242452

gerritbot added a project: Patch-For-Review.Feb 23 2026, 5:42 PM

Change #1242452 merged by Elukey: