Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
root@ms-fe2009:~# swift stat docker_registry_codfw
Account: AUTH_docker
Container: docker_registry_codfw
Objects: 1963747
Bytes: 9865351275542
[..]vs
oot@ms-fe1009:~# swift stat docker_registry_codfw
Account: AUTH_docker
Container: docker_registry_codfw
Objects: 1513819
Bytes: 7579165002418So it doesn't seem to be a metric issue :( The replication has been broken since May, we'll need an alarm for it.
Comment Actions
I understand why SRE-swift-storage got tagged, but: replication between eqiad and codfw does not happen for the ms clusters - the user is responsible for any replication (we fudge this a little as far as mediawiki is concerned with the weekly rclone job).
By which I mean: something above swift is meant to do this, not swift itself.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-12-18T09:36:43Z] <Emperor> restart swift-container-sync on ms-be2081 T413008
Comment Actions
At least one of the problems is that the container is damaged - there are objects in the container db that don't exist on disk e.g.:
root@ms-fe2009:/home/mvernon# swift list docker_registry_codfw --prefix files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/ files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267 root@ms-fe2009:/home/mvernon# swift stat docker_registry_codfw files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267 Object HEAD failed: http://swift.svc.codfw.wmnet/v1/AUTH_docker/docker_registry_codfw/files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267 404 Not Found Failed Transaction ID: tx1f5edef4832f4bf68cc87-006943d3ef
This then breaks the replicator which doesn't even understand the error:
Dec 18 09:36:31 ms-be2081 container-sync[809088]: ERROR Syncing /srv/swift-storage/accounts1/containers/19492/375/4c244b8ad285838b1df91b5193ff8375/4c244b8ad285838b1df91b5193ff8375.db {'ROWID': 4401258, 'name': 'files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267', 'created_at': '1744938403.01136', 'size': 108, 'content_type': 'application/octet-stream', 'etag': '04b7b763bc85b4d2b3d95769bc87b669', 'deleted': 0, 'storage_policy_index': 0}: #012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/swift/container/sync.py", line 622, in container_sync_row#012 raise Exception(#012Exception: Unknown exception trying to GET: 'AUTH_docker' 'docker_registry_codfw' 'files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267' (txn: tx6502691d5bc343ebbd651-006943cb1f)Comment Actions
And we can see on that server that the sync is going nowhere...
background.log:Dec 18 10:11:20 ms-be2081 container-sync: Container sync report: AUTH_docker/docker_registry_codfw, time window start: 1766052380.2261772, time window end: 1766052680.2847583, puts: 0, posts: 0, deletes: 0, bytes: 0, sync_point1: 4407377, sync_point2: 4401204, total_rows: 5812137 (txn: tx8410df6ea677402da9c00-006943d345) background.log:Dec 18 10:16:20 ms-be2081 container-sync: Container sync report: AUTH_docker/docker_registry_codfw, time window start: 1766052680.2902746, time window end: 1766052980.3537154, puts: 0, posts: 0, deletes: 0, bytes: 0, sync_point1: 4407377, sync_point2: 4401204, total_rows: 5812137 (txn: txc002c6b9731942da96673-006943d466)
Comment Actions
And the summary:
Dec 18 08:51:02 ms-be2081 container-sync: Since Thu Dec 18 07:50:57 2025: 12 synced [0 deletes, 0 puts], 0 skipped, 1583 failed (txn: tx71bc061fffed4b3b87059-006943c06f)
Comment Actions
elukey@ms-be2081:~$ sudo journalctl -u swift-container-sync.service| egrep "\.db " | grep "Unknown exception" | egrep -o "'name': 'files/.*', 'created" | cut -d "'" -f 4 | sort | uniq files/docker/registry/v2/repositories/amd-gpu-tester/_uploads/f0f238f5-4ac9-4b33-97d3-02343642e0c4/hashstates/sha256/0 files/docker/registry/v2/repositories/amd-pytorch23/_uploads/4d81ae3f-3209-4b31-a5b3-eb90ecf96546/hashstates/sha256/2497414924 files/docker/registry/v2/repositories/bookworm/_uploads/269ba3c5-53f0-41eb-abde-cf376b5b77f6/startedat files/docker/registry/v2/repositories/buster/_uploads/58dec3a3-b944-4dfa-9ab0-94a189e700be/hashstates/sha256/0 files/docker/registry/v2/repositories/ceph/_uploads/14779110-08bc-484f-96c5-fd21dc2f9a6b/data files/docker/registry/v2/repositories/cfssl-issuer/_uploads/06a2ab6d-e463-4db1-b2ce-6dee93cd63e0/startedat files/docker/registry/v2/repositories/coredns/_uploads/6fc5947f-5894-4608-8e9d-5f9a65e9b0de/hashstates/sha256/0 files/docker/registry/v2/repositories/envoy-future/_uploads/af9aacfe-5d88-4b25-bb5e-a21d6563e91a/hashstates/sha256/0 files/docker/registry/v2/repositories/eventrouter/_uploads/f4815d2c-6df2-4a86-ba4a-c33a03fe9d78/hashstates/sha256/12299914 files/docker/registry/v2/repositories/flink/_uploads/4a673e0e-bcd4-4e1c-8e0f-96b5e561a721/data files/docker/registry/v2/repositories/flink/_uploads/897de4bc-5377-4620-ab1b-9c37e0f421b0/hashstates/sha256/74683302 files/docker/registry/v2/repositories/golang1.15/_uploads/ceffb44c-df40-48e5-8fb3-1c6b5da88806/data files/docker/registry/v2/repositories/httpd-fcgi/_uploads/2d58ea7e-0933-4adb-b4a6-b49dc38d754e/hashstates/sha256/3873 files/docker/registry/v2/repositories/httpd-fcgi/_uploads/4d76687b-c0a0-4b91-8777-5d4c9861f416/hashstates/sha256/0 files/docker/registry/v2/repositories/knative-build/_uploads/06f610a9-84c2-4e98-8fc9-ddc6209d8b8e/startedat files/docker/registry/v2/repositories/knative-serving-domain-mapping-webhook/_uploads/f7a0cd68-388e-4fa4-9521-142257852e0b/hashstates/sha256/2365 files/docker/registry/v2/repositories/knative-serving-webhook/_uploads/9cd617aa-73bd-453a-8643-136f3d4e4cbc/data files/docker/registry/v2/repositories/kserve-agent/_uploads/2138b7cf-520b-41f3-9ad2-47cecf89bddb/data files/docker/registry/v2/repositories/kserve-controller/_uploads/8f6ca1c1-1508-4741-a396-942973b99976/hashstates/sha256/30247021 files/docker/registry/v2/repositories/kserve-storage-initializer/_uploads/6c16e3ce-871d-4319-9688-95782e73d058/startedat files/docker/registry/v2/repositories/mediawiki-httpd/_uploads/1f88554d-6b94-4072-9d5f-ad90788e4f2e/data files/docker/registry/v2/repositories/mediawiki-httpd/_uploads/9534a76e-1e0c-4140-90f1-1c7f29c77202/hashstates/sha256/0 files/docker/registry/v2/repositories/nodejs18-slim/_uploads/99d9f18a-4244-4015-bfcf-6da56bc505dd/hashstates/sha256/0 files/docker/registry/v2/repositories/openjdk-17-jre/_uploads/06c27615-b713-46e7-976b-8371be665552/startedat files/docker/registry/v2/repositories/php7.4-cli/_uploads/04e69509-1923-4b66-bb36-b9cf32c1b3fe/hashstates/sha256/0 files/docker/registry/v2/repositories/php7.4-fpm-multiversion-base/_uploads/08fb6155-9bf6-4981-a84c-c01367609b88/hashstates/sha256/0 files/docker/registry/v2/repositories/php7.4-fpm-multiversion-base/_uploads/a335c6f0-6bc4-498e-874c-1a9b24ab8f2c/hashstates/sha256/0 files/docker/registry/v2/repositories/php7.4-fpm/_uploads/ec5346ce-f205-4b36-872d-184697650848/data files/docker/registry/v2/repositories/php8.1-cli/_uploads/96ee4152-fcf8-4aba-90f9-b60e6c0eebe8/hashstates/sha256/4826 files/docker/registry/v2/repositories/python2-build-buster/_uploads/9d781fe7-3f7e-4f95-b61d-ee592645ef5c/hashstates/sha256/0 files/docker/registry/v2/repositories/python3-bookworm/_uploads/12d4819b-28e9-4da9-af9c-4da2e8816132/data files/docker/registry/v2/repositories/python3-build-bullseye/_uploads/3fecd93c-7d63-4ebe-a4e6-6b7cb954019c/hashstates/sha256/892 files/docker/registry/v2/repositories/python3-build-buster/_uploads/0b67a151-d150-4a85-b183-ae977ad857c9/hashstates/sha256/0 files/docker/registry/v2/repositories/releng/quibble-bullseye-php74-coverage/_uploads/32e7bdec-1de2-4ac2-aae4-d44f469a7809/hashstates/sha256/248 files/docker/registry/v2/repositories/releng/quibble-bullseye-php74-coverage/_uploads/d94f4a31-c3ec-43d8-b392-49b056a6f90e/hashstates/sha256/0 files/docker/registry/v2/repositories/releng/quibble-bullseye-php74/_uploads/ac285a4f-869d-4db7-a1ff-f5fbe360b882/hashstates/sha256/0 files/docker/registry/v2/repositories/releng/quibble-fresnel/_uploads/3ad30e3a-d0d1-4398-942a-5e3c643ba95d/hashstates/sha256/0 files/docker/registry/v2/repositories/releng/quibble-fresnel/_uploads/6888c6a0-d618-4da2-b8d5-2debfccff496/data files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all/_uploads/0da217d3-6926-4e2c-a409-b8aa2996f3c2/hashstates/sha256/267 files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all/_uploads/79874efc-0f95-477a-830b-975a141f0505/hashstates/sha256/0 files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all/_uploads/bcd0fdd2-60ff-4760-a9d0-cd6626410dd2/startedat files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all/_uploads/d092e89d-3e16-4722-bd8f-a8bf2e904f94/startedat files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-python3-all/_uploads/29663922-8d72-44dc-8fe8-936f70c007dd/hashstates/sha256/190 files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-python3-all/_uploads/6ba2642b-b29c-4e38-babc-0df76e0ea7ab/data files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-python3-all/_uploads/702059f0-d58e-4402-8afb-477d15ca18c5/hashstates/sha256/0 files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-orchestrator/_uploads/361a506d-5a53-43a1-a4e1-295f068de790/data files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-orchestrator/_uploads/75e852af-8318-4958-8943-cfb603f8cc05/hashstates/sha256/3247 files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-orchestrator/_uploads/d2cec1db-dd97-4eb3-8b57-f61d2b7a32d5/startedat files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-orchestrator/_uploads/d6330de2-86b0-4627-aa80-170f1316eb72/data files/docker/registry/v2/repositories/repos/abstract-wiki/wikifunctions/function-orchestrator/_uploads/f3021e72-7deb-41c1-b701-3f9ada0cc820/data files/docker/registry/v2/repositories/repos/releng/scap/bullseye/_uploads/071c2ef5-71eb-43ff-bdb1-4fce2ba3517c/hashstates/sha256/3265 files/docker/registry/v2/repositories/repos/releng/scap/bullseye/_uploads/11d38b4e-97b1-47a4-bf69-7f6da7633d36/hashstates/sha256/2101 files/docker/registry/v2/repositories/repos/releng/scap/buster/_uploads/01ca907e-3658-40b8-807b-63ec316cf75a/data files/docker/registry/v2/repositories/repos/releng/scap/buster/_uploads/34c06852-ac28-4133-b2a0-53d8d3495ac3/startedat files/docker/registry/v2/repositories/repos/search-platform/cirrus-streaming-updater/producer/_uploads/26ba7c5c-14df-477b-8ec1-8f51203f694c/hashstates/sha256/0 files/docker/registry/v2/repositories/repos/test-platform/catalyst/patchdemo/_uploads/43ee8765-6802-4010-91db-da66427f1bd4/startedat files/docker/registry/v2/repositories/repos/test-platform/catalyst/patchdemo/_uploads/ea1b346a-3b89-4cca-b3e5-b82389d69354/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/09ea819a-843c-483b-9555-50bd9a93ba23/hashstates/sha256/453266 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/12afce86-9bdb-4501-beb9-7bd970a290f6/hashstates/sha256/44921528 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/13d977a0-90fe-47c9-b2af-75234a801ab6/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/1cca1f50-2fa6-4754-8a8f-e732cd4508aa/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/22c40b12-2e5a-44d6-86ca-4dced81a71dd/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/349f3ca0-932c-402e-afd0-f5cfe64b3f21/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/37c4789d-656a-413e-9a62-0d6bc1038078/hashstates/sha256/453257 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/3f520157-d61c-4113-8e0f-85590ca64d1a/startedat files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/4b9caf2a-e3bb-4cad-b3f5-b0d7413f4a92/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/5713fed0-58f7-4fa7-843b-5e90ce5d2299/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/5bfbb668-a678-45f8-bde8-2c2c09205a4a/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/823c4c19-1c1a-438b-9986-a54546c64e86/hashstates/sha256/16535 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/87ed016e-f4a8-4ddf-a65c-0b7eaca88b52/hashstates/sha256/16255 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/964f7f8f-9c5a-441b-85a5-322b38654a6e/hashstates/sha256/40285656 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/97f71b9d-597e-48ad-b289-74be296be902/hashstates/sha256/15591 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/a3202d28-afd2-4dc2-9705-c61fc26992cd/hashstates/sha256/40285733 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/aa4bc24d-6d3f-4160-b7b4-cda370d26962/hashstates/sha256/40285204 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/ac59f6e3-5cf5-4557-94e5-3014030b5dbe/hashstates/sha256/40285441 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/b8917150-c2c8-47fc-933c-6546b8e5e14d/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/bda2e41e-b339-40d3-8fa9-73d5c0f80b44/startedat files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/fbbfb3ae-0187-4719-a1e7-c42051023172/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/17dac947-80db-4afc-88fd-078d8c3eb115/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/19416d03-3ebb-4dcc-bfa1-65ee65d59fa0/startedat files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/26133e00-adb8-4514-827b-eb4c07e16a94/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/2e040bfb-435d-4c0f-86b4-fa31122565c0/hashstates/sha256/16417 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/4ffa0ae9-f50d-4bd9-878e-52b4ce5187d9/hashstates/sha256/18480 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/67ac9c4b-16b0-4602-9d7d-cd9bf4d69753/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/692d9f06-1198-4bc4-9eb8-3faaa7c65400/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/7c281cfa-6620-40e5-a6d8-aba84a6bb812/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/9cd255ae-4b8b-4d30-bb2c-1e660ef99474/hashstates/sha256/15744 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/ae7278ce-3485-4d77-b6f2-80e5ecb10a49/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/bb4a2a6b-f63b-4569-8a68-aa462ec27827/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/02255485-604c-4ea7-98b2-927feaaa136a/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/097e6b87-e3b7-4cf2-b406-42d838b0b646/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/0eb89240-9dde-44e5-a7eb-cf49a9f362f4/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/160b45af-90d2-4218-8215-1e6a931b0c2e/startedat files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/2f47b04e-875b-41c1-8fbe-1e1af8758193/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/2fcdea47-d7cf-453e-9150-1a393470889e/hashstates/sha256/14969996 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/3108fdb1-d8ec-465f-ab34-18809303196e/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/331f6d84-0aa8-46a7-8df7-ed42366129ae/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/44ed0f29-26cf-43f5-a24d-d61d7cb26eb6/hashstates/sha256/14362 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/4cf08b59-e063-4a0d-996e-2dd9ba11e484/hashstates/sha256/479 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/57636755-d856-466a-af89-a72e66cc2a52/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/64d5a418-4750-42a6-af46-2edd79698d84/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/77e18cdf-6755-4538-ba29-ecc0d32f9b61/hashstates/sha256/120338 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/7f542620-aa2b-40cc-a2e6-6cb9722ac2d0/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/993915c1-3b97-46c3-8793-b0eef677f0df/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/9bddc9da-d904-49e5-aa00-f5dd8d339ec0/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/bbeecd77-0452-41eb-ac47-f51dab3b0b91/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/c5f80786-d9b7-4477-afa9-8ca886153714/hashstates/sha256/15653 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/d9b40320-2fdf-49a7-a81a-ecb0015a3412/hashstates/sha256/7738 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/ee881ee8-c236-4527-ab7d-224b0de06bb9/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/f134043c-2836-43e4-b415-c9bff0a955dc/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/fbbd58db-1362-4d4c-8ca3-4f5271077c66/hashstates/sha256/13378 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/ff21da74-817b-4757-97a0-fc393012c88f/startedat files/docker/registry/v2/repositories/restricted/mediawiki-webserver/_uploads/3358c059-2b89-4679-9544-9800396dbd9f/data files/docker/registry/v2/repositories/rsyslog/_uploads/eeef6437-307a-404a-9d7c-affe779819cd/hashstates/sha256/2503726 files/docker/registry/v2/repositories/wikimedia/labs-codesearch/_uploads/ee61ca79-5b6f-4e16-bbc6-e0638f33088f/data files/docker/registry/v2/repositories/wikimedia/labs-striker/_uploads/049eec7d-1a1f-40c0-8e30-eeb4d60f5f4b/hashstates/sha256/274 files/docker/registry/v2/repositories/wikimedia/machinelearning-liftwing-inference-services-edit-check/_uploads/3982b64e-b242-4b7b-bb3f-5307d565db43/hashstates/sha256/8952 files/docker/registry/v2/repositories/wikimedia/mediawiki-libs-shellbox/_uploads/38cf3056-e4a6-477f-8e05-c7d5ca742f08/data files/docker/registry/v2/repositories/wikimedia/mediawiki-libs-shellbox/_uploads/9e8fe6f7-04b0-4e79-9daa-36bdaa693654/startedat files/docker/registry/v2/repositories/wikimedia/mediawiki-libs-shellbox/_uploads/ac50f642-0e1a-4ea7-89de-7c05a04b444b/data files/docker/registry/v2/repositories/wikimedia/mediawiki-services-chromium-render/_uploads/81a6fe90-910e-48d4-bec4-0fbc92291ef2/data files/docker/registry/v2/repositories/wikimedia/mediawiki-services-chromium-render/_uploads/f177e4b6-fe58-4020-9ebb-bce667ad59a9/hashstates/sha256/3573 files/docker/registry/v2/repositories/wikimedia/operations-software-bitu/_uploads/48457049-8af8-48a6-af0b-0558e7d1a4cf/data files/docker/registry/v2/repositories/wikimedia/operations-software-bitu/_uploads/a54eee8f-214c-46c8-a5fe-91929b5df14b/data files/docker/registry/v2/repositories/wikimedia/operations-software-bitu/_uploads/c6afe58f-0aac-41bf-9497-f0f02b4fe9aa/startedat files/docker/registry/v2/repositories/wikimedia/wikimedia-developer-portal/_uploads/0207a0dd-2a1a-4d84-9ad5-8135244dbf39/hashstates/sha256/2432905 files/docker/registry/v2/repositories/wikimedia/wikimedia-toolhub/_uploads/1b2872c1-b9a0-4018-a8d2-860cce887608/hashstates/sha256/0
Comment Actions
There are false-positives in that list (e.g. the last one is a good object, but there was a timeout). To pick out the bad ones only, I used:
journalctl -o cat -u swift-container-sync.service -g 'Unknown exception trying to GET' | sed -e "s/.* 'name': '\([^']*\)'.*/\1/" | sort -u
Which gives us 134 objects, which I fed to phaste -
Comment Actions
Earlier, we tested an approach (used before with ghost swift objects cf T327253) of deleting one such bad object (files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/0ac43c8d-5203-4cbb-9b28-a9a3cec7d60f/hashstates/sha256/453267) which did work (after a couple of minutes).
So we could feed this list (or the pipeline) to something like:
if swift stat docker_registry_codfw objectname 2>&1 | grep -q '404 Not Found' then swift delete docker_registry_codfw objectname 2>&1 | grep -q '404 Not Found' fi
which is what the remove ghost objects cookbook does (we can't use that cookbook because it's solving a different but related problem).
Comment Actions
A few other notes (beyond "we should stop using swift_container_sync already"):
Locate the container DB files with swift-get-nodes /etc/swift/container.ring.gz AUTH_docker docker_registry_codfw, which tells us:
/srv/swift-storage/accounts1/containers/19492/375/4c244b8ad285838b1df91b5193ff8375 on ms-be2089, /srv/swift-storage/accounts1/containers/19492/375/4c244b8ad285838b1df91b5193ff8375 on ms-be2081, and /srv/swift-storage//accounts0/containers/19492/375/4c244b8ad285838b1df91b5193ff8375 on ms-be2083
And have a quick check for consistent-count with sudo sqlite3 --readonly [path] "SELECT COUNT(*) FROM object WHERE deleted == 0", we had two containers with 1963748, one with 1963747, which I don't propose to worry about.
I did kick off a shell fragment to do a comprehensive check of the container, but it's going to take Quite A While, as it has to stat 2million objects:
sad=0 happy=0 for obj in $(swift list docker_registry_codfw) ; do if swift stat docker_registry_codfw "$obj" 2>&1 | grep -q '404 Not Found' then echo "$obj" >> /home/mvernon/sadobjs let sad++ else let happy++ fi done
I think the pragmatic next step is to delete the 134 objects in P86731 (iff they still 404), and wait a bit.
Comment Actions
Many thanks for investigating this @MatthewVernon and @elukey.
It's interesting how _uploads paths dominate the set of bad objects, since it suggests there's something about how the registry's swift driver implements push storage that tickles something odd in swift. While I doubt there's anything we can take specifically from that observation to make this less likely to recur in the future (though it's questionable whether upload storage should be replicated at all), it's still notable.
In any case, I'm happy to take on adding an alert for this - e.g., based on cross-cluster swift_account_stats_bytes_total drift.
Comment Actions
I've executed the following from ms-fe2009, deleting the objects that Matthew highlighted above:
while read object; do if swift stat docker_registry_codfw $object 2>&1 | grep -q '404 Not Found'; then swift delete docker_registry_codfw $object; fi; done < ghost_objects.txt
I got 404 Not Found for all Deletes.
Comment Actions
My unfounded suspicion is that the bad objects were trying to be uploaded during an incident affecting ms swift (commons scraping, DDoS, etc.)
Comment Actions
After the container-sync restart on ms-be2081, I noticed the following errors and I tried to safely delete them too:
files/docker/registry/v2/repositories/amd-gpu-tester/_uploads/f0f238f5-4ac9-4b33-97d3-02343642e0c4/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_layers/sha256/00a1bf442834e61efe15d7a9f88e30fa89858e336ce7672547367098e790d57f/link files/docker/registry/v2/repositories/restricted/mediawiki-multiversion/_uploads/6ff59f3d-de7d-4bc7-9368-ebba76896db3/hashstates/sha256/53349 files/docker/registry/v2/repositories/wikimedia/operations-software-bitu/_uploads/75c0b003-8b9f-4c25-bb17-acde65da8fa6/hashstates/sha256/0 files/docker/registry/v2/repositories/mediawiki-httpd/_uploads/f527d720-0e7a-4829-a457-02473b8c86d6/startedat
Comment Actions
root@ms-fe1009:~# swift stat docker_registry_codfw
Account: AUTH_docker
Container: docker_registry_codfw
Objects: 1514864The number of objects is already higher than earlier on, and we don't see any error now in the contaiener sync logs. What we did was picking up the object files from the container-sync logs erroring out, and trying to delete them if swift stat already returned HTTP 404 not found. This seems to propagate the DELETE to all the three nodes handling the db sync files (at least, this is my understanding).
Let's wait some time to see if the metrics show some improvement as well. The sync process is very slow so let's be patient :)
Comment Actions
Yes, I think we're at "give it some time", but I think we've unblocked replication codfw->eqiad now.
Comment Actions
Dec 18 14:36:26 ms-be2081 container-sync: Since Thu Dec 18 13:36:25 2025: 12 synced [0 deletes, 0 puts], 0 skipped, 1559 failed (txn: tx37e5caaaa6a247a498cbd-0069441160) Dec 18 16:24:39 ms-be2081 container-sync: Since Thu Dec 18 15:24:26 2025: 12 synced [0 deletes, 2890 puts], 0 skipped, 550 failed (txn: txde00e43295194486b58d0-0069442ac7) Dec 18 17:24:44 ms-be2081 container-sync: Since Thu Dec 18 16:24:39 2025: 12 synced [0 deletes, 520 puts], 0 skipped, 1176 failed (txn: txa2608cc82361456586343-00694438db)
Not totally ideal.
Comment Actions
Found some more with journalctl -o cat -u swift-container-sync.service -g 'Unknown exception trying to GET' -S 15:33 | sed -e "s/.* 'name': '\([^']*\)'.*/\1/" | sort -u (another 99), which are
P86733
I'll remove them as before.
Comment Actions
Dec 18 18:24:54 ms-be2081 container-sync: Since Thu Dec 18 17:24:44 2025: 12 synced [0 deletes, 1901 puts], 0 skipped, 285 failed (txn: tx42793638805744b7be84e-00694446f6)
Comment Actions
This morning, there are 1519692 objects in docker_registry_codfw in eqiad, most recent sync from ms-be2081:
Dec 19 08:25:29 ms-be2081 container-sync: Since Fri Dec 19 07:25:28 2025: 12 synced [0 deletes, 0 puts], 0 skipped, 55 failed (txn: tx53474ae789ac4f3e87e02-0069450acd)
Since 17:33 yesterday, 4 more bad objects from ms-be2081:
mvernon@ms-be2081:/var/log/swift$ journalctl -o cat -u swift-container-sync.service -g 'Unknown exception trying to GET' -S "2025-12-18 17:33" | sed -e "s/.* 'name': '\([^']*\)'.*/\1/" | sort -u files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/643ba817-e0f6-4ef8-8be4-40d09c7f6f51/startedat files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-cli/_uploads/f4358af0-819e-4825-a919-a1b07995fc34/data files/docker/registry/v2/repositories/restricted/mediawiki-multiversion-debug/_uploads/48436a83-d25f-42ee-8a04-1cacb1beb793/hashstates/sha256/0 files/docker/registry/v2/repositories/restricted/mediawiki-webserver/_uploads/45704fdf-6da9-4987-bfd3-9a64980e8e4b/hashstates/sha256/0
I will do the delete-if-still-404 operation on those 4.
Comment Actions
Dec 19 10:25:35 ms-be2081 container-sync: Since Fri Dec 19 09:25:31 2025: 12 synced [0 deletes, 3855 puts], 0 skipped, 5 failed (txn: tx3dc9763b5b7b43baa458e-006945281e)
and 1531266 objects
Comment Actions
This has continued to proceed today:
Dec 19 17:26:37 ms-be2081 container-sync: Since Fri Dec 19 16:26:33 2025: 12 synced [0 deletes, 4154 puts], 0 skipped, 2 failed (txn: tx30b3c67d1a084f869d3cd-0069458acc)
And the container now has 1575546 objects in. At a rough rate of 6000 objects/hour (what we've seen today), we'd hope to have eqiad caught up to codfw around Monday lunchtime...
Comment Actions
The sync seems now complete! Nice work :)
Next steps before closing: update documentation in Wikitech and write a summary here for posterity.
Comment Actions
I think we're almost-but-not-quite-entirely caught up, but yes, I think the reported issue is done; If the follow-ups aren't going to be finished before year-end, can we close this task and make another for them?
Comment Actions
Summary:
The Docker Registry is the only Swift client that uses the native container-sync replication to mirror codfw images to eqiad. For some reason that we didn't get, the 3 swift-container-sync replicas running on ms-be2* hosts that are/were handling the Registry's container replication got into an inconsistent state, and they started to report problems while mirroring to eqiad. We discovered that all the errors reported in container-sync logs were related to Docker images/layers already deleted, so we got all the file/object names listed in the logs and we re-issued DELETE swift requests for all of those. This seemed to have forced the 3 replicas to get into a consistent state, and as a consequence they restarted to mirror to eqiad without issues. The backlog caught up in a few days.