Search for something in the config repo, for example: https://codesearch.wmcloud.org/search/?q=wgWp25EasterEggsEnable&files=&excludeFiles=&repos=operations%2Fmediawiki-config
Click on one of the filenames that show up as a result
notice the commit hash in the url
remove the path to see the info about the commit

What happens?:

for me, that's https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/3ff486473879abe7eafcf61e60f09784ee5071eb => a commit from mid February 2026

What should have happened instead?:
it should have index a recent commit

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
codesearch: fix invalid calendar format on cleanup systemd timer	operations/puppet	production	+1 -1
codesearch: let the lock file cleanup run only once an hour	operations/puppet	production	+2 -2
codesearch: create script/timer to delete zombie lock files	operations/puppet	production	+74 -2
write_config: clarify "skipping restart" log message	labs/codesearch	master	+1 -1
write_config: Fix incomplete wmf_gitlab_group_projects paging	labs/codesearch	master	+4 -2

Customize query in gerrit

Related Objects

Mentioned In: T414163: [EXP] Evaluate removing WANObjectCache 'minAsOf' option, in favor of 'touchedCallback'
T416865: Remove references to MetricsPlatform extension
T405346: Remove okvpn/clock-lts post WMF PHP 8.3 upgrade

Event Timeline

Michael created this task.Mar 24 2026, 5:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2026, 5:07 PM

SomeRandomDeveloper subscribed.Mar 24 2026, 5:13 PM

Nemoralis subscribed.Mar 24 2026, 5:34 PM

Yeah, the "Deployed" index is 3 weeks behind.

https://codesearch.wmcloud.org/deployed/?q=lcobucci%2Fclock%7CLcobucci.%5B%5EJ%5D#MediaWiki%20core
vs
https://codesearch.wmcloud.org/core/?q=lcobucci%2Fclock%7CLcobucci.%5B%5EJ%5D

The separate MediaWiki core index is fine, but the copy of it in the Deployed index is 3 weeks behind.

It is e.g. missing these two changes:

Change #1260063 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/codesearch@master] write_config: clarify "skipping restart" log message

https://gerrit.wikimedia.org/r/1260063

gerritbot added a project: Patch-For-Review.Mar 24 2026, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-24T18:00:36Z] <mutante> codesearch9.codesearch - systemctl restart hound_proxy (T421147)

Hound is still starting up, please wait a few minutes for the initial indexing to complete. See <https://codesearch.wmcloud.org/_health> for more information.

The deployed index should have been re-built. Any difference?

Reedy mentioned this in T405346: Remove okvpn/clock-lts post WMF PHP 8.3 upgrade.Mar 24 2026, 6:06 PM

In T421147#11745820, @Dzahn wrote:

The deployed index should have been re-built. Any difference?

I still see e.g. the difference noted in T421147#11745763 between the Core & Deployed indexes :/

Just commenting to confirm that this is still a problem - the search at https://codesearch.wmcloud.org/search/?q=wmgusecsp&files=InitialiseSettings%5C.php%24&excludeFiles=&repos=operations%2Fmediawiki-config&i=fosho shows wmgUseCSP as having 'default' => false; however, FWICS, that was changed to 'default' => true in https://gerrit.wikimedia.org/r/1248515 on 2026-03-05.

Change #1267285 had a related patch set uploaded (by Krinkle; author: Krinkle):

[labs/codesearch@master] write_config: Fix incomplete wmf_gitlab_group_projects paging

https://gerrit.wikimedia.org/r/1267285

Change #1267285 merged by jenkins-bot:

[labs/codesearch@master] write_config: Fix incomplete wmf_gitlab_group_projects paging

https://gerrit.wikimedia.org/r/1267285

Change #1260063 merged by jenkins-bot:

[labs/codesearch@master] write_config: clarify "skipping restart" log message

https://gerrit.wikimedia.org/r/1260063

The status is generally "up" and the nightly restart/reindex appears to finish without errors.

krinkle@codesearch9:~$ sudo journalctl -u codesearch-write-config -n10000 | grep -E 'hound-core|hound-depl'
…
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-deployed: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-deployed: restarting...
…
Apr 02 06:11:36 codesearch9 write_config.py[4075676]: hound-core: writing new config
Apr 02 06:11:36 codesearch9 write_config.py[4075676]: hound-core: skipping restart
Apr 02 06:11:36 codesearch9 write_config.py[4075676]: hound-deployed: writing new config
Apr 02 06:11:36 codesearch9 write_config.py[4075676]: hound-deployed: skipping restart

In case there is an error without that prefix, or if the process ended before the next service restart in the same loop, looking for surrouding lines:

sudo journalctl -u codesearch-write-config -n10000 | grep -C10 -E 'hound-deployed: restart' 
--

Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-bundled: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-bundled: skipping restart
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-deployed: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-deployed: restarting...
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-services: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-services: skipping restart
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-libraries: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-libraries: skipping restart
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-analytics: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-analytics: skipping restart
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-wmcs: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-wmcs: skipping restart
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-puppet: writing new config
Mar 12 06:00:41 codesearch9 write_config.py[93269]: hound-puppet: skipping restart

All looks good at glance.

krinkle@codesearch9:/var/log$ sudo systemctl status hound-core
* hound-core.service - hound-core
     Loaded: loaded (/lib/systemd/system/hound-core.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-03-24 17:59:51 UTC; 1 week 2 days ago
   Main PID: 3207673 (docker)
      Tasks: 10 (limit: 9515)
     Memory: 14.5M
        CPU: 1min 17.077s
     CGroup: /system.slice/hound-core.service
             `-3207673 /usr/bin/docker run -p 6084:6080 --name hound-core --user=root -v /srv/hound/hound-core:/data -v /etc/hound-gitconfig:/root/.gitconfig docker-registry.wikimedia.org/wikimedia/lab>

Apr 02 21:40:38 codesearch9 docker[3207673]: 2026/04/02 21:40:38 Rebuilding MediaWiki core for 66be08b6ca1e580d434ec8e9a16d1eaa66c0520a
Apr 02 21:41:08 codesearch9 docker[3207673]: 2026/04/02 21:41:08 merge 1 files + mem
Apr 02 21:41:09 codesearch9 docker[3207673]: 2026/04/02 21:41:09 167233692 data bytes, 24740561 index bytes
Apr 02 21:41:10 codesearch9 docker[3207673]: HeapInUse = 1.21
Apr 02 21:41:10 codesearch9 docker[3207673]: HeapIdle  = 467.86
Apr 02 23:11:14 codesearch9 docker[3207673]: 2026/04/02 23:11:14 Rebuilding MediaWiki core for b8af5fe4f3b21a2421aac5f3e1a74c2a95ee9a9e
Apr 02 23:11:42 codesearch9 docker[3207673]: 2026/04/02 23:11:42 merge 1 files + mem
Apr 02 23:11:42 codesearch9 docker[3207673]: 2026/04/02 23:11:42 167233603 data bytes, 24740546 index bytes
Apr 02 23:11:44 codesearch9 docker[3207673]: HeapInUse = 1.25
Apr 02 23:11:44 codesearch9 docker[3207673]: HeapIdle  = 467.79

krinkle@codesearch9:/var/log$ sudo systemctl status hound-deployed
* hound-deployed.service - hound-deployed
     Loaded: loaded (/lib/systemd/system/hound-deployed.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-03-24 17:59:50 UTC; 1 week 2 days ago
   Main PID: 3207406 (docker)
      Tasks: 10 (limit: 9515)
     Memory: 15.5M
        CPU: 1min 22.616s
     CGroup: /system.slice/hound-deployed.service
             `-3207406 /usr/bin/docker run -p 6090:6080 --name hound-deployed --user=root -v /srv/hound/hound-deployed:/data -v /etc/hound-gitconfig:/root/.gitconfig docker-registry.wikimedia.org/wikim>

Apr 03 00:05:42 codesearch9 docker[3207406]: remove the file manually to continue.
Apr 03 00:05:42 codesearch9 docker[3207406]: Continuing...
Apr 03 00:16:26 codesearch9 docker[3207406]: 2026/04/03 00:16:26 Failed to git fetch /data/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92, see output below
Apr 03 00:16:26 codesearch9 docker[3207406]: fatal: Unable to create '/data/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92/.git/shallow.lock': File exists.
Apr 03 00:16:26 codesearch9 docker[3207406]: Another git process seems to be running in this repository, e.g.
Apr 03 00:16:26 codesearch9 docker[3207406]: an editor opened by 'git commit'. Please make sure all processes
Apr 03 00:16:26 codesearch9 docker[3207406]: are terminated then try again. If it still fails, a git process
Apr 03 00:16:26 codesearch9 docker[3207406]: may have crashed in this repository earlier:
Apr 03 00:16:26 codesearch9 docker[3207406]: remove the file manually to continue.
Apr 03 00:16:26 codesearch9 docker[3207406]: Continuing...

So here we have a some kind of failure at least, and a potential reason as to why the hound-deployed index is stale.

krinkle@codesearch9:/var/log$ sudo systemctl status hound-deployed -n100000

Mar 24 17:59:48 codesearch9 systemd[1]: Starting hound-deployed.service - hound-deployed...
Mar 24 17:59:48 codesearch9 docker[3206845]: Error response from daemon: Cannot kill container: hound-deployed: Container 26911ab2b84325613b9c072ca399e0d913150ee171d24d10593acdc6a4758bba is not running
Mar 24 17:59:48 codesearch9 docker[3207001]: hound-deployed
Mar 24 17:59:50 codesearch9 systemd[1]: Started hound-deployed.service - hound-deployed.
Mar 24 17:59:54 codesearch9 docker[3207406]: 2026/03/24 17:59:54 Searcher started for mediawiki/skins/CologneBlue
Mar 24 17:59:54 codesearch9 docker[3207406]: 2026/03/24 17:59:54 Searcher started for mediawiki/extensions/WikimediaEvents
Mar 24 17:59:54 codesearch9 docker[3207406]: 2026/03/24 17:59:54 Failed to git fetch /data/data/vcs-078dae5115472a663af97b870a7738c1058c77a2, see output below
Mar 24 17:59:54 codesearch9 docker[3207406]: fatal: Unable to create '/data/data/vcs-078dae5115472a663af97b870a7738c1058c77a2/.git/shallow.lock': File exists.
Mar 24 17:59:54 codesearch9 docker[3207406]: Another git process seems to be running in this repository, e.g.
Mar 24 17:59:54 codesearch9 docker[3207406]: an editor opened by 'git commit'. Please make sure all processes
Mar 24 17:59:54 codesearch9 docker[3207406]: are terminated then try again. If it still fails, a git process
Mar 24 17:59:54 codesearch9 docker[3207406]: may have crashed in this repository earlier:
Mar 24 17:59:54 codesearch9 docker[3207406]: remove the file manually to continue.
Mar 24 17:59:54 codesearch9 docker[3207406]: Continuing...
Mar 24 17:59:54 codesearch9 docker[3207406]: 2026/03/24 17:59:54 Searcher started for wikibase-serialization
Mar 24 17:59:54 codesearch9 docker[3207406]: 2026/03/24 17:59:54 Searcher started for mediawiki/extensions/QuickSurveys
Mar 24 17:59:55 codesearch9 docker[3207406]: 2026/03/24 17:59:55 Searcher started for mediawiki/extensions/WikibaseLexemeCirrusSearch
Mar 24 17:59:56 codesearch9 docker[3207406]: 2026/03/24 17:59:56 Searcher started for mediawiki/extensions/WikiLove
Mar 24 17:59:56 codesearch9 docker[3207406]: 2026/03/24 17:59:56 Searcher started for mediawiki/extensions/ExternalGuidance
Mar 24 17:59:57 codesearch9 docker[3207406]: 2026/03/24 17:59:57 Searcher started for mediawiki/extensions/CommunityConfiguration
Mar 24 17:59:57 codesearch9 docker[3207406]: 2026/03/24 17:59:57 Searcher started for mediawiki/extensions/MapSources
Mar 24 17:59:58 codesearch9 docker[3207406]: 2026/03/24 17:59:58 Searcher started for mediawiki/services/kartotherian
Mar 24 17:59:59 codesearch9 docker[3207406]: 2026/03/24 17:59:59 Searcher started for mediawiki/skins/Modern
Mar 24 18:00:00 codesearch9 docker[3207406]: 2026/03/24 18:00:00 Searcher started for mediawiki/php/luasandbox
Mar 24 18:00:00 codesearch9 docker[3207406]: 2026/03/24 18:00:00 Searcher started for mediawiki/services/chromium-render/deploy
Mar 24 18:00:01 codesearch9 docker[3207406]: 2026/03/24 18:00:01 Searcher started for wikibase-data-model
Mar 24 18:00:02 codesearch9 docker[3207406]: 2026/03/24 18:00:02 Searcher started for mediawiki/extensions/Gadgets
Mar 24 18:00:03 codesearch9 docker[3207406]: 2026/03/24 18:00:03 Searcher started for mediawiki/extensions/DismissableSiteNotice
Mar 24 18:00:03 codesearch9 docker[3207406]: 2026/03/24 18:00:03 Searcher started for mediawiki/extensions/GoogleNewsSitemap
Mar 24 18:00:04 codesearch9 docker[3207406]: 2026/03/24 18:00:04 Searcher started for mediawiki/extensions/MassMessage
Mar 24 18:00:04 codesearch9 docker[3207406]: 2026/03/24 18:00:04 Searcher started for mediawiki/extensions/OATHAuth
Mar 24 18:00:06 codesearch9 docker[3207406]: 2026/03/24 18:00:06 Searcher started for mediawiki/extensions/PageTriage
Mar 24 18:00:07 codesearch9 docker[3207406]: 2026/03/24 18:00:07 Searcher started for mediawiki/services/wikispeech/pronlex
Mar 24 18:00:08 codesearch9 docker[3207406]: 2026/03/24 18:00:08 Searcher started for mediawiki/extensions/GlobalWatchlist
Mar 24 18:00:09 codesearch9 docker[3207406]: 2026/03/24 18:00:09 merge 0 files + mem
Mar 24 18:00:09 codesearch9 docker[3207406]: 2026/03/24 18:00:09 1628391 data bytes, 1077809 index bytes
Mar 24 18:00:09 codesearch9 docker[3207406]: 2026/03/24 18:00:09 Searcher started for mediawiki/extensions/MobileApp
Mar 24 18:00:10 codesearch9 docker[3207406]: 2026/03/24 18:00:10 Searcher started for mediawiki/extensions/WikibaseCirrusSearch
Mar 24 18:00:10 codesearch9 docker[3207406]: 2026/03/24 18:00:10 Searcher started for Wikidata Query RDF
Mar 24 18:00:10 codesearch9 docker[3207406]: 2026/03/24 18:00:10 Failed to git fetch /data/data/vcs-38c97027593c6e0cc89f3327f80974563db94805, see output below
Mar 24 18:00:10 codesearch9 docker[3207406]: fatal: Unable to create '/data/data/vcs-38c97027593c6e0cc89f3327f80974563db94805/.git/shallow.lock': File exists.
…

This suggests that the "git fatal" error might not be fatal, because the same process does continue past it to start searchers for other repos. So.. does it have such an error for MediaWiki core?

krinkle@codesearch9:/var/log$ sudo systemctl status hound-deployed -n100000 | grep -C10 'MediaWiki core'
Mar 24 18:00:59 codesearch9 docker[3207406]: 2026/03/24 18:00:59 Searcher started for mediawiki/extensions/Calendar
Mar 24 18:00:59 codesearch9 docker[3207406]: 2026/03/24 18:00:59 Searcher started for mediawiki/extensions/Echo
Mar 24 18:01:00 codesearch9 docker[3207406]: 2026/03/24 18:01:00 Searcher started for mediawiki/extensions/timeline
Mar 24 18:01:00 codesearch9 docker[3207406]: 2026/03/24 18:01:00 Searcher started for mediawiki/extensions/WikimediaMessages
Mar 24 18:01:00 codesearch9 docker[3207406]: 2026/03/24 18:01:00 Searcher started for mediawiki/services/ores/wikiclass
Mar 24 18:01:01 codesearch9 docker[3207406]: 2026/03/24 18:01:01 Searcher started for mediawiki/extensions/ChessBrowser
Mar 24 18:01:01 codesearch9 docker[3207406]: 2026/03/24 18:01:01 Searcher started for mediawiki/extensions/Quiz
Mar 24 18:01:01 codesearch9 docker[3207406]: 2026/03/24 18:01:01 Searcher started for mediawiki/extensions/WikibaseMediaInfo
Mar 24 18:01:01 codesearch9 docker[3207406]: 2026/03/24 18:01:01 Searcher started for mediawiki/extensions/NearbyPages
Mar 24 18:01:02 codesearch9 docker[3207406]: 2026/03/24 18:01:02 Searcher started for mediawiki/services/wikifeeds
Mar 24 18:01:02 codesearch9 docker[3207406]: 2026/03/24 18:01:02 Searcher started for MediaWiki core
Mar 24 18:01:02 codesearch9 docker[3207406]: 2026/03/24 18:01:02 Searcher started for mediawiki/extensions/EmailAuth
Mar 24 18:01:02 codesearch9 docker[3207406]: 2026/03/24 18:01:02 Searcher started for mediawiki/extensions/FileExporter
Mar 24 18:01:03 codesearch9 docker[3207406]: 2026/03/24 18:01:03 Failed to git fetch /data/data/vcs-a553edcd387e3d3c0fb8c0e5d1ae4272ada6f5c9, see output below
Mar 24 18:01:03 codesearch9 docker[3207406]: fatal: Unable to create '/data/data/vcs-a553edcd387e3d3c0fb8c0e5d1ae4272ada6f5c9/.git/shallow.lock': File exists.
Mar 24 18:01:03 codesearch9 docker[3207406]: Another git process seems to be running in this repository, e.g.
Mar 24 18:01:03 codesearch9 docker[3207406]: an editor opened by 'git commit'. Please make sure all processes
Mar 24 18:01:03 codesearch9 docker[3207406]: are terminated then try again. If it still fails, a git process
Mar 24 18:01:03 codesearch9 docker[3207406]: may have crashed in this repository earlier:
Mar 24 18:01:03 codesearch9 docker[3207406]: remove the file manually to continue.
Mar 24 18:01:03 codesearch9 docker[3207406]: Continuing...

Seemingly not.

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2026, 12:30 AM

There's quite a few. It's currently 2AM in UTC and the daily restarts are done (all backends up), which means there shouldn't be any git repo in a locked state right now. And yet there are 147 locked repos right now:

Find all .git/shallow.lock files

krinkle@codesearch9:/srv/hound$ ls hound-*/data/vcs-*/.git/shallow.lock | cat
hound-analytics/data/vcs-2ffd36da8bf341414e76d3589430ea9bc84052a0/.git/shallow.lock
hound-analytics/data/vcs-906d0655aaec12e1b1f8b47ba85c187a708b96cd/.git/shallow.lock
hound-analytics/data/vcs-a97e2c874ff6e18727907f67290607749a013711/.git/shallow.lock
hound-analytics/data/vcs-c7a85ae5a5e8c17fa4bfec616bf8836667cd7f34/.git/shallow.lock
hound-apps/data/vcs-2a54726bf2a68fdec0390aeda27c13a3a491a842/.git/shallow.lock
hound-apps/data/vcs-7de76661721b0e242ca934bb1e1e542e13b19976/.git/shallow.lock
hound-bundled/data/vcs-19def42b774942cff2bc99087d847d39c547c058/.git/shallow.lock
hound-bundled/data/vcs-1a2fae9fe06d4e30dbd210c94f2cfd5c6d070509/.git/shallow.lock
hound-bundled/data/vcs-1bdcf83cb0685a031d2f2ea57d81ad61ab2bf386/.git/shallow.lock
hound-bundled/data/vcs-1d007b8a2fb723be3cb3a06a10d1adcf177f0e50/.git/shallow.lock
hound-bundled/data/vcs-297789461394415150c2467a94343af6fb429c8e/.git/shallow.lock
hound-bundled/data/vcs-3503a2a1df37ed2b9d17bed5035ea7fe28865b42/.git/shallow.lock
hound-bundled/data/vcs-5d8e31ee84d3a004bf1008c0687fbb23450f0b2f/.git/shallow.lock
hound-bundled/data/vcs-6197ecc09134e4cc8fa0b351a67722529bb86a98/.git/shallow.lock
hound-bundled/data/vcs-6e636f69cd6f9627737b65fa07d2537d6f122553/.git/shallow.lock
hound-bundled/data/vcs-7b6f2b11fea817f3769369316a6abbee601568ac/.git/shallow.lock
hound-bundled/data/vcs-7c91571fdbba94d25f3d69eefae0b625ba3f6867/.git/shallow.lock
hound-bundled/data/vcs-a4504f93c44b3244d5fbe42576182233804f8ce0/.git/shallow.lock
hound-bundled/data/vcs-bd0efdf405c937830c8f06a4bee299318c88dd30/.git/shallow.lock
hound-bundled/data/vcs-c1b5c0fc085b03452d1881d7a8464d37485aa0e6/.git/shallow.lock
hound-bundled/data/vcs-d5308595e467829b6b19adb97f3eedeb7686d1b4/.git/shallow.lock
hound-deployed/data/vcs-05bdecfc3ad5b9503919309f9825d190899c6f94/.git/shallow.lock
hound-deployed/data/vcs-078dae5115472a663af97b870a7738c1058c77a2/.git/shallow.lock
hound-deployed/data/vcs-19def42b774942cff2bc99087d847d39c547c058/.git/shallow.lock
hound-deployed/data/vcs-285b83401475a40e4d574ac3f0885ec13c62898d/.git/shallow.lock
hound-deployed/data/vcs-38c97027593c6e0cc89f3327f80974563db94805/.git/shallow.lock
hound-deployed/data/vcs-428f30b0f97c4cbd4df7921666b2a14b481896d5/.git/shallow.lock
hound-deployed/data/vcs-5aff22bc3287c9eb5ca54c882e5bf0e6ae6329e9/.git/shallow.lock
hound-deployed/data/vcs-8645d04e6624709ae9a354887f1772fe54d7d8f3/.git/shallow.lock
hound-deployed/data/vcs-9555b74c602d7298c96517808ac71da29033516c/.git/shallow.lock
hound-deployed/data/vcs-9fc14a3a306053fb2c80a5e6afe4dc1d63dd64dd/.git/shallow.lock
hound-deployed/data/vcs-a2f7a56b4a10a4168c4aaeaa1e3d6fb754179407/.git/shallow.lock
hound-deployed/data/vcs-a553edcd387e3d3c0fb8c0e5d1ae4272ada6f5c9/.git/shallow.lock
hound-deployed/data/vcs-e26e246828835c69047947db4a3f3319d77a369f/.git/shallow.lock
hound-deployed/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92/.git/shallow.lock
hound-deployed/data/vcs-ec438a21e45aae69fad7d6adce045e3702560dbc/.git/shallow.lock
hound-deployed/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b/.git/shallow.lock
hound-deployed/data/vcs-f2341417154b058989715b27d1bdc44bd0b491ac/.git/shallow.lock
hound-devtools/data/vcs-129acfe2c75e42d5e55cbc3df4343cd869846348/.git/shallow.lock
hound-devtools/data/vcs-4e69c71401ad0b7e3ba95593a54f1aeb63132db0/.git/shallow.lock
hound-devtools/data/vcs-53eb1f68f321fd97f0c6c3257e1800bf8cd35c6f/.git/shallow.lock
hound-devtools/data/vcs-6ca8dd9f23ee294ba77450c4eca585e2f9c816b0/.git/shallow.lock
hound-devtools/data/vcs-7432d862bd33811c97f8d40001296627c5ffe915/.git/shallow.lock
hound-devtools/data/vcs-7a7c8fcafc07f35ae595c968c80b95d50e136ff1/.git/shallow.lock
hound-devtools/data/vcs-7bf842e76b139520a99452eb4d83a15240b7603e/.git/shallow.lock
hound-devtools/data/vcs-8c0c094cd2515f8feefaeca4f12229335c285829/.git/shallow.lock
hound-devtools/data/vcs-9d2e92b8ecf8b44e6755d6f9f5e63b85fb0772f1/.git/shallow.lock
hound-devtools/data/vcs-e8acf910314ee8ca6d170977f655dc827644e8cd/.git/shallow.lock
hound-extensions/data/vcs-23979644d7333624ccfba824e584d2e04bc42c8b/.git/shallow.lock
hound-extensions/data/vcs-42d1a46398d7d0cc319e32049e09de0b2b4f6b70/.git/shallow.lock
hound-extensions/data/vcs-5da343ea41b5837e83faedba961e8e46cd28f610/.git/shallow.lock
hound-extensions/data/vcs-88a79793e8f4eff5844fd708de0efc0a42004a89/.git/shallow.lock
hound-extensions/data/vcs-9fd4fec7cf8b487966e47f64d76c73e2597dfd98/.git/shallow.lock
hound-extensions/data/vcs-a2f7c1927e6308602b57e840f3c8eb0a2ebb951c/.git/shallow.lock
hound-extensions/data/vcs-b0936e3f40ad139bdb9f34e4adfa9daa0ad99ccd/.git/shallow.lock
hound-extensions/data/vcs-b5c6940fe26235b9e8ebf988708d145ad4275bd7/.git/shallow.lock
hound-extensions/data/vcs-c71bc62cba7f5488194654ced4e5de8e44e71290/.git/shallow.lock
hound-extensions/data/vcs-d5308595e467829b6b19adb97f3eedeb7686d1b4/.git/shallow.lock
hound-extensions/data/vcs-d6b81156d335f8fefaa8e4332f87b02b7eeca6dd/.git/shallow.lock
hound-extensions/data/vcs-ec221c8411478c3a391d1f3999b01d4b05b18f62/.git/shallow.lock
hound-extensions/data/vcs-f97484641d9df450c30ac901793f86aa4c675408/.git/shallow.lock
hound-extensions/data/vcs-fda820ab3b5b2a010c3391b89a59fe1281e345fc/.git/shallow.lock
hound-libraries/data/vcs-1423de0dd4054e43dbb12910892201b1de88f1e3/.git/shallow.lock
hound-libraries/data/vcs-193ecad21f4da1e3030af56c4198c1df0ebc83b7/.git/shallow.lock
hound-libraries/data/vcs-1e54627f86da395b320768e06b4402d8b50b2dd8/.git/shallow.lock
hound-libraries/data/vcs-6515a4550d596e09bc6d7a24dc8418a3e6125354/.git/shallow.lock
hound-libraries/data/vcs-78bd28aba1fff063cfa43bc448196c9a30bec65f/.git/shallow.lock
hound-libraries/data/vcs-b741c5460e00dac2695540dc72c861af795399d2/.git/shallow.lock
hound-libraries/data/vcs-c124c44115da5c5bea4bc4f609e9cc619c6c22b8/.git/shallow.lock
hound-libraries/data/vcs-d6ac8e0f1a1542c1e988ece9dd87ba1220c0b82c/.git/shallow.lock
hound-libraries/data/vcs-ed300c146e5869892ac9a694ac93e85fa6401bb0/.git/shallow.lock
hound-libraries/data/vcs-f0cdf0b6fb191cf8550a4a38464f0ca2c9cc1f4e/.git/shallow.lock
hound-libraries/data/vcs-f2bd1d3b9c20764f7747d228397950314cc356d3/.git/shallow.lock
hound-libraries/data/vcs-f9a5fdf6502975d2d5207d84ad33ecded652180b/.git/shallow.lock
hound-operations/data/vcs-1524bd23979b070eacf827c79cea36286518afb5/.git/shallow.lock
hound-operations/data/vcs-248decefca80499a81403fcaa978fdced0eb37fe/.git/shallow.lock
hound-operations/data/vcs-47de9159bd3e9520af47887d530e0159eae91a05/.git/shallow.lock
hound-operations/data/vcs-49f13ca722ba6520023a22840e31df5a3f921191/.git/shallow.lock
hound-operations/data/vcs-633ff201a8f01024f3729850794496daaad5a659/.git/shallow.lock
hound-operations/data/vcs-86edd3054853fb139633cd5ddb529847eccda9fb/.git/shallow.lock
hound-operations/data/vcs-97db4e845e14bd5e395c8ad53d871d6e89b6e302/.git/shallow.lock
hound-operations/data/vcs-a1a333c5ac6349adea5658b98a4d66f6e6ff1647/.git/shallow.lock
hound-operations/data/vcs-a7bd55849221162ffca255831023bea703515ec4/.git/shallow.lock
hound-operations/data/vcs-dea2489a142228d9019d2c60dd8f004cdfe149cd/.git/shallow.lock
hound-operations/data/vcs-e540e3a4c7861e9b5d9b359192a72928f851d563/.git/shallow.lock
hound-operations/data/vcs-eb9d195cb94d65e4d255be3d9cc3f2a0d5cd9047/.git/shallow.lock
hound-search/data/vcs-0eb9fccab589eefc2b36b77f4278af7f3f733896/.git/shallow.lock
hound-search/data/vcs-16abb82f3b5394413af3b349804353e15bf920f3/.git/shallow.lock
hound-search/data/vcs-18d37ffbae55b652fdd2984c4bc6b4776e863ab3/.git/shallow.lock
hound-search/data/vcs-26a59b5afe2dfca872f793d2c474eefd1e15b447/.git/shallow.lock
hound-search/data/vcs-3b3491a5eb914d353c0c9655d9a528e8137cf0c4/.git/shallow.lock
hound-search/data/vcs-3d3d589c72b32e3f528e18295e83e3152ca80546/.git/shallow.lock
hound-search/data/vcs-3ea7b6317a5c7858449efab5f9e31ba58aa316cd/.git/shallow.lock
hound-search/data/vcs-4500db3b54a9ef19331477ca4dd704ef52990162/.git/shallow.lock
hound-search/data/vcs-4562d92fcdc1f4a91445c3f21b26b259ee98c970/.git/shallow.lock
hound-search/data/vcs-4ec27220a54b9c6e7c408e487cb324da62591df0/.git/shallow.lock
hound-search/data/vcs-544b4cb985bc7f37749494c78afcbdfb44b80f1a/.git/shallow.lock
hound-search/data/vcs-5833d8172b8f167869564fb23f6a935505dcbff2/.git/shallow.lock
hound-search/data/vcs-5d8e31ee84d3a004bf1008c0687fbb23450f0b2f/.git/shallow.lock
hound-search/data/vcs-7926d418c859d5e8c6e32943d262e9dfcf982d11/.git/shallow.lock
hound-search/data/vcs-7b45dbecc6d41c14b9bbb48d30fab346af51c11e/.git/shallow.lock
hound-search/data/vcs-927ccabfa293313cc43f7ce1fb7de4b6cb905ccb/.git/shallow.lock
hound-search/data/vcs-ac8542ff01af6a490546ad7f17b1e8046b7bc491/.git/shallow.lock
hound-search/data/vcs-ae6b08553be313cfd617c51b7b932b20f0a899fd/.git/shallow.lock
hound-search/data/vcs-b704fecdb0f614b843a472a64abfbd3fd39fb29c/.git/shallow.lock
hound-search/data/vcs-c24290e3ae7f737b2a946db9f5659a6cfe289d34/.git/shallow.lock
hound-search/data/vcs-c70018a8dfb66145b6b24d893646bb8eadb3ff66/.git/shallow.lock
hound-search/data/vcs-cf3a85362e8a31c8e501bb44849378977ca4a562/.git/shallow.lock
hound-search/data/vcs-d48c0945fe6d4cd426e2fc011dca647402359793/.git/shallow.lock
hound-search/data/vcs-e1c847680e48c0293166c01f1206d0a37bbc5adb/.git/shallow.lock
hound-search/data/vcs-fdbc221e56c754fc9ab1f713c316d1388fc620a2/.git/shallow.lock
hound-services/data/vcs-428f30b0f97c4cbd4df7921666b2a14b481896d5/.git/shallow.lock
hound-services/data/vcs-476bc1b694939da73469c4aca5d9373864daf5ea/.git/shallow.lock
hound-services/data/vcs-479a6f6ac50f3f0e9b2b93926f2a7b5166d96186/.git/shallow.lock
hound-services/data/vcs-745dfc2892184807fa127b318f24652ab7b4d57d/.git/shallow.lock
hound-services/data/vcs-acaf8617505a7df8949feb2ce1877330c2494a40/.git/shallow.lock
hound-services/data/vcs-b2cd92fac7e54e5bebaf9185d2778f48eddc6d03/.git/shallow.lock
hound-services/data/vcs-b7868e101b05c8926d097aad6a85bd0c383c57b1/.git/shallow.lock
hound-skins/data/vcs-2917132c860e009190612b0c1cfd3de9c79e0f90/.git/shallow.lock
hound-skins/data/vcs-2c528dee5dd361aeb9247cad1ab6a4228ca059d6/.git/shallow.lock
hound-skins/data/vcs-30b3ab9917d928c2862643d65e7be18b693ffc91/.git/shallow.lock
hound-skins/data/vcs-3503a2a1df37ed2b9d17bed5035ea7fe28865b42/.git/shallow.lock
hound-skins/data/vcs-621eb06a5d5302ae43dd53357a98920fa50d986c/.git/shallow.lock
hound-skins/data/vcs-96fa48fd06603024251019dfc94e8232d35979b5/.git/shallow.lock
hound-skins/data/vcs-a5181844d53889c2e88ff4ca9076bd634330b0df/.git/shallow.lock
hound-skins/data/vcs-b20cedb5a6e6f416754eec417a05bc9aed4da62b/.git/shallow.lock
hound-skins/data/vcs-b840125d115e429cce65bbc6f1785e0b5100009e/.git/shallow.lock
hound-skins/data/vcs-cbac0359a33d334b3fc0fc897011bcabef83c036/.git/shallow.lock
hound-skins/data/vcs-ee6de18d916937aea2c20930de178bb6ac599bbf/.git/shallow.lock
hound-skins/data/vcs-f8db585c5d8dc033773eb596abb8407aefc863ee/.git/shallow.lock
hound-skins/data/vcs-fdbc221e56c754fc9ab1f713c316d1388fc620a2/.git/shallow.lock
hound-things/data/vcs-0765c4acdcff97db81c93e624cddece7bc612c6d/.git/shallow.lock
hound-things/data/vcs-2a5e9981ae8bf01116b1af2655979a003309d02b/.git/shallow.lock
hound-things/data/vcs-3481d36fded539d8c05d8e6d2d60495665e5e202/.git/shallow.lock
hound-things/data/vcs-3fdffdbf14cec31bcd97c8dc77bbcc722e5f279f/.git/shallow.lock
hound-things/data/vcs-5ce8d873e1006b8d3bc1a13b1a145b6b6384b64e/.git/shallow.lock
hound-things/data/vcs-6197ecc09134e4cc8fa0b351a67722529bb86a98/.git/shallow.lock
hound-things/data/vcs-69458aa31ad7734b6007cbcb9a17969aadf63d86/.git/shallow.lock
hound-things/data/vcs-72ed3bb1bbc6aef7a5839ebb2bb538d8fa44ffe5/.git/shallow.lock
hound-things/data/vcs-90ee1675672bb7bcf4fdabe564e8dd484c08ee6d/.git/shallow.lock
hound-things/data/vcs-92e2feecb230a25ad7645493a2675e02b5eade11/.git/shallow.lock
hound-things/data/vcs-a1ccb21c21c2f4eec17d2fd1479d678894ac909d/.git/shallow.lock
hound-things/data/vcs-a2821e8f142866fe4ba0acb0e35c0d725f216e35/.git/shallow.lock
hound-things/data/vcs-ec91fcc8b73bef756b7f61cfc19cded06986436d/.git/shallow.lock
hound-things/data/vcs-ec9340734b699128c03453af2b4abe7593890280/.git/shallow.lock
hound-things/data/vcs-f2341417154b058989715b27d1bdc44bd0b491ac/.git/shallow.lock
hound-wmcs/data/vcs-ad5f2059c4f1ce3885eb608d9ed071b507f56d70/.git/shallow.lock

Of these, 17 are in hound-deployed, and guess since when they've been held? Feb 12.

krinkle@codesearch9:/srv/hound$ ls -halF hound-deployed/data/*/.git/shallow.lock  
-rw-r--r-- 1 root root 1.6K Feb  8 13:33 hound-deployed/data/vcs-05bdecfc3ad5b9503919309f9825d190899c6f94/.git/shallow.lock
-rw-r--r-- 1 root root  12K Feb 11 05:41 hound-deployed/data/vcs-078dae5115472a663af97b870a7738c1058c77a2/.git/shallow.lock
-rw-r--r-- 1 root root 8.1K Feb  9 10:06 hound-deployed/data/vcs-19def42b774942cff2bc99087d847d39c547c058/.git/shallow.lock
-rw-r--r-- 1 root root  943 Feb 10 10:41 hound-deployed/data/vcs-285b83401475a40e4d574ac3f0885ec13c62898d/.git/shallow.lock
-rw-r--r-- 1 root root 3.6K Feb 12 15:41 hound-deployed/data/vcs-38c97027593c6e0cc89f3327f80974563db94805/.git/shallow.lock
-rw-r--r-- 1 root root  205 Feb  9 16:58 hound-deployed/data/vcs-428f30b0f97c4cbd4df7921666b2a14b481896d5/.git/shallow.lock
-rw-r--r-- 1 root root  13K Feb 10 03:01 hound-deployed/data/vcs-5aff22bc3287c9eb5ca54c882e5bf0e6ae6329e9/.git/shallow.lock
-rw-r--r-- 1 root root 2.0K Feb 12 15:51 hound-deployed/data/vcs-8645d04e6624709ae9a354887f1772fe54d7d8f3/.git/shallow.lock
-rw-r--r-- 1 root root 3.3K Feb 12 18:58 hound-deployed/data/vcs-9555b74c602d7298c96517808ac71da29033516c/.git/shallow.lock
-rw-r--r-- 1 root root   41 Feb  8 13:26 hound-deployed/data/vcs-9fc14a3a306053fb2c80a5e6afe4dc1d63dd64dd/.git/shallow.lock
-rw-r--r-- 1 root root  11K Feb 11 00:48 hound-deployed/data/vcs-a2f7a56b4a10a4168c4aaeaa1e3d6fb754179407/.git/shallow.lock
-rw-r--r-- 1 root root 1.9K Feb 13 11:59 hound-deployed/data/vcs-a553edcd387e3d3c0fb8c0e5d1ae4272ada6f5c9/.git/shallow.lock
-rw-r--r-- 1 root root   41 Feb  8 13:07 hound-deployed/data/vcs-e26e246828835c69047947db4a3f3319d77a369f/.git/shallow.lock
-rw-r--r-- 1 root root  12K Feb 11 01:05 hound-deployed/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92/.git/shallow.lock
-rw-r--r-- 1 root root 5.1K Feb 12 23:34 hound-deployed/data/vcs-ec438a21e45aae69fad7d6adce045e3702560dbc/.git/shallow.lock
-rw-r--r-- 1 root root 6.1K Feb 11 05:49 hound-deployed/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b/.git/shallow.lock
-rw-r--r-- 1 root root  21K Feb 11 00:50 hound-deployed/data/vcs-f2341417154b058989715b27d1bdc44bd0b491ac/.git/shallow.lock

And for which repos?

krinkle@codesearch9:/srv/hound$ for file in hound-deployed/data/*/.git/shallow.lock; do grep url $(dirname "$file")/config; done
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/TocTree.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/WikimediaEvents.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/WikiEditor.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SearchExtraNS.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/WikibaseCirrusSearch.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/push-notifications.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/SecurePoll.git
	url = https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/TemplateSandbox.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/wikispeech/wikispeech-server.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/GlobalBlocking.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/FileExporter.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/change-propagation/deploy.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/core.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/BetaFeatures.git
	url = https://gerrit-replica.wikimedia.org/r/operations/mediawiki-config.git
	url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ContentTranslation.git

$ sudo rm hound-deployed/data/*/.git/shallow.lock

krinkle@codesearch9:/srv/hound$ sudo systemctl status hound-deployed
* hound-deployed.service - hound-deployed
     Loaded: loaded (/lib/systemd/system/hound-deployed.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-03-24 17:59:50 UTC; 1 week 2 days ago
   Main PID: 3207406 (docker)
      Tasks: 10 (limit: 9515)
     Memory: 15.5M
        CPU: 1min 22.771s
     CGroup: /system.slice/hound-deployed.service
             `-3207406 /usr/bin/docker run -p 6090:6080 --name hound-deployed --user=root -v /srv/hound/hound-deployed:/data -v /etc/hound-gitconfig:/root/.gitconfig docker-registry.wikimedia.org/wikim>

Apr 03 00:05:42 codesearch9 docker[3207406]: remove the file manually to continue.
Apr 03 00:05:42 codesearch9 docker[3207406]: Continuing...
Apr 03 00:16:26 codesearch9 docker[3207406]: 2026/04/03 00:16:26 Failed to git fetch /data/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92, see output below
Apr 03 00:16:26 codesearch9 docker[3207406]: fatal: Unable to create '/data/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92/.git/shallow.lock': File exists.
Apr 03 00:16:26 codesearch9 docker[3207406]: Another git process seems to be running in this repository, e.g.
Apr 03 00:16:26 codesearch9 docker[3207406]: an editor opened by 'git commit'. Please make sure all processes
Apr 03 00:16:26 codesearch9 docker[3207406]: are terminated then try again. If it still fails, a git process
Apr 03 00:16:26 codesearch9 docker[3207406]: may have crashed in this repository earlier:
Apr 03 00:16:26 codesearch9 docker[3207406]: remove the file manually to continue.
Apr 03 00:16:26 codesearch9 docker[3207406]: Continuing...
krinkle@codesearch9:/srv/hound$ sudo systemctl restart hound-deployed
krinkle@codesearch9:/srv/hound$ sudo systemctl status hound-deployed
* hound-deployed.service - hound-deployed
     Loaded: loaded (/lib/systemd/system/hound-deployed.service; enabled; preset: enabled)
     Active: active (running) since Fri 2026-04-03 00:47:30 UTC; 26s ago
    Process: 1165731 ExecStartPre=/usr/bin/docker kill hound-deployed (code=exited, status=1/FAILURE)
    Process: 1165739 ExecStartPre=/usr/bin/docker rm -f hound-deployed (code=exited, status=0/SUCCESS)
    Process: 1165747 ExecStartPre=/srv/codesearch/wait.py (code=exited, status=0/SUCCESS)
   Main PID: 1165752 (docker)
      Tasks: 9 (limit: 9515)
     Memory: 30.5M
        CPU: 564ms
     CGroup: /system.slice/hound-deployed.service
             `-1165752 /usr/bin/docker run -p 6090:6080 --name hound-deployed --user=root -v /srv/hound/hound-deployed:/data -v /etc/hound-gitconfig:/root/.gitconfig docker-registry.wikimedia.org/wikim>

Apr 03 00:47:54 codesearch9 docker[1165752]: 2026/04/03 00:47:54 Searcher started for mediawiki/extensions/EventLogging
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/extensions/LandingCheck
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/extensions/ArticleCreationWorkflow
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/services/wikispeech/symbolset
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/extensions/XAnalytics
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for wikimedia/discovery/discovery-parent-pom
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/extensions/GlobalPreferences
Apr 03 00:47:55 codesearch9 docker[1165752]: 2026/04/03 00:47:55 Searcher started for mediawiki/extensions/NewUserMessage
Apr 03 00:47:56 codesearch9 docker[1165752]: 2026/04/03 00:47:56 Searcher started for mediawiki/extensions/WikibaseQualityConstraints
Apr 03 00:47:56 codesearch9 docker[1165752]: 2026/04/03 00:47:56 Searcher started for mediawiki/extensions/WikiEditor
Apr 03 00:47:56 codesearch9 docker[1165752]: 2026/04/03 00:47:56 Searcher started for mediawiki/extensions/DiscussionTools

Let's see how this goes.

Re-indexing of hound-deployed backend finished. The example at T421147#11745763 now returns fresh results.

I've gone ahead and deleted the git locks in other backends as well.

krinkle@codesearch9:/srv/hound$ sudo rm hound-*/data/vcs-*/.git/shallow.lock

And I've done a rolling restart of the affected backends, which is done meanwhile.

sudo systemctl restart hound-analytics
sudo systemctl restart hound-apps
sudo systemctl restart hound-bundled
sudo systemctl restart hound-devtools
sudo systemctl restart hound-extensions …
sudo systemctl restart hound-libraries
sudo systemctl restart hound-operations
sudo systemctl restart hound-services
sudo systemctl restart hound-skins
sudo systemctl restart hound-things …
sudo systemctl restart hound-wmcs
sudo systemctl restart hound-search …

All up at https://codesearch.wmcloud.org/_health

Now, while I'm closing this task as resolved, this will of course happen again. We know from this that, somehow, upstream Hound can leave behind a lock. This probably happens when it unexpectedly dies or is killed by us for one reason or another, and it either generally doesn't or sometimes fails to complete needed cleanup steps. Andd when it starts back up, it isn't ensuring a clean state.

There may be cases where a Git lock shouldn't simply be removed (not without knowing more about that specific lock and whether another git processes might be running), so ideally this would be fixed upstream. But, given that each Hound instance has its own data directory, and runs in a container with nothing else in it, we can be reasonably sure that, unless the instance is currently starting up, no other git processes are running, so locks are inherently stale from a past failure (i.e. not a concurrent process). I don't know enough about this lock if that means it is always safe to remove, but at least it narrows down the bulk of scenarios where such operation might be unsafe.

That is to say, if you encounter this again, make sure Hound isn't starting up, but otherwise, at least for shallow.lock, it sems fine to remove the locks and restart the service.

Dzahn awarded a token.Apr 3 2026, 12:56 PM

A_smart_kitten mentioned this in T416865: Remove references to MetricsPlatform extension.Apr 11 2026, 9:49 AM

Is this reoccuring (or e.g. did it never stop occurring, at least for operations/mediawiki-config in the 'Everything' group)?

https://codesearch.wmcloud.org/search/?q=MetricsPlatform&files=&excludeFiles=&repos=operations%2Fmediawiki-config&i=fosho returns several results for MetricsPlatform, but FWICS much (if not all) of the code returned by this query was removed in https://gerrit.wikimedia.org/r/1247547 on 2026-03-11.

Clicking through from the results, it looks like Codesearch is referencing the state of the repo as-of https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/3ff486473879abe7eafcf61e60f09784ee5071eb, which is a commit with the date Thu Feb 12 22:24:23 2026 +0000.

Interestingly, doing the same search but under the 'MediaWiki & services at WMF' group, no results are returned: https://codesearch.wmcloud.org/deployed/?q=MetricsPlatform&files=&excludeFiles=&repos=operations%2Fmediawiki-config&i=fosho

In T421147#11810517, @A_smart_kitten wrote:

Is this reoccuring (or e.g. did it never stop occurring, at least for operations/mediawiki-config in the 'Everything' group)?

Can confirm this, I just spent a few WTF seconds wondering why something that was supposed to be there wasn't there. So, boldly reopening.

Ladsgroup mentioned this in T414163: [EXP] Evaluate removing WANObjectCache 'minAsOf' option, in favor of 'touchedCallback'.Apr 14 2026, 2:45 PM

Volker_E subscribed.Apr 14 2026, 4:55 PM

There are no .git/shallow.lock files this time. We instead have a case of .git/index.lock.

Last time, ref T421147#11784788 (and the three comments after that).

krinkle@codesearch9:/srv/hound$ ls  hound-*/data/vcs-*/.git/shallow.lock
ls: cannot access 'hound-*/data/vcs-*/.git/shallow.lock': No such file or directory

krinkle@codesearch9:/srv/hound/hound-search/data$ sudo systemctl status hound-search -n100000 | grep -C10 'mediawiki-config'
Apr 15 06:01:33 codesearch9 docker[3591731]: 2026/04/15 06:01:33 Failed to git reset /data/data/vcs-b3eae4ca7118c7628a2e55d552b653e0fe9809ee, see output below
Apr 15 06:01:33 codesearch9 docker[3591731]: fatal: ambiguous argument 'origin/(unknown)': unknown revision or path not in the working tree.
Apr 15 06:01:33 codesearch9 docker[3591731]: Use '--' to separate paths from revisions, like this:
Apr 15 06:01:33 codesearch9 docker[3591731]: 'git <command> [<revision>...] -- [<file>...]'
Apr 15 06:01:33 codesearch9 docker[3591731]: Continuing...
Apr 15 06:01:33 codesearch9 docker[3591731]: 2026/04/15 06:01:33 exit status 128
Apr 15 06:01:33 codesearch9 docker[3591731]: 2026/04/15 06:01:33 Searcher started for toolforge-repos/archivace-aktualit
Apr 15 06:01:33 codesearch9 docker[3591731]: 2026/04/15 06:01:33 Searcher started for Extension:FundraisingEmailUnsubscribe
Apr 15 06:01:34 codesearch9 docker[3591731]: 2026/04/15 06:01:34 Searcher started for Extension:PageExchange
Apr 15 06:01:36 codesearch9 docker[3591731]: 2026/04/15 06:01:36 Searcher started for mwstake/mediawiki-extensions-PHPEditor
Apr 15 06:01:36 codesearch9 docker[3591731]: 2026/04/15 06:01:36 Searcher started for operations/mediawiki-config
Apr 15 06:01:37 codesearch9 docker[3591731]: 2026/04/15 06:01:37 Failed to git reset /data/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b, see output below
Apr 15 06:01:37 codesearch9 docker[3591731]: fatal: Unable to create '/data/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b/.git/index.lock': File exists.
Apr 15 06:01:37 codesearch9 docker[3591731]: Another git process seems to be running in this repository, e.g.
Apr 15 06:01:37 codesearch9 docker[3591731]: an editor opened by 'git commit'. Please make sure all processes
Apr 15 06:01:37 codesearch9 docker[3591731]: are terminated then try again. If it still fails, a git process
Apr 15 06:01:37 codesearch9 docker[3591731]: may have crashed in this repository earlier:
Apr 15 06:01:37 codesearch9 docker[3591731]: remove the file manually to continue.
Apr 15 06:01:37 codesearch9 docker[3591731]: Continuing...
Apr 15 06:01:37 codesearch9 docker[3591731]: 2026/04/15 06:01:37 Searcher started for analytics/ua-parser/uap-java

So, how widespread is this issue? 45 repos

krinkle@codesearch9:/srv/hound$ ls -hl hound-*/data/vcs-*/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 10:53 hound-analytics/data/vcs-086ef6661557f978a5bd4584c7b6a8c1744e12e0/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 20:12 hound-analytics/data/vcs-2bafa13fc287d6dbba561b48f9e50a587044105d/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 20:10 hound-analytics/data/vcs-a97e2c874ff6e18727907f67290607749a013711/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 10:49 hound-analytics/data/vcs-baeeed715e0d52411ff6580cafd70c685bb6bf5d/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 20:05 hound-analytics/data/vcs-d1b94d92fd31ac1ec2fa97ab1b20287c07544a55/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 00:54 hound-apps/data/vcs-2a54726bf2a68fdec0390aeda27c13a3a491a842/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 00:54 hound-apps/data/vcs-7de76661721b0e242ca934bb1e1e542e13b19976/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 01:54 hound-bundled/data/vcs-0f8997c4f84abec520591c6f768e0a248aaaac0c/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 20:42 hound-bundled/data/vcs-19def42b774942cff2bc99087d847d39c547c058/.git/index.lock
-rw-r--r-- 1 root root  24K Feb 11 08:51 hound-bundled/data/vcs-1bdcf83cb0685a031d2f2ea57d81ad61ab2bf386/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:20 hound-bundled/data/vcs-3503a2a1df37ed2b9d17bed5035ea7fe28865b42/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 09:30 hound-bundled/data/vcs-4fa4b0a0213985eda914a282d566c95945b96c90/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:19 hound-bundled/data/vcs-7c91571fdbba94d25f3d69eefae0b625ba3f6867/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 01:59 hound-bundled/data/vcs-c1b5c0fc085b03452d1881d7a8464d37485aa0e6/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 21:18 hound-bundled/data/vcs-e2fbd330d32d7def1f4e586f769b73e7cbd15e92/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 08:14 hound-deployed/data/vcs-62f9bddf903a353c6129447be96a71eab5f9c50b/.git/index.lock
-rw-r--r-- 1 root root 144K Feb 10 10:44 hound-deployed/data/vcs-7c929b693214537592f4d8fafcc89c71f47e299d/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 05:41 hound-deployed/data/vcs-984844e9a9ab8b1472f555d43430cc12804cb385/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:23 hound-deployed/data/vcs-ab5b207527c331d9c18d6e95c59219f01cc21ec4/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:24 hound-deployed/data/vcs-eff90015512182fb945eb1dbb545709c2e86421a/.git/index.lock
-rw-r--r-- 1 root root 1.1K Feb 10 19:11 hound-devtools/data/vcs-02def5efaf7cb9e875432740409af459e58c6de1/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 14:19 hound-devtools/data/vcs-1027be7f6697d24340b08242b3f11e238505fad5/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 22:36 hound-devtools/data/vcs-53eb1f68f321fd97f0c6c3257e1800bf8cd35c6f/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 08:33 hound-extensions/data/vcs-00860fb624417fa67c142b820ba63895fe013528/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 08:31 hound-extensions/data/vcs-7139d024f1e52a8042f41a83a5a69cfe93d0fdcc/.git/index.lock
-rw-r--r-- 1 root root    0 Mar 28  2025 hound-extensions/data/vcs-b6c9d8cd2f4dbc488bc72dd6a394966a2e01a837/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 09:09 hound-libraries/data/vcs-e8c572d18a8a67deb8880d9d63019899ad337bd8/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 12 07:46 hound-operations/data/vcs-aa1e386bba98c9be8151c05472b9b46ac818dfc2/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 06:55 hound-operations/data/vcs-b9d5a0c852c5828701f14485c792f5f16d6023c2/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:27 hound-operations/data/vcs-d48c0945fe6d4cd426e2fc011dca647402359793/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 22:32 hound-operations/data/vcs-e653b385b838940c6b469b6d1034e62c39980e99/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 13:46 hound-search/data/vcs-0cfe8f6eab278627ae13fde33de26560f6e04660/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 13:31 hound-search/data/vcs-15f99d198990813c020a5d012f4a9e21cc0f4655/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 07:11 hound-search/data/vcs-2a54726bf2a68fdec0390aeda27c13a3a491a842/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 03:53 hound-search/data/vcs-3a94306d7646131028821817372b9127bdcf130b/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 10 07:44 hound-search/data/vcs-6c6fdbbd04aaa5cc6d843bc70e4ae852bebcb21e/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 13:52 hound-search/data/vcs-ab3ce89bd112de7bf1663c0b1632f967bebbd990/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 13 11:31 hound-search/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 12 19:03 hound-services/data/vcs-4562d92fcdc1f4a91445c3f21b26b259ee98c970/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 12 09:13 hound-skins/data/vcs-b20cedb5a6e6f416754eec417a05bc9aed4da62b/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 12 09:02 hound-skins/data/vcs-bfe9c02cc5676a109a7f5f2c31e9e9f7178f77ff/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 09:04 hound-things/data/vcs-18d8038c8a6734ff89b1581ab7f7a1c14e0fd69d/.git/index.lock
-rw-r--r-- 1 root root    0 Feb  9 08:19 hound-things/data/vcs-42d1a46398d7d0cc319e32049e09de0b2b4f6b70/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 12 17:44 hound-things/data/vcs-f2341417154b058989715b27d1bdc44bd0b491ac/.git/index.lock
-rw-r--r-- 1 root root    0 Feb 11 16:38 hound-wmcs/data/vcs-c98a46468a97cc089d21fb2b9206e3a3562ecd3a/.git/index.lock

krinkle@codesearch9:/srv/hound$ for file in hound-*/data/*/.git/index.lock; do echo $(realpath $(dirname "$file")/../../..) $(grep url $(dirname "$file")/config); done
/srv/hound/hound-analytics url = https://gitlab.wikimedia.org/repos/data-engineering/patches/wmf-sparksqlclidriver.git
/srv/hound/hound-analytics url = https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary.git
/srv/hound/hound-analytics url = https://gerrit-replica.wikimedia.org/r/analytics/turnilo/deploy.git
/srv/hound/hound-analytics url = https://gerrit-replica.wikimedia.org/r/analytics/pivot/deploy.git
/srv/hound/hound-analytics url = https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git
/srv/hound/hound-apps url = https://github.com/wikimedia/wikipedia-ios
/srv/hound/hound-apps url = https://github.com/commons-app/apps-android-commons
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/CiteThisPage.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/WikiEditor.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/vendor.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/skins/Vector.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Scribunto.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/skins/MinervaNeue.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/OATHAuth.git
/srv/hound/hound-bundled url = https://gerrit-replica.wikimedia.org/r/mediawiki/core.git
/srv/hound/hound-deployed url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/wikispeech/mary-tts.git
/srv/hound/hound-deployed url = https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata.git
/srv/hound/hound-deployed url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/MobileFrontend.git
/srv/hound/hound-deployed url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/FlaggedRevs.git
/srv/hound/hound-deployed url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/TestKitchen.git
/srv/hound/hound-devtools url = https://gerrit-replica.wikimedia.org/r/mediawiki/tools/generateHookInterfaces.git
/srv/hound/hound-devtools url = https://gerrit-replica.wikimedia.org/r/integration/tox-wikimedia.git
/srv/hound/hound-devtools url = https://gerrit-replica.wikimedia.org/r/mediawiki/vagrant.git
/srv/hound/hound-extensions url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Spark.git
/srv/hound/hound-extensions url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/EmailAuth.git
/srv/hound/hound-extensions url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Monstranto.git
/srv/hound/hound-libraries url = https://gerrit-replica.wikimedia.org/r/design/codex.git
/srv/hound/hound-operations url = https://gerrit-replica.wikimedia.org/r/operations/puppet.git
/srv/hound/hound-operations url = https://gerrit-replica.wikimedia.org/r/operations/docker-images/docker-report.git
/srv/hound/hound-operations url = https://gitlab.wikimedia.org/repos/sre/liberica.git
/srv/hound/hound-operations url = https://gerrit-replica.wikimedia.org/r/performance/docroot.git
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/performance/xhgui-vendor.git
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/analytics/aqs/deploy.git
/srv/hound/hound-search url = https://github.com/wikimedia/wikipedia-ios
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/mathjax.git
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/UniversalLanguageSelector.git
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/BlueSpiceExtendedSearch.git
/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/operations/mediawiki-config.git
/srv/hound/hound-services url = https://gerrit-replica.wikimedia.org/r/mediawiki/services/parsoid.git
/srv/hound/hound-skins url = https://github.com/thaider/Tweeki
/srv/hound/hound-skins url = https://github.com/wiki-chan/ModernSkylight
/srv/hound/hound-things url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/Wikibase.git
/srv/hound/hound-things url = https://github.com/SemanticMediaWiki/SemanticMediaWiki
/srv/hound/hound-things url = https://gerrit-replica.wikimedia.org/r/mediawiki/extensions/ContentTranslation.git
/srv/hound/hound-wmcs url = https://gitlab.wikimedia.org/toolforge-repos/digero.git

Including the one that @A_smart_kitten reports:

-rw-r--r-- 1 root root    0 Feb 13 11:31 hound-search/data/vcs-f0a1d28c75f7394589b74dd9a2580b39c489490b/.git/index.lock

…

/srv/hound/hound-search url = https://gerrit-replica.wikimedia.org/r/operations/mediawiki-config.git

I've manually deleted these lock files and restarted two the affected indexes (Everywhere "search", and MW tarball "bundled"). The rest will naturally reindex in a few hours around UTC midnight.

krinkle@codesearch9:/srv/hound$ sudo rm hound-*/data/vcs-*/.git/index.lock
krinkle@codesearch9:/srv/hound$ sudo systemctl restart hound-search
krinkle@codesearch9:/srv/hound$ sudo systemctl restart hound-bundled

Confirming that, for me, https://codesearch.wmcloud.org/search/?q=MetricsPlatform&files=&excludeFiles=&repos=operations%2Fmediawiki-config&i=fosho (from T421147#11810517) now returns no results.

Should we re-resolve this for now, or e.g. is there still more that needs to be done?

Volker_E unsubscribed.Apr 19 2026, 4:40 PM

CDanis subscribed.May 7 2026, 5:08 PM

Change #1285488 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: create script/timer to delete zombie lock files

https://gerrit.wikimedia.org/r/1285488

gerritbot added a project: Patch-For-Review.May 9 2026, 2:17 PM

Change #1285488 merged by Dzahn:

[operations/puppet@production] codesearch: create script/timer to delete zombie lock files

https://gerrit.wikimedia.org/r/1285488

I deployed the code change above. This created a new systemd service and timer:

root@codesearch9:~# systemctl list-units | grep zombie

  codesearch-delete-zombie-locks.service                                                                loaded activating start        start Delete lock files when no process is running
  codesearch-delete-zombie-locks.timer                                                                  loaded active     running            Periodic execution of codesearch-delete-zombie-locks.service

This runs a shell script every 10 minutes:

root@codesearch9:~# systemctl cat codesearch-delete-zombie-locks.service

# /lib/systemd/system/codesearch-delete-zombie-locks.service
[Unit]
Description=Delete lock files when no process is running
Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/delete-hound-zombie-locks.sh

This shell script has a config file /etc/hound-delete-zombie-locks.conf and runs a find command to delete lock files older than 60 minutes.

If a lock file has not been removed after that time we assume a git operation was killed somehow.

find "${BASE_DIR}" \
    \( -name "index.lock" -o -name "shallow.lock" \) \
    -path "*/.git/*" \
    -type f \
    -mmin +"${MIN_AGE}" \
    -print \
    -delete

A problem might just be how long that find actually runs.

Currently testing it.

Maintenance_bot removed a project: Patch-For-Review.Tue, May 19, 10:30 PM

Change #1289433 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: let the lock file cleanup run only once an hour

https://gerrit.wikimedia.org/r/1289433

Change #1289433 merged by Dzahn:

[operations/puppet@production] codesearch: let the lock file cleanup run only once an hour

https://gerrit.wikimedia.org/r/1289433

Yea, so the puppet code and find command work as intended.. just that it took 13 minutes to run one time.

So could not leave the timer at "every 10 minutes" and increased to 1 hour.

So now it's "check every hour if a lock file is older than an hour", both 60 minutes.

Hope this helps to prevent another outage.. well.. makes it shorter.

Maintenance_bot removed a project: Patch-For-Review.Tue, May 19, 11:31 PM

Change #1290025 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: fix invalid calendar format on cleanup systemd timer

https://gerrit.wikimedia.org/r/1290025

Change #1290025 merged by Dzahn:

[operations/puppet@production] codesearch: fix invalid calendar format on cleanup systemd timer

https://gerrit.wikimedia.org/r/1290025

Maintenance_bot removed a project: Patch-For-Review.Wed, May 20, 5:31 PM

Codesearch stuck at Feb 12th?Open, Needs TriagePublicBUG REPORTActions

Description

Details

Related Objects

Event Timeline

Codesearch stuck at Feb 12th?
Open, Needs TriagePublicBUG REPORT
Actions