Page MenuHomePhabricator

CodeSearch: repositories are randomly disappearing from hound
Closed, ResolvedPublic

Event Timeline

I am not sure if I am doing something wrong but I couldn't find UW in my search results when it was showing on Github:

https://github.com/search?utf8=%E2%9C%93&q=getSelectedItemsData%20org%3Awikimedia&type=Code
https://codesearch.wmflabs.org/search/?q=getSelectedItemsData&i=nope&files=&repos=

I also couldn't find UW in the SelectRepo select box:

Screen Shot 2018-02-28 at 10.08.25 AM.png (506×1 px, 124 KB)

So....something is going very wrong. This is basically the same as T188450: CodeSearch: SocialProfile extension is not being indexed by Codesearch that I thought I fixed with a hound restart earlier.

OK, so if I use the "extensions" hound instance (which I restarted earlier), then UW shows up: https://codesearch.wmflabs.org/extensions/?q=getSelectedItemsData&i=nope&files=&repos=

Mentioned in SAL (#wikimedia-cloud) [2018-02-28T04:44:14Z] <legoktm> restarted hound-search service (T188477), also I restarted hound-extensions earlier (T188450)

Restart fixed it. But we need to figure out why this is happening.

Legoktm renamed this task from CodeSearch: Add Extension:UploadWizard to CodeSearch: repositories are randomly disappearing from hound.Feb 28 2018, 4:47 AM

I just restarted everything so we should have baselines and logging info now onwards to watch on https://tools.wmflabs.org/nagf/?project=codesearch

Legoktm claimed this task.

What happened was that the / partition on codesearch2 ran out of inodes, so the disk was full even though it had more space. I recreated the instance, this time creating a special /srv partition, and then used a command that bd808 found to quintuple the number of inodes. We should have enough headroom for a decent amount of growth now. I'll also document the settings I used in the README for next time.