One potential concern that comes to mind regarding an OAuth gate is whether it would add a noticeable amount of time to (what is currently able to be) a very quick experience when using Codesearch, e.g. if someone wants to quickly perform a search for a specific string.

Another potential idea I mentioned in T403323#11134042 is to put Codesearch behind Anubis. In my own personal experience, the Anubis check when visiting XTools (xref T400229) is very quick to complete; but I don't know either way regarding whether or not that's reflective of most users' experiences.

FWIW a good write up why we shouldn't use Anubis: https://lock.cmpxchg8b.com/anubis.html

In T403474#11139123, @Ladsgroup wrote:

FWIW a good write up why we shouldn't use Anubis: https://lock.cmpxchg8b.com/anubis.html

To be fair, the idea that AI companies would have enough money/compute to get past Anubis (and/or that it'd cost a trivial amount of money to do so) seems like a valid critique... but at a first glance, if that was happening, then (AFAICS) it wouldn't seem to explain the massive ("amazeballs"!) success that XTools has apparently seen since deploying it (xref the Conclusion section in T400229: Add Anubis to XTools's task description; https://grafana.wmcloud.org/goto/PGMRMH9Hg?orgId=1). Granted, though, XTools only added Anubis relatively recently, so I guess time will tell whether it keeps working for it as well as it currently does.

[disclaimer: I haven't looked into Anubis in detail, this is just a first reaction to reading through that post!]

imho the real fix should be in the direction of somehow unifying the approaches used in production and in cloud VPS/toolforge to how we deal with this problem.

Because basically every single service has it and implementing different solutions for it across different teams / departments or reinventing the wheel seems suboptimal.

Ideally we should have requestctl for cloud VPS and toolforge. Maybe a copy of it could create VCL used in cloud VPS domain proxies.

(the other real fix would be to move tools that we rely on and aren't just testing to actual production.. fully aware that is easier said than done but imho that we basically don't have that pipeline is a root cause too)

In T403323#11134019, @Ladsgroup wrote:

[…]
I do see a lot of garbage views in logs done by bots which could be the reason it's ooming

The current Codesearch frontend doesn't run queries on GET, it relies on client-side JavaScript to start the query. This means most scapers and bots don't add load to Hound.

I was working on a refactor that improves performance and removes the JavaScript requirement, by formatting the results server-side instead. However, even in that model I intend to keep it so that it doesn't make expensive queries on GET. Instead, it would require a POST submission and there'd continue to be an inline script that auto-submits the form in real browsers.