Page MenuHomePhabricator

Headless browser
Closed, DeclinedPublic

Description

Would it be possible to install some modern browser with headless capability (Chrome, maybe Chromium or Firefox if licensing issues) on the grid?

Event Timeline

Headless browsers come with a lot of dependencies. I believe we've rejected suggestions for that in the past.

I don't see any dedicated headless packages for chromium/firefox in debian repos or apt.wm.o, CI uses regular chromium/firefox-esr packages, which would involve pulling down a fairly large pile of dependencies.

The use case is https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/ProcBot_8. No API endpoints exist for editing abusefilters. So I decided to mimic the HTTP requests, but that approach can turn up a few caveats and is slightly less maintainable. I cooked up a web scraping approach using Selenium as well and that might work better. But to run it'll need a browser with headless and modern JS support (both Chrome and Firefox work, so I mentioned those two; either will work).

I once downloaded the Firefox binary and it mostly worked (this was using slimerjs back when that was still usable) - https://www.mozilla.org/en-US/firefox/all/#product-desktop-release

I mean, you can always download the binary to your tool's home dir and see if that will work headlessly without the libraries for x11. That would be much lighter than installing X libraries on everything.

We have Xvfb on the grid too which may or may not be useful in running some browser. The ws-export tool used to use it to run Calibre on the grid.

I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation ]]. Please see the instructions page on how to migrate your tool to Kubernetes. Thanks.