Steps to reproduce

Paste a URL into Citoid (e.g. https://myserver.com/foo). Look at the access logs for the server (say, myserver.com) for access to the resource (/foo).

Observed behaviour

There are two HTTP GET requests for the resource: the first with Citoid in the user agent string, and the second with Zotero in the user agent string. That means the entire resource gets downloaded twice. Eg. for a 30 MB resource, there would be a total of 60 MB of data transfer.

Expected behaviour

Only one GET request should happen.

Cause

Before handing a URL to the Zotero service to download, Citoid tests the URL itself to see whether the response is a redirection to another URL. If so, Zotero will be given the redirected URL rather than the original URL. However, the way Citoid tests the URL is using HTTP GET, i.e. it requests and downloads the entire resource. Then when Zotero gets the URL, it downloads the resource a second time.

Fix

We can change the Citoid redirection testing to use HTTP HEAD instead of HTTP GET, which doesn't fetch the entire resource (only information about the resource).

Based on source code comments, the historical reason we use GET is in case HEAD might not return correct redirection information if the resource is on some misconfigured third-party server. But in the light of T362379, it doesn't seem acceptable to impose double the data transfer on every third-party server we access, just in case some third-party server is misconfigured. (And there's no security advantage to using GET, because in any case we cannot stop a nefarious server serving different responses to our two identical GET requests).