Page MenuHomePhabricator

Reconsider Citoid and Zotero using different user agents
Closed, DeclinedPublic

Description

T367452 revised how Citoid emits requests by shifting:

  • From two GET requests
  • To a HEAD (metadata) request plus a GET (full) request.

In doing the above we did NOT make any changes to the user agents generating these requests.

We've come to learn the user agents generating these requests are causing some proportion of web application firewalls to block Citoid requests. [i]

This task involves the work of revising this user agent behavior to avoid web application firewalls and as a result, reducing the rate at which Citoid requests fail.


i. Thank you to @akosiaris for making us aware of this issue

Event Timeline

Adding for transparency that a user informed us via email that, on their system, mod_security flagged Citoid+Zotero traffic (the HEAD+GET thing). The weren't able to provide more information as to which rule actually triggered, nor which rule set it was from. They made the point that bad robots are known to cycle through lists of UAs, which is, to my experience, a correct point.

Esanders renamed this task from Reconsider different user agents for Citoid and Zotero to Reconsider Citoid and Zotero using different user agents.Sep 11 2024, 12:51 PM

Should we trial this for a month or something? I don't like it for the long term but maybe worth seeing if there's an impact?