Context
In logstash we saw failing request logged from the Citoid service that refered to URLs including PDFs, logged as 200 with a log level 'Warning'. When testing some of these URLs manually we saw them succeeding in the interface of the Citoid extension though.
Investigation
We assume, that his is happening due to different approaches running in parallel. Some of them might succeed and some not. If one fails including the PDF we'll still see the issue in logstash but on the users end it's fine.
We might figure out a way to make that more transparent in logstash so that we have a more reliable source on how many user request fail.
Example URIs
Further ressoures
Logstash https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=h@c823129&_a=h@c75efd5
Citoid Grafana https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=now-30m&to=now&timezone=utc&var-dc=000000026