Change #1034555 merged by jenkins-bot:

[mediawiki/services/citoid@master] Switch back to using HEAD instead of GET for redirect tracking

https://gerrit.wikimedia.org/r/1034555

@Robertsky We heard back from the property owner who showed us graphs of traffic and traffic levels and said that this was clearly more than what they considered acceptable. Their concern was purely about volume and not about suspicious looking user agent strings.

Update: 14 June

We've updated the task description to include the Strategy and State section that's meant to help us all understand:

1. The strategy that is guiding the action we're taking to address this issue
2. The state of the "actions" mentioned in "1." (above)

If anything you see brings questions/ideas to mind, we'd value knowing.

In T362379#9890395, @SCherukuwada wrote:

@Robertsky We heard back from the property owner who showed us graphs of traffic and traffic levels and said that this was clearly more than what they considered acceptable. Their concern was purely about volume and not about suspicious looking user agent strings.

And what kind of volume were they seeing?

And just to be sure: there is no excessive traffic generated by some random (non-Wikimedia) spider/harvester operator who might be using the Citoid user agent?

Why are the details about the problems sites report so vague? 😅
Did T367452 resolve any of these sites problems? Every visit being a double-download might have been enough of a spike for some of them to block.

As editors visit the pages themselves while reading, a longer term solution might be something that captures cites client-side while browsing and lets you post those to a wikibase as Mvolz mentioned, then checks that base first in citoid before pinging out to the web.

For instance, when I'm doing writing that involves dozens of PDFs, the last thing I want is to generate all those cites by hand. But could quickly capture them all while reading them and then insert them while editing.

In T362379#9913315, @Sj wrote:

Why are the details about the problems sites report so vague? 😅

@Sj can you say a bit more what you mean by "vague" in this context? What might you expect us to be able to know about the problems that you've not seen documented?

Did T367452 resolve any of these sites problems? Every visit being a double-download might have been enough of a spike for some of them to block.

Great question! We're investigating this as part of T368802 and will share what we learn...

For instance, when I'm doing writing that involves dozens of PDFs, the last thing I want is to generate all those cites by hand. But could quickly capture them all while reading them and then insert them while editing.

Oh! What a nifty idea...

Could you please read the newly-created T368980 and boldly edit the description to align with what you had in mind...?

Thanks Peter, I commented on T368980; the description is fine.

Vagueness:

• "told by at least one organisation that the block is deliberate." (which orgs?)
• "owners of one property said that the traffic pattern looks like abuse... showed us graphs of traffic... [t]heir concern was purely about volume.
" (what patterns? how much volume?) - @AlexisJazz asked about this above

One could imagine smoke tests of our outgoing traffic against a glossary of patterns we've tried to avoid in the past.

Another user report: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Adding_Hindustan_Times_sources

Looks like Hindustan Times potentially is IP blocking us as it works locally but not in production.

FYI, it seems that straitstimes.com may be blocked now as well.

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

In T362379#10343726, @Mvolz wrote:

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

Tested locally and it doesn't work either - might have been an infrastructure change rather than a deliberate block.

In T362379#10343746, @Mvolz wrote:

In T362379#10343726, @Mvolz wrote:

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

Tested locally and it doesn't work either - might have been an infrastructure change rather than a deliberate block.

Was the user agent string changed around then as well? If so, maybe we can drop an email to them to see if they can unblock the user agent string?

In T362379#10351488, @Robertsky wrote:

In T362379#10343746, @Mvolz wrote:

In T362379#10343726, @Mvolz wrote:

In T362379#10343714, @Robertsky wrote:

FYI, it seems that straitstimes.com may be blocked now as well.

Yup. Very clear block starting Nov 5th 2024 visible: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3M%2Cto%3Anow))

We're not sending them more than 30 requests/day so :/ The day they blocked us, only 3 requests!

Tested locally and it doesn't work either - might have been an infrastructure change rather than a deliberate block.

Was the user agent string changed around then as well? If so, maybe we can drop an email to them to see if they can unblock the user agent string?

Sorry, that was ambiguous - I meant an infrastructure change on their end, maybe, because it doesn't work on a different IP and with a different user-agent either.

In T362379#10356773, @Mvolz wrote:

Sorry, that was ambiguous - I meant an infrastructure change on their end, maybe, because it doesn't work on a different IP and with a different user-agent either.

Interesting. This works with https://citer.toolforge.org though. fyi, their sister site, zaobao.sg also cannot be parsed through citiod.

papers.ssrn.com is another one: (user report here). This appears to be an actual block, AFAIK. https://www.mediawiki.org/wiki/Topic:Ybg5cq7nal5z0dom

In T362379#10356782, @Robertsky wrote:

Interesting. This works with https://citer.toolforge.org though. fyi, their sister site, zaobao.sg also cannot be parsed through citiod.

This might not be the case here, but almost a year ago, I started to see many sites becoming inaccessible to citer. After some investigation, I found that the issue was mostly caused by TLS fingerprinting processes used by Cloudflare servers, which refuse to respond to non-browser-like TLS handshakes. As a workaround, citer now uses curl_cffi, which is a Python binding for a fork of curl-impersonate which is able to perform TLS and HTTP handshakes in a manner identical to that of real browsers. This change rectified citer's issue for many such sites.

In T362379#10383665, @Dalba wrote:

In T362379#10356782, @Robertsky wrote:

Interesting. This works with https://citer.toolforge.org though. fyi, their sister site, zaobao.sg also cannot be parsed through citiod.

This might not be the case here, but almost a year ago, I started to see many sites becoming inaccessible to citer. After some investigation, I found that the issue was mostly caused by TLS fingerprinting processes used by Cloudflare servers, which refuse to respond to non-browser-like TLS handshakes. As a workaround, citer now uses curl_cffi, which is a Python binding for a fork of curl-impersonate which is able to perform TLS and HTTP handshakes in a manner identical to that of real browsers. This change rectified citer's issue for many such sites.

That's good to know! We're definitely also seeing that for a lot of websites. We're hoping registering as a friendly bot with cloudflare T370118 will ameliorate things... but this hasn't happened yet.