Maniphest T319241

IABot not getting past Cloudflare protection
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Harej
	Oct 3 2022, 9:02 PM

Description

https://journals.sagepub.com/doi/abs/10.1177/03058298880170020901

https://en.m.wikipedia.org/wiki/Special:MobileDiff/1113580423?diffmode=source

Event Timeline

Harej created this task.Oct 3 2022, 9:02 PM

Restricted Application added a subscriber: Cyberpower678. · View Herald TranscriptOct 3 2022, 9:02 PM

Harej triaged this task as High priority.Oct 3 2022, 9:02 PM

Turns out while IABot could have used better HTTP/2 handling, this appears to be very very sophisticated CloudFlare protection, that is able to distinguish cURL requests from actual browser requests even if a perfectly imitated browser-generated cURL request is made.

Will have to go to plan B which is implement Chromium support which was on my todo list, but a lesser priority.

Cyberpower678 moved this task from Backlog: URLs to Next on the InternetArchiveBot board.Oct 11 2022, 5:42 PM

Cyberpower678 moved this task from Next to Current Work on the InternetArchiveBot board.Oct 11 2022, 8:47 PM