I can confirm these urls work locally, but in production we're getting 504 gateway timeout errors. It's been ongoing as long as we have data (3 months).

This is slightly weird error code for a deliberate block, but we could ask them.

I've had issues with CBC auto citation for many many years now, was wondering if any progress has been made on this (new to Phabricator and have no technical knowledge to speak of, but seeing this fixed would vastly improve QoL for Canadian editors)

Would love to see this working aswell.

@Mvolz so we are having trouble with specific news sources blocking our tools from generating citations from them?

In T415981#12281152, @Doc_James wrote:

@Mvolz so we are having trouble with specific news sources blocking our tools from generating citations from them?

I'm not sure from my end whether it's a 504 error or something else, the auto citation simply points out the failure to generate. If it is a 504 error and that implies it's something on the news outlets side, then I suppose there's not much to do here, and I can reach out to the CBC directly.

I have sent the CBC a note, asking them to whitelist.

I can confirm this does look like a deliberate block based on our user-agent.

In T415981#12281152, @Doc_James wrote:

@Mvolz so we are having trouble with specific news sources blocking our tools from generating citations from them?

Yes, added the umbrella task accordingly.

We are not a real browser so a lot of bot protection tools and so forth detect us as a bot, also as the popularity of the service has increased we get increased traffic, so sometimes the traffic (or traffic patterns) can also trigger a block. We've gotten a few publishers to allowlist us specifically, but it's a bit of a whack a mole issue!

Found this issue while working on an unrelated fix for the CBC Zotero translator. I sent a request to the CBC Help Centre Contact Us page, under Technical -> CBC Website -> Trouble accessing content (error codes, missing content...). Another option may have been CBC News -> Technical -> News website -> Articles / NA.

Hopefully enough people contact them (or a senior-enough person at WMF does) that they unblock the citation crawlers.