Page MenuHomePhabricator

Fix Newspapers.com Library Bundle Configuration issues
Open, In Progress, HighPublic

Assigned To
Authored By
sjvipin
Nov 11 2022, 10:48 AM
Referenced Files
F41721934: image.png
Jan 27 2024, 7:22 AM
F41436405: image.png
Nov 3 2023, 4:06 AM
F37819456: Screenshot_20230927-201747.png
Sep 28 2023, 3:25 AM
F37817066: image.png
Sep 27 2023, 1:15 PM
F36890219: image.png
Mar 3 2023, 2:32 PM
F36886800: image.png
Mar 1 2023, 1:33 PM
F36886136: Screenshot 2023-03-01 at 10.47.58.png
Mar 1 2023, 10:50 AM
F36885788: Screenshot 2023-03-01 at 10.45.27.png
Mar 1 2023, 10:50 AM
Tokens
"Love" token, awarded by HouseBlaster."Like" token, awarded by Novem_Linguae.

Description

Current issues

  • We do not currently seem to have access to some Publisher Extra content
  • Users cannot log in via Ancestry
  • Users cannot log in via Facebook
  • Users cannot register a new account via Email

Past issues
Cloudflare is preventing access. Users get stuck at www.newspapers.com needs to review the security of your connection before proceeding.

Users currently cannot log out of their account while proxied, or log in via Ancestry.

This change has now been deployed, however we are seeing some access issues. Some users cannot retrieve search results, and logging in to accounts sometimes doesn't work.

Users can't login as they see the following error:

image.png (133×482 px, 13 KB)

Newspapers.com fixed the initial issue here, which was enabling reCaptcha, however we're now getting a slightly different error about an "invalid domain key". We should be able to fix this issue, and are investigating.

Original task
Our existing partner newspapers.com has agreed to move to library bundle. We need to update their EZProxy configuration.

Partner id: 26 (https://wikipedialibrary.wmflabs.org/partners/26/)

Before making this change please send me a list of the user emails (privately) with active authorisations so I can let them know.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Confirming that I'm seeing this too.

image.png (831×981 px, 27 KB)

Marked as temporarily unavailable in the library, linking to this task. We'll update as we investigate this issue.

When we set this proxy-based access up Newspapers.com did some work to get us around Captchas. It looks like they might have changed something causing that to now fail again. We're going to contact them to see if they can resolve the issue, we don't think we can do anything on our end.

jsn.sherman changed the task status from In Progress to Stalled.Sep 27 2023, 3:15 PM

Now that we have reached out, I'm marking as stalled while we wait for a response.

Screenshot_20230927-201747.png (1×720 px, 151 KB)
Email I received from support@newspapers.com on the current subject.

Apparently, newspapers.com, to believe the email, wants us to order a subscriprion to the library edition of the site from ProQuest via a direct request from the latter that can be made here: http://www.proquest.com/libraries/academic/news-newspapers/Newspaperscom-Library-Edition.html

Is this good advice?

I don't know how to say this is happening to me too so I'll just repeat what was said.

I received an error message reading "www.newspapers.com needs to review the security of your connection before proceeding" in an attempt to use this website via the library. Can you fix it?

I tried signing in the normal way and am told I need a subscription.

The only solution to our dilemma offered so far by newspapers.com is that of directly contacting ProQuest about a subscription to the website at the link below:

https://about.proquest.com/en/products-services/Newspaperscom-Library-Edition/

ProQuest also offers databases for over forty-five Historical Newspapers and for some Recent Newspapers, and those databases may be added to provide Wikipedia Library users access to other newspapers while the issue of access to newspapers.com is being resolved. The links are as follows:

  1. https://about.proquest.com/products-services/pq-hist-news.html
  2. https://about.proquest.com/en/products-services/ProQuest-Recent-Newspapers/

The only solution to our dilemma offered so far by newspapers.com is that of directly contacting ProQuest about a subscription to the website at the link below:

https://about.proquest.com/en/products-services/Newspaperscom-Library-Edition/

ProQuest also offers databases for over forty-five Historical Newspapers and for some Recent Newspapers, and those databases may be added to provide Wikipedia Library users access to other newspapers while the issue of access to newspapers.com is being resolved. The links are as follows:

  1. https://about.proquest.com/products-services/pq-hist-news.html
  2. https://about.proquest.com/en/products-services/ProQuest-Recent-Newspapers/

Does TWL not already subscribe to Newspapers.com through ProQuest? Did you make clear that you already have an institutional subscription and aren't looking for a new one? The email you received makes it sound like the support person didn't fully understand your query.

Sent a new email to support@newspapers.com with this clarification.

Perhaps it'd be better to let our people work on this (WMF Moderator Tools team talking to their contact at Newspapers.com) instead of Wikipedia Library users interfacing with Newspapers.com directly. Interfacing with them directly might just confuse and bother their customer service reps.

Perhaps it'd be better to let our people work on this (WMF Moderator Tools team talking to their contact at Newspapers.com) instead of Wikipedia Library users interfacing with Newspapers.com directly. Interfacing with them directly might just confuse and bother their customer service reps.

Agreed - we have an email directly out to tech folks at Newspapers.com who are best placed to help us with this. Although we appreciate the support, additional emails to their general tech support are likely to make things more confusing.

We received a response from our technical contact at newspapers.com, and we'll be working with them to troubleshoot the issue. At the moment this issue is still stalled as the ball is in their court. We'll share updates as they come.

I have tried several times in the past to get newspapers.com to solve the problem when I didn't know how to get my subscription renewed. The person who responded in each case didn't know what was going on and couldn't give real advice. It didn't help that my subscription used a different email from my Wikipedia email because my Wikipedia email didn't work for whatever reason and I had to use a different email for Wikipedia.

I had to go to my Wikipedia email when I couldn't get into newspapers.com because I knew the people who could solve the problem had emailed me there, but it's not an email I am in the habit of checking. In fact, I frequently get emails saying my emails will be deleted if I don't sign in to my account. I need notifications on my talk page when there are updates.

The only solution to our dilemma offered so far by newspapers.com is that of directly contacting ProQuest about a subscription to the website at the link below:

https://about.proquest.com/en/products-services/Newspaperscom-Library-Edition/

ProQuest also offers databases for over forty-five Historical Newspapers and for some Recent Newspapers, and those databases may be added to provide Wikipedia Library users access to other newspapers while the issue of access to newspapers.com is being resolved. The links are as follows:

  1. https://about.proquest.com/products-services/pq-hist-news.html
  2. https://about.proquest.com/en/products-services/ProQuest-Recent-Newspapers/

Why does ProQuest have anything to do with newspapers.com? Although one newspaper I look at using newspapers.com has full text articles on ProQuest.

The only solution to our dilemma offered so far by newspapers.com is that of directly contacting ProQuest about a subscription to the website at the link below:

https://about.proquest.com/en/products-services/Newspaperscom-Library-Edition/

ProQuest also offers databases for over forty-five Historical Newspapers and for some Recent Newspapers, and those databases may be added to provide Wikipedia Library users access to other newspapers while the issue of access to newspapers.com is being resolved. The links are as follows:

  1. https://about.proquest.com/products-services/pq-hist-news.html
  2. https://about.proquest.com/en/products-services/ProQuest-Recent-Newspapers/

Why does ProQuest have anything to do with newspapers.com? Although one newspaper I look at using newspapers.com has full text articles on ProQuest.

For the Wikipedia Library, it doesn't. ProQuest has a licensed offering that provides the newspapers.com content through the ProQuest platform. It is what many libraries subscribe to. We have a partnership directly with newspapers.com, and so our integration is with their direct offering. If you reach out to their technical support service, they will almost certainly not be aware of our integration, since it is unique.

Newspapers.com aren't aware of what would be causing this issue, but we've given one of their staff temporary library access so that they can view the issue for themselves. This helped us resolve the last issue like this we have, so I'm hopeful a fix is on the horizon.

This is not a new issue, but I wanted to voice that I am encountering this error message when attempting to create a newspapers.com account. I accessed the website through the Wikipedia Library link and tried to register that way. I hit a paywall on an account that I'd been using prior so it led me to attempt to fix the issue by making a new account (per a recommendation I received). I see that someone earlier in this thread encountered this issue, so I wanted to mention that I have run into this issue too.

image.png (124×486 px, 7 KB)

Going to post about this here since, although I solved the problem myself, this was what came up when I searched for it, so maybe other people will be here scratching their heads:

When I tried to log into newspapers.com (through the proxy, but it was the same at the original URL), the "Log in via Newspapers.com" button on the login screen was grayed out for no apparent reason. I just backspaced the login information and typed it in again -- in fact, I did it so I could type in a different email address and take a screenshot to post here, and suddenly the "log in via Newspapers.com" logo turned blue and activated, so I typed my actual email back in and it worked again.

Bizarre.

NOTE: I got another CloudFlare message while trying to log in to Newspapers.com through the library.
image.png (869×1 px, 44 KB)

I'm also getting the CloudFlare message when trying to log into Newspapers.com through the library.

jsn.sherman changed the task status from Stalled to Open.Jan 28 2024, 3:50 AM
jsn.sherman removed jsn.sherman as the assignee of this task.
jsn.sherman updated Other Assignee, removed: jsn.sherman.

I've marked this as a service issue on the website and put it on our kanban board. One of the Moderatator Tools team members will need to reach out to our contact at newspapers.com on Monday and request assistance with this.

Could someone PLEASE make Newspapers.com work or explain how I can access it using the WikiMedia Library! This is so frustrating, I have things I need to look up for Wikipedia research I'm doing and it has been months. My email is SusanGerbic@gmail.com if someone can come up with a solution. Please contact me.

Same issue as That_Article_Editing_Guy. Occurs whether using home wi-fi or mobile network (both on mobile phone). Changing to different browser not effective. Was working OK yesterday (Jan 26).

jsn.sherman changed the task status from Open to In Progress.Jan 30 2024, 5:20 PM
jsn.sherman claimed this task.
jsn.sherman changed the task status from In Progress to Stalled.Jan 30 2024, 5:43 PM

I requested assistance from our technical contact. Marking as stalled until we hear back.

Do we at least have an explanation of why this happens (repeatedly).

Is there any chance we could go back to the old mehod, where we simply had accounts recognized on teh standard newspapers.com site, especially since that made using auoated citations significantly easier?

jsn.sherman changed the task status from Stalled to In Progress.Jan 30 2024, 11:11 PM
jsn.sherman moved this task from In Progress to QA on the Moderator-Tools-Team (Kanban) board.

These errors should stop showing up for everyone within the hour. Apparently there was activity on the OCLC network that tripped security rules for the site. OCLC hosts our web proxy, so when they get blocked, we get blocked.

I finally remembered to test it and it worked.

I don't know if this is a related problem or should be filed as a separate ticket, but there are a very small number of papers which cannot be accessed through our library bundle. When you try to visit them, it gives you the same "Upgrade to a Publisher Extra Subscription to view this page" blocking pop-up that you would receive if you didn't have a subscription at all. Some examples I have encountered: https://www-newspapers-com.wikipedialibrary.idm.oclc.org/image/864192942/ https://www-newspapers-com.wikipedialibrary.idm.oclc.org/image/548374608/ I assume this is an error and Newspapers.com is not deliberately restricting these from us?

To clarify, 99.9% of Publisher Extra content still works totally fine, at least for me right this moment. It's just a scattering of these few papers that don't work, like the ones I linked. I think it applies to the entire run of a given paper, so we can't access any of the Lima News or the Brattleboro Reformer. It's a small issue, I just don't know who else to raise it with.

Hmmm… can't reach this pagewikipedialibrary.idm.oclc.org refused to connect.
Try:

Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED

I also can't connect to any Wikipedia Library OCLC service right now (Firefox says "Unable to connect - An error occurred during a connection to www-newspapers-com.wikipedialibrary.idm.oclc.org."). The problem persists across different devices, and started sometime early this afternoon. I work for a branch of the University of Wisconsin and have heard a lot in recent weeks about transitory OCLC outages affecting the university's services, so I suspect this could be another manifestation of that, rather than anything on Wikimedia's end.

It looks like OCLC is performing maintenance on their servers, per https://oclc.service-now.com/status

I also can't connect to any Wikipedia Library OCLC service right now (Firefox says "Unable to connect - An error occurred during a connection to www-newspapers-com.wikipedialibrary.idm.oclc.org."). The problem persists across different devices, and started sometime early this afternoon. I work for a branch of the University of Wisconsin and have heard a lot in recent weeks about transitory OCLC outages affecting the university's services, so I suspect this could be another manifestation of that, rather than anything on Wikimedia's end.

This seems to be an ongoing issue beyond maintenance - see T365553 for updates.

I could have sworn I posted here this morning.

Seems to be working now, though.

I've been unable to log into Newspapers.com through Ancestry using my Wikipedia Library account for weeks now. I can log into my Library account just fine. When, in Newspapers.com, I try to create a clipping to cite in a Wikipedia article, the Newspapers.com signin popup appears. I click "Ancestry" and the popup for name and password appears. I enter those. That popup disappears but the signin popup remains. Clicking the "Ancestry" button again makes it disappear and instantly reappear. When I go to Newspapers.com, I am logged in there, but I don't have Publisher's Extra.

Is there any expectation of this being fixed? It really impairs our ability to present information responsibly on Wikipedia.

Thanks so much for all the hard work you do that we never see!

@Oona_Wikiwalker, it might not be terribly convenient, but have you tried creating a Newspapers.com account entirely separate from your Ancestry account and using that? You should be able to use the same email address for both. It works for me, at least. That may be the best workaround until or unless that integration is fixed.

Per my note in T366070, we are considering moving Newspapers.com off proxy configuration back over to the manual account setup and renew method we had before this. Unlike most of the proxy configurations in the library, this one is a manually configured proxy which took us some time to set up and is very fragile to breaking. It's a big time sink for us to continue maintaining and fixing it.

For those who never had an individual account for Newspapers.com, the process worked like this: You would need to register an account for Newspapers.com (if you don't have one already), then file an application with us via the library to get your account upgraded. This would last 12 months, at which point you would need to request a renewal (we send automated reminders). This means you would access Newspapers.com directly on their website, rather than going via the library - it would be easier to directly cite Newspapers.com content without unproxying the URLs, and the website would behave normally without any proxy weirdness like Ancestry logins not working. The downside, of course, is the application-and-renewal process, which can result in some access downtime, and requires a little extra effort. Please let me know what you think.

I personally preferred the old way, even ignoring the issue of reliability, just because you could copy and paste URLs directly off the page without having to manually turn "https://www-newspapers-com.wikipedialibrary.idm.oclc.org" back into "https://www.newspapers.com" so ordinary users could access the article. That being said, if we do go back to the old way, I would hope that the application process could be streamlined somehow. It was always a little nerve-wracking to wait for weeks to get re-approved, and sit there wondering "are they looking at my edit history? have I been a good enough Wikipedian to get renewed?"

I use the Zotero browser plugin, which can automatically proxy and un-proxy links, so I've not had a problem dealing with proxied links.

It was always a little nerve-wracking to wait for weeks to get re-approved, and sit there wondering "are they looking at my edit history? have I been a good enough Wikipedian to get renewed?"

I never previously applied for this reason.

I'm currently getting an "Upgrade to a Publisher Extra Subscription to view this page" message on all pages. Just thought I should mention it.

I've never minded unproxying the links. It's no more annoying than having to wait around while Archive.Today creates its saved page so I can grab the long link. Is continually renewing accounts' access to newspapers.com less work than dealing with the fragility of the manual proxy? I don't want to be unreasonable.

As for how many of us access Newspapers.com, I know I do it all the time, because I don't put in articles what isn't supported by sources. Since I haven't had access, I've been hunting down geographic coordinates instead. However, tonight I'm trying to fix the reference section of someone's MONSTER of an article that doesn't link its references. There are over a hundred of them and many are newspapers.

So. I vote for whatever whatever lets me fix stuff because there is so much in Wikipedia that's below par!

Thank you so much for your efforts. You must know you make a difference, but I know it too. (I've been "suffering" without my Newspapers access, lol! I guess Wikipedia does too.) There have to be others like me, they just don't know about Phabricator.