Page MenuHomePhabricator

Fix Newspapers.com access issues
Closed, ResolvedPublic

Assigned To
Authored By
sjvipin
Nov 11 2022, 10:48 AM
Referenced Files
F57768197: Screen Shot 2024-12-01 at 11.27.33 AM.png
Sun, Dec 1, 4:27 PM
F57533843: Screenshot 2024-09-24 at 10.54.04.png
Sep 24 2024, 9:54 AM
F57410153: Screen Shot 2024-09-02 at 1.35.01 PM.png
Sep 2 2024, 5:41 PM
F12: project.png
Aug 29 2024, 8:31 AM
File Not Attached
Restricted File
Jul 4 2024, 4:12 AM
F41721934: image.png
Jan 27 2024, 7:22 AM
F41436405: image.png
Nov 3 2023, 4:06 AM
F37819456: Screenshot_20230927-201747.png
Sep 28 2023, 3:25 AM
Tokens
"Barnstar" token, awarded by ClydeFranklin."Stroopwafel" token, awarded by mxn."Manufacturing Defect?" token, awarded by AntiCompositeNumber."Burninate" token, awarded by czar."Love" token, awarded by HouseBlaster."Like" token, awarded by Novem_Linguae.

Description

August 1 2024: For a full update on Newspapers.com please see T322916#10036209

Current issues

  • We do not currently seem to have access to some Publisher Extra content
  • Users cannot log in via Ancestry
  • Users cannot log in via Facebook
  • Users cannot register a new account via Email

Past issues
Cloudflare is preventing access. Users get stuck at www.newspapers.com needs to review the security of your connection before proceeding.

Users currently cannot log out of their account while proxied, or log in via Ancestry.

This change has now been deployed, however we are seeing some access issues. Some users cannot retrieve search results, and logging in to accounts sometimes doesn't work.

Users can't login as they see the following error:

image.png (133×482 px, 13 KB)

Newspapers.com fixed the initial issue here, which was enabling reCaptcha, however we're now getting a slightly different error about an "invalid domain key". We should be able to fix this issue, and are investigating.

Original task
Our existing partner newspapers.com has agreed to move to library bundle. We need to update their EZProxy configuration.

Partner id: 26 (https://wikipedialibrary.wmflabs.org/partners/26/)

Before making this change please send me a list of the user emails (privately) with active authorisations so I can let them know.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

"There was a problem loading the search results. Please try again.

If the problem persists, please contact us."

what happened?

"There was a problem loading the search results. Please try again.

If the problem persists, please contact us."

what happened?

Nevertheless, the problem persisted! :)

Samwalton9-WMF renamed this task from Fix Newspapers.com Library Bundle Configuration issues to Fix Newspapers.com access issues.Aug 14 2024, 8:51 AM
Samwalton9-WMF updated the task description. (Show Details)

It looks like search results might be working again - at least they are when I test them. I've downgraded the status of Newspapers.com in the library to 'Service Issue' so that the 'Access Collection' button is available again. Could someone else give it a go and let me know what you see?

Searches are working for me. I’m still unable to log in with my personal account through the proxy due to the CloudFlare error I mentioned earlier. Fortunately, I seem to have a workaround in Firefox:

  1. Log into Newspapers.com through the official site.
  2. Open the Storage tab of developer tools and copy the values of the user_data and user_token cookies, as well as their expiration dates.
  3. Go to www-newspapers-com.wikipedialibrary.idm.oclc.org and clear all cookies and local storage associated with the site.
  4. In the Storage tab of developer tools, add new user_data and user_token cookies matching what you copied from Newspapers.com.

This worked well enough for me to create a clipping (view without proxy).

I can search and can view some results but others show a page with "We're having technical difficulties"

The search function has always occasionally worked since the issues emerged six weeks ago. It stills goes down half the time, as it did right now for me. Nothing has changed or has been fixed with the search function, and logging in continues to not work.

Finally! I can see the full images after searches. Still can't create clippings.

Searches are working for me. I’m still unable to log in with my personal account through the proxy due to the CloudFlare error I mentioned earlier. Fortunately, I seem to have a workaround in Firefox:

  1. Log into Newspapers.com through the official site.
  2. Open the Storage tab of developer tools and copy the values of the user_data and user_token cookies, as well as their expiration dates.
  3. Go to www-newspapers-com.wikipedialibrary.idm.oclc.org and clear all cookies and local storage associated with the site.
  4. In the Storage tab of developer tools, add new user_data and user_token cookies matching what you copied from Newspapers.com.

This worked well enough for me to create a clipping (view without proxy).

This worked perfectly for me as well, at least for the time being.

@Reschultzed I'm trying this without success, apparently because I can't copy "Session" as an expiration date. Any assistance?

EDIT: Was easiest to use CookieCopy addon, trim down the output to just the cookies I needed, and run document.Cookie="(output)" in console.

Searches are working for me. I’m still unable to log in with my personal account through the proxy due to the CloudFlare error I mentioned earlier. Fortunately, I seem to have a workaround in Firefox:

  1. Log into Newspapers.com through the official site.
  2. Open the Storage tab of developer tools and copy the values of the user_data and user_token cookies, as well as their expiration dates.
  3. Go to www-newspapers-com.wikipedialibrary.idm.oclc.org and clear all cookies and local storage associated with the site.
  4. In the Storage tab of developer tools, add new user_data and user_token cookies matching what you copied from Newspapers.com.

This worked well enough for me to create a clipping (view without proxy).

I am truly hoping this is not because of an exploitable bug in the Newspapers.com backend because that would concern me an awful lot.

I am truly hoping this is not because of an exploitable bug in the Newspapers.com backend because that would concern me an awful lot.

The cookie just identifies your user and the token keep you logged in for a while. It's normal for many websites and why you should keep your cookies secret and always use https. Basically this work-around is copying the access your account legitimately has, you are just using your front door key on the Wikipedia side door.

I found you only have to copy the user_token (name, value, expiry) no need to clear the other cookies or copy the user_data

I think I'm heartened by this development. Newspapers.com is an essential resource whenever I expand or create articles. However, I admit to being practically illiterate in computer matters. Would somebody please kindly explain this workaround to me in beginner terms? Thank you all.

Here's exactly what I did to get it to work for me (on Firefox – if you use Chrome or any other browser, someone else will have to provide those instructions):

  1. Go directly to newspapers.com, not the proxy
  2. Log into your account
  3. Press Ctrl+Shift+I to open the developer console
  4. In the window that opens up on the bottom half of your screen, click on the "Storage" tab along the top of the window
  5. On the left side, there are different options. Click on "Cookies"
  6. Under "Cookies", click on "https://www.newspapers.com"
  7. There will be a whole bunch of lines of data. We only need the bottom two, that start with "user_data" and "user_token"
  8. Copy the long strings of seemingly random letters and numbers that appear under "Value" for both "user_data" and "user_token" and keep them in a notepad or whatever.
  9. Also copy the date that appears under "Expires / Max-Age" on the "user_token" row. The "user_data" row will say "Session" instead of a date, but as far as I can tell it's safe to ignore that.
  10. Go back to the proxy version of newspapers.com
  11. Press Ctrl+Shift+I to open the console again
  12. Follow steps 4-6 again in this new console window
  13. In the top right corner of the console window, there will be a plus sign button. Click that to add a new row
  14. Double-click on the stuff that appears under "Name" in the new row and type in "user_data" instead
  15. Do the same to replace the "Value" of the new row with the string you copied out of the other user_data row
  16. Do the same to replace the "Expires / Max Age" date with the one you copied out of "user_token"
  17. Create another new row and name it "user_token"
  18. For the "Value", paste in the value from the other "user_token" row
  19. Paste in the expiration date into the "Expires / Max Age" column for this row, too
  20. Close the console and refresh the page

These steps were enough for me to get it working properly, I didn't need to mess with anything else.

The instructions from @Reschultzed works for Microsoft Edge as well. Haven't tested Chrome, but appears to be similar as well.

For Step 4, once you open up developer tools, go to the "Application" tab for storage.
For Step 13, instead of a plus button, you need to scroll to the bottom. Right click on the empty line, and click "Add new".
The rest of the steps are exactly the same.

I can't create clippings. It tells me to sign in and when I do I see in a pink box "There was an error while loading the form. Please contact customer support".

I also tried to sign in on newspapers.com, but that doesn't change anything.

The instructions from @mxn and @Reschultzed worked for me as well, and allowed me to successfully create clippings; however, when trying to view those clippings or any I've made previously, I'm redirected to a "This Page is Not Available" screen.

I can't create clippings. It tells me to sign in and when I do I see in a pink box "There was an error while loading the form. Please contact customer support".

I was getting that until I cleared my cookies and local storage, and only then copied my user token over from newspapers.com. I think some users have gotten away with not clearing their cookies because of when exactly they had previously logged in and used the site.

The instructions from @mxn and @Reschultzed worked for me as well, and allowed me to successfully create clippings; however, when trying to view those clippings or any I've made previously, I'm redirected to a "This Page is Not Available" screen.

I get this sometimes too, but if you convert the clipping’s URL to a standard newspapers.com URL, it works fine for me. You’re supposed to do that anyways if you use Newspapers.com as a source on a Wikimedia wiki.

A good idea might be to separately sign in to Newspapers.com (non-proxied) and manage clippings from there.

Here's exactly what I did to get it to work for me (on Firefox – if you use Chrome or any other browser, someone else will have to provide those instructions):

  1. Go directly to newspapers.com, not the proxy
  2. Log into your account
  3. Press Ctrl+Shift+I to open the developer console
  4. In the window that opens up on the bottom half of your screen, click on the "Storage" tab along the top of the window
  5. On the left side, there are different options. Click on "Cookies"
  6. Under "Cookies", click on "https://www.newspapers.com"
  7. There will be a whole bunch of lines of data. We only need the bottom two, that start with "user_data" and "user_token"
  8. Copy the long strings of seemingly random letters and numbers that appear under "Value" for both "user_data" and "user_token" and keep them in a notepad or whatever.
  9. Also copy the date that appears under "Expires / Max-Age" on the "user_token" row. The "user_data" row will say "Session" instead of a date, but as far as I can tell it's safe to ignore that.
  10. Go back to the proxy version of newspapers.com
  11. Press Ctrl+Shift+I to open the console again
  12. Follow steps 4-6 again in this new console window
  13. In the top right corner of the console window, there will be a plus sign button. Click that to add a new row
  14. Double-click on the stuff that appears under "Name" in the new row and type in "user_data" instead
  15. Do the same to replace the "Value" of the new row with the string you copied out of the other user_data row
  16. Do the same to replace the "Expires / Max Age" date with the one you copied out of "user_token"
  17. Create another new row and name it "user_token"
  18. For the "Value", paste in the value from the other "user_token" row
  19. Paste in the expiration date into the "Expires / Max Age" column for this row, too
  20. Close the console and refresh the page

These steps were enough for me to get it working properly, I didn't need to mess with anything else.

My friend, this worked perfectly. Thank you so much!

A good idea might be to separately sign in to Newspapers.com (non-proxied) and manage clippings from there.

What does this mean, exactly? I am signed in to newspapers.com. I try to create a clipping and it tells me to sign in.

A good idea might be to separately sign in to Newspapers.com (non-proxied) and manage clippings from there.

What does this mean, exactly? I am signed in to newspapers.com. I try to create a clipping and it tells me to sign in.

This is referring to a separate but possibly related problem, where trying to view a previously-existing clipping from inside the proxy gives you a "This page is not available" error. Those of us who have successfully been able to use the aforementioned workaround to sign into our accounts in the proxy are still unable to view clippings in the proxy, and thus still need to manage those clippings from outside of the proxy.

The cookie-duplication trick worked for me on the proxy site on Firefox 129.0, but Newspapers.com is still reporting that I am only a Registered Guest, and I cannot make any clippings whatsoever (whether Publisher Extra content or not).

I still have a problem with the website. I am using the most recent version of Chrome - and even deleted my cookies.

I'm not deleting any cookies. That causes too many problems.

Furthermore, now I can't sign into newspapers.com, period. It tells me either my username or password is incorrect. I am told to enter a code to reset my password but it never gets emailed to me.

Oh, wait, I'm signed in now. Because I typed "newspapers.com". But I'm betting I can't create clippings.

This comment was removed by BJCHK.
This comment was removed by KylieTastic.

Is this a new problem, and is it related?

There was a problem loading the search results. Please try again.
If the problem persists, please contact us.

Oh, wait, I'm signed in now. Because I typed "newspapers.com". But I'm betting I can't create clippings.

On a Mac it is: "Fn + F12" or "Cmd + Option + I"

https://developer.chrome.com/docs/devtools/open#shortcuts

I'm not on a Mac.

I was replying to the comment made before that was someone saying they were having issues opening the dev tools on a Mac.... they deleted the comment.. Not sure how I managed to respond to the wrong one. sorry for the confusion

I was having issues with searching (I saw the error 429 Too Many Requests in DevTools), but creating an account and copying the cookies fixed it!

This has been incredibly frustrating. I had Newspapers.com access before the new system, which just made things more annoying, and then this happened-making doing research much harder.

I'm still unable to sign into newspapers.com via the WikLibrary proxy. I repeatedly get the message "There was an error while loading the form. Please contact customer support" This issue has been ongoing for months.

@KewGardens613 and @Flibirigit : you may be able to get access by following the steps outlined above in the comments on August 14 and August 15. It's a bit complicated, but it worked for me and several others in this thread.

I was replying to the comment made before that was someone saying they were having issues opening the dev tools on a Mac.... they deleted the comment.. Not sure how I managed to respond to the wrong one. sorry for the confusion

Well, one thing's for sure, I wouldn't know how to do anything involving developer tools.

Is there a way to restore my access to the Publisher Extra content? IIRC when I first signed up for access there was a bullet point on the Wikipedia Library portal saying something to the effect of "You must renew your Newspapers.com subscription by reapplying annually", but that text seems to be gone. Was it just bad luck that the expiration coincided with this implementation of proxy servers, or am I misremembering things? My Newspapers.com account name is "digitaliceage"; my profile page currently has me listed as a "Registered Guest" when it wasn't before.

Is there a way to restore my access to the Publisher Extra content? IIRC when I first signed up for access there was a bullet point on the Wikipedia Library portal saying something to the effect of "You must renew your Newspapers.com subscription by reapplying annually", but that text seems to be gone. Was it just bad luck that the expiration coincided with this implementation of proxy servers, or am I misremembering things? My Newspapers.com account name is "digitaliceage"; my profile page currently has me listed as a "Registered Guest" when it wasn't before.

Please see T322916#10036209 for the latest update.

I think that there might be some confusion as to using the backend for the NP proxy login. It has worked for me when I am simultaneously logged into NP proper and the cookies are directly copied over into the proxy; this includes the duration date especially. If the login session on NP ends, as in a timed sign out after a month or so, you HAVE to do the process over again, which isn't terribly hard, but that's solely because the cookies need to be in sync with each other.

Would that there could be a fix for the non-technically inclined layman (which fortunately I am not), but this is probably the best we can do for the long term.

On Chrome, I was able to view the images that make up the page by going to the Sources tab, selecting the Page tab below that and "img-newspapers-com ...", opening the image I wanted in a new tab and changing the crop in the url. The first two numbers move the crop from side to side and the last two control the zoom in or out. Useful if you just want to verify something that doesn't show up on Ancestry's Obituary Index, or (as in my case) the workaround doesn't work.

When I get to this URL: https://www-newspapers-com.wikipedialibrary.idm.oclc.org/signin/?next_url=/, I get this error message: There was an error while loading the form. Please contact customer support. I tried changing the URL to www.newspapers.com instead of the hyphens but it automatically reverts back. Is there a fix for this?

When I get to this URL: https://www-newspapers-com.wikipedialibrary.idm.oclc.org/signin/?next_url=/, I get this error message: There was an error while loading the form. Please contact customer support. I tried changing the URL to www.newspapers.com instead of the hyphens but it automatically reverts back. Is there a fix for this?

The official update is in T322916#10036209. For a workaround in Firefox (with similarities to Chrome), see T322916#10063620 or the more detailed version in T322916#10065665.

Earlier this week I had trouble signing in to post when I was at a library, but I saved a URL in the form https://www-newspapers-com.wikipedialibrary.idm.oclc.org/image/1110761182 because I wanted to print a photo and couldn't make a clipping. It told me to subscribe. I was able to print the photo when I changed to https://www-newspapers-com.wikipedialibrary.idm.oclc.org/browse/united-states/north-carolina/asheville/asheville-citizen-times_4039/2024/08/20/

I've noticed I've not been able to search anything from Wikipedia Library. I always get this error:

"There was a problem loading the search results. Please try again.

If the problem persists, please contact us."

When I hit the "contact us" button and fill out the form, I get an error that doesn't allow the email to be sent.

And I'm getting an error message "There was a problem verifying you are not a robot. Please try again or refresh the page." (Trying again and/or refreshing the page doesn't resolve the problem.) I've also tried several different browsers in Private mode to avoid any possible issues with cookies.

Good news! It looks like we'll be able to move back to the individual account setup method. Further details TBC.

Samwalton9-WMF assigned this task to sjvipin.

TL;DR: To regain access to Newspapers.com please go to https://wikipedialibrary.wmflabs.org/partners/26/ and click Apply. Access requests will be processed and sent to Newspapers.com over the coming weeks.

Newspapers.com have agreed to move back to our old method of granting access, whereby users must first register at Newspapers.com, then apply via The Wikipedia Library, and then accounts are upgraded to a subscribing one by Newspapers.com. We have updated Newspapers.com to accept applications in the library, so you can apply now! This change is unfortunately mutually exclusive with the limited proxy-based access working, so any of the technical workarounds that have been discussed here should no longer work, along with any proxy-based access to Newspapers.com. I hope you agree that this brief bump will be worth it in the long run. As a reminder, this change means that you will need to renew your access on a yearly basis. If you have an email configured in The Wikipedia Library you should receive an email a few weeks before this expiry to remind you to submit an extension request, but brief yearly loss of access may be a side effect of this access method. The good news is that this access method means that you shouldn't encounter any technical problems besides normal operational problems on Newspapers.com's end, which should be few and resolved quickly - otherwise the website should work just as if you were a normal customer.

We are planning to directly notify all users who have used Newspapers.com via the proxy over the past year separately to this announcement.

Thanks may be directed towards @sjvipin who has been working tirelessly to find a way forward with Newspapers.com :)

This is good news, thank you! However, the Apply button isn't working for me. I've tried it in Safari and Chrome, logged into my Wikipedia Library account. Nothing happens when I click the Apply button.

This is good news, thank you! However, the Apply button isn't working for me. I've tried it in Safari and Chrome, logged into my Wikipedia Library account. Nothing happens when I click the Apply button.

Very odd! This might be related to you having a previous application, but I see that other users have been able to request renewal.

Can you instead try clicking the 'View my applications' button in the library, and then clicking 'Reuest renewal' next to the Newspapers.com entry?

Screenshot 2024-09-24 at 10.54.04.png (582×1 px, 75 KB)

Thank you, that seems to have worked!

What is renewal of existing access grant? Have I done something wrong if it says no?

What is renewal of existing access grant? Have I done something wrong if it says no?

We still had data on past manually-granted accesses, so if you had an account prior to the shift to proxy-based access, your application may be filed as a renewal of your past application. This isn't anything to be worried about :)

Thank you to all concerned for the good work. I have a couple of questions, if you please. Upon having my application approved, I signed into my account, but was met with a prompt urging me to upgrade whenever I tried to view an article. Do I have to open a brand new account or is it possible to continue using my old one? If the latter, why am I getting this prompt?

@CurryTime7-24 : I believe having your application approved does not mean that it is ready to go. Approval comes from the people running TWL, after which your request is sent to Newspapers.com. It may take them awhile to set up approved account, so I'd just wait a while. My approval email mentioned that "You can expect to receive access details within a week or two once it has been processed." so I wouldn't worry til then.

@ARandomName123 Ah! I jumped the gun then. :) Thanks so much for clearing that up.

I had asked this question of Samwalton9 privately and was told,

Yep, we just need to wait for Newspapers.com to activate this batch of accounts. I think we sent them close to 200 so I don't know how quickly they'll get through them

Until yesterday I could get to individual pages. Today, even if I sign in to newspapers.com, I cannot.

The way we have been doing this was working, but now I get only part of the page, meaning the links and descriptions, but not the page itself.

When I used the regular newspapers.com address, I got to the page but was told to subscribe. When I signed in, I was told to get a subscription.

@CurryTime7-24 My account has now been updated, and is good to go. Should be the same for you!

I only saw this now somehow-I wish I got a notification.

TL;DR: To regain access to Newspapers.com please go to https://wikipedialibrary.wmflabs.org/partners/26/ and click Apply. Access requests will be processed and sent to Newspapers.com over the coming weeks.

Newspapers.com have agreed to move back to our old method of granting access, whereby users must first register at Newspapers.com, then apply via The Wikipedia Library, and then accounts are upgraded to a subscribing one by Newspapers.com. We have updated Newspapers.com to accept applications in the library, so you can apply now! This change is unfortunately mutually exclusive with the limited proxy-based access working, so any of the technical workarounds that have been discussed here should no longer work, along with any proxy-based access to Newspapers.com. I hope you agree that this brief bump will be worth it in the long run. As a reminder, this change means that you will need to renew your access on a yearly basis. If you have an email configured in The Wikipedia Library you should receive an email a few weeks before this expiry to remind you to submit an extension request, but brief yearly loss of access may be a side effect of this access method. The good news is that this access method means that you shouldn't encounter any technical problems besides normal operational problems on Newspapers.com's end, which should be few and resolved quickly - otherwise the website should work just as if you were a normal customer.

We are planning to directly notify all users who have used Newspapers.com via the proxy over the past year separately to this announcement.

Thanks may be directed towards @sjvipin who has been working tirelessly to find a way forward with Newspapers.com :)

I only saw this now somehow-I wish I got a notification.

We emailed everyone who had accessed Newspapers.com recently, but it's possible this ended up in spam or otherwise got missed :)