Flagged for preprod!

The proxy access is working but the website is still displaying to me as a free trial when accessed at newspapers.com

Yeah, I also noticed that. @jsn.sherman, do you think this is something Newspapers.com need to solve on their end, or is there something we need to do?

Depends on what's been done so far:
Have they setup IP-based authentication for the proxy?

They have. Vipin is going to ask if there's another entry point we should be using, we haven't actually clarified that with them yet.

Partner has shared an update that this was due to a patch they released and access should work again now.

@Samwalton9 Can you check please?

In T322916#8643383, @sjvipin wrote:

Partner has shared an update that this was due to a patch they released and access should work again now.

@Samwalton9 Can you check please?

Oh, great. @Scardenasmolinar could we get this back up on Staging if it isn't already?

Flagged for preprod!

GitHub PR: https://github.com/WikipediaLibrary/twlight_ezproxy/pull/32

OK! This is looking better. We're recognised as Wikimedia Foundation. When I try to search, however, I get an error message:

Can you try again? It's working for me now.

In T322916#8653432, @Scardenasmolinar wrote:

Can you try again? It's working for me now.

I'm seeing exactly the same as my previous screenshot.

@Samwalton9 It's working for me on staging:
edge linux:

firefox windows:

chrome windows:

it might be worth looking at your javascript console in dev tools to see if you have an error there, since it looks like an ajax failure.

@Scardenasmolinar I recommend we promote this to ezproxy production and keep an eye on it

Pushing to prod!

It looks to be working in production. @HouseBlaster @jsn.sherman can you please confirm?

It is working!

Because we no longer use accounts, we are no longer able to take "clippings", which allow everyone to view a specific part of a newspaper. These are often used in citations, allowing everyone to verify the information. At least on enwiki, this is not a barrier to use, but I would consider it a "nice-to-have". Would it be possible to get this feature enabled, or is this a necessary and worthwhile sacrifice to allow all TWL users to access this resource?

@Scardenasmolinar working on all the browsers I previously mentioned; thanks!

In T322916#8653990, @HouseBlaster wrote:

It is working!

Because we no longer use accounts, we are no longer able to take "clippings", which allow everyone to view a specific part of a newspaper. These are often used in citations, allowing everyone to verify the information. At least on enwiki, this is not a barrier to use, but I would consider it a "nice-to-have". Would it be possible to get this feature enabled, or is this a necessary and worthwhile sacrifice to allow all TWL users to access this resource?

I'm not sure, but I did verify that account login is not currently working. It looks like some of the sign in bits break out of our proxied session, which might be the source.
@Scardenasmolinar I did notice the following in the response body for
https://www-newspapers-com.wikipedialibrary.idm.oclc.org:9443/search/?query=test+keyword

        <script>
            var ncom = {
                "cookieDomain": {
                    "site": ".newspapers.com",
                    "www": ".newspapers.com"
                },
                "stripePublishableKey": "pk_live_MbUhjwBlD5iFBj5dkC5eA9xL00JrfJyfku",
                "googleTagManager": {
                    "global": "GTM-N6D6HQR",
                    "international": "GTM-TR8H2TF"
                },
                "sourcepointPrivacyManagerId": 565931,
                "saveToAncestryApiScript": "https://www.ancestry.com/savetoancestry/javascript/saveToAncestryApi.js",
                "facebookAppId": "397139687020404",
                "institution": {
                    "id": 3297,
                    "name": "Wikimedia Foundation",
                    "k12": false
                },
                "paths": {
                    "image": "https://img-newspapers-com.wikipedialibrary.idm.oclc.org:9443",
                    "landing": "https://go-newspapers-com.wikipedialibrary.idm.oclc.org:9443",
                    "site": "https://www-newspapers-com.wikipedialibrary.idm.oclc.org:9443",
                    "sso": "https://www.ancestry.com",
                    "www": "https://www-newspapers-com.wikipedialibrary.idm.oclc.org:9443"
                },
                "ancestry": {
                    "site": "https://www.ancestry.com",
                    "phone": "1-800-718-7276",
                    "accountUrl": "https://www.ancestry.com/secure/account",
                    "supportEmail": "support@ancestry.com",
                    "hoursWeekday": "Monday to Friday: 10am-10pm Eastern Time",
                    "hoursWeekend": "Saturday to Sunday: 10am-6pm Eastern Time",
                    "onlineHelpUrl": "https://support.ancestry.com/s/",
                    "termsAndConditionsUrl": "https://www.ancestry.com/cs/legal/termsandconditions",
                    "giftTermsUrl": "https://www.ancestry.com/c/legal/gift-terms",
                    "giftFaqUrl": "https://www.ancestry.com/c/legal/gift-faq",
                    "privacyStatementUrl": "https://www.ancestry.com/cs/legal/privacystatement",
                    "contactSupport": "https://support.ancestry.com/s/contactsupport/account?language=en_US"
                },
...
        </script>

We might need to try to catch and rewrite that .newspapers.com reference in the cookieDomain part of the script.

I'm still getting the same error on both production and staging. When I'm not proxied (logged in to an account) it works fine. Console log below. I'm not seeing those errors in my non-proxied sessions.

In terms of sign-in, I'd hope that users could still sign in to accounts and save clippings, but I haven't tested that yet. When I try to register a new account I get an error:

I just tried again and didn't get that error, and am now signed into a fresh account. Obviously I can't test clippings right now due to the search error.

@Fehufanga tried this out and was able to get search results as expected. They tried to login via Ancestry and weren't able to.

@Samwalton9 Hello, I'm VulcanSphere on English Wikipedia. As requested, I hit this error when tried to login on https://www-newspapers-com.wikipedialibrary.idm.oclc.org with error message "Your sign in was blocked because of suspicious activity."

Unable to access site; "Your sign in was blocked because of suspicious activity."

After clearing caches related to oclc.org, now I can login into TWL proxy of Newspapers.com and search function working properly

Thanks for checking, @ReaperDawn

I verified that signin with newspapers.com accounts works with cleared cookies as you said. Signout and ancestry logins are still not working as intended.

Clearing my oclc.org cookies got search results working for me too. I was also able to sign in to my account successfully.

It's a shame that the interface displays "Try 7 Days Free" language despite the institutional access being recognised. I've asked if anything can be done about this.

@jsn.sherman noted that we were still getting some errors with reCaptcha - is this still true?

@Samwalton9 clearing cookies resolved that, though logout and ancestry sign-in were still not working.

Progress update: signout is now working for me on staging, though now I need to do a little cleanup to try to make the fix as narrow as possible.

Since I'm now testing pulling part of the config back out, you may find that it doesn't work for you on staging.

okay, this is ready for verification on staging. Login, clipping, and logout all work in staging so far as I can tell.

In T322916#8777106, @jsn.sherman wrote:

okay, this is ready for verification on staging. Login, clipping, and logout all work in staging so far as I can tell.

Looking good to me too!

pr merged and config marked for deployment to ezproxy production

Deployed to production and I verified the fix is working there.

Everything except the Ancestry login is now working. We're going to look at Ancestry in T327653, which may resolve this problem. Logging in via Ancestry isn't something that seems to be a common workflow for users so we'll backlog this for now pending the Ancestry ticket.