Wiley requests for DOI and some other publishers don't work in production
Open, NormalPublic

Description

Several requests are not working in production via the new API:

Wiley publishers: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x
AGU publications: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1029%2F94WR00436

However, they are working from citoid on master directly:

http://localhost:1970/api?format=mediawiki&search=10.1046%2Fj.1365-2117.2002.00186.x
http://localhost:1970/api?format=mediawiki&search=10.1029%2F94WR00436

TODO:

  • Get unblocked by Wiley
  • Allow requests to crossref and try to make citation solely from DOI even if the url is determined to be private
  • Maybe submit every url in a redirect chain to Zotero.
Albert created this task.May 11 2017, 10:14 PM
Restricted Application added a project: VisualEditor. · View Herald TranscriptMay 11 2017, 10:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Whatamidoing-WMF renamed this task from Cotoid; several publishers recently missing? to Citoid: several publishers recently missing?.May 11 2017, 10:47 PM
Whatamidoing-WMF added a subscriber: Mvolz.
Whatamidoing-WMF added a subscriber: Whatamidoing-WMF.

Albert, do you happen to know whether those publishers have recently re-arranged or re-formatted their websites?

Hi,

I'm not aware of any recent changes of these websites. Especially the front
end of AGU publications looks identical since the last several years.....

Thanks for the quick response!,
Albert.

One more comment I would like to add; both journals are included in the pubmed database. If it helps these are their specifics:

JrId: 22628
JournalTitle: Journal of geophysical research
MedAbbr: J Geophys Res
ISSN (Print): 0148-0227
ISSN (Online): 2156-2202
IsoAbbr: J. Geophys. Res.
NlmId: 100971480

JrId: 22725
JournalTitle: Water resources research
MedAbbr: Water Resour Res
ISSN (Print): 0043-1397
ISSN (Online): 1944-7973
IsoAbbr: Water Resour Res
NlmId: 7501965

But I noticed that actually all publications published by "Wiley" are not available through Citoid, so maybe some setting have changed on the "Wiley" publisher site?

Thanks for looking into this,
Albert.

czar added a subscriber: czar.EditedMay 16 2017, 3:39 PM

The DOIs resolve (https://dx.doi.org/) to:
http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2117.2002.00186.x/abstract
http://onlinelibrary.wiley.com/doi/10.1029/94WR00436/abstract

and the regular Zotero "Wiley Online Library" translator works fine from those webpages

VE's Citoid widget isn't able to process either URL or http://dx.doi.org/10.1029/94WR00436

Also I haven't used this API format before—is it choking on the DOI input? Or the slash's formatting (%2F)?

Thank you,
Yes the DOIs get well resolved by the dx.doi.org site, that works fine. However, I want to like to get the metadata of publications through the https://en.wikipedia.org/api/rest_v1/data/citation/zotero site as it gives back json information that then can be imported in a mediawiki. This The "%2F format works fine for other publications through the API so I don't think this is a problem either. So see for example: https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1111%2Fj.1751-8369.2002.tb00087.x
It needs the "%2F" in the DOI to solve the URL (won't work with "/".

Hope that explains the problem a bit more. Thanks, Albert.

Mvolz added a comment.May 18 2017, 7:15 AM

Hmm, we recently switched to more concurrent architecture - it's possible that something about this particular url exposes a race condition of some sort. I'll look into it further.

Mvolz triaged this task as Normal priority.May 18 2017, 7:16 AM
Mvolz claimed this task.
Mvolz removed Mvolz as the assignee of this task.May 25 2017, 10:45 AM

These are working on citoid on master directly. Seems to be a problem with the RestBase wrapper.

Mvolz renamed this task from Citoid: several publishers recently missing? to Some search queries that work on citoid directly are broken from restbase.May 25 2017, 10:46 AM
Mvolz added a project: RESTBase.
Mvolz added a subscriber: mobrovac.
Mvolz renamed this task from Some search queries that work on citoid directly are broken from restbase to Some search queries that work on citoid directly are causing unknown hyperswitch errors in restbase.May 25 2017, 10:50 AM
Mvolz raised the priority of this task from Normal to High.
Mvolz updated the task description. (Show Details)
Mvolz added a subscriber: Pchelolo.
Mvolz moved this task from Backlog to Production on the Citoid board.May 25 2017, 11:13 AM
mobrovac renamed this task from Some search queries that work on citoid directly are causing unknown hyperswitch errors in restbase to Citoid confuses DOI with URL in production.May 29 2017, 7:41 PM

For some reason, Citoid in production resolves the given input as a URL instead of a DOI. Issuing a query for localhost:1970/api?format=mediawiki&search=10.1029%2F94WR00436 directly on a production host results in:

{"Error":"Invalid host supplied"}

Citoid logs show that it blocked the request because it thinks it should look for a private IP address:

http://127.0.0.1/?DOI=10.1029/94WR00436 is not public, and is disallowed (levelPath=warn/hostIsAllowed)

Further investigation is needed here.

Full debug log from production:

[2017-05-29T21:45:22.883Z]  INFO: citoid/17369 on scb1002: Worker 17369 listening on 0.0.0.0:1977 (levelPath=info)
[2017-05-29T21:45:34.596Z] DEBUG: citoid/17369 on scb1002: Adding response function (levelPath=debug/CitoidService)
[2017-05-29T21:45:34.598Z] DEBUG: citoid/17369 on scb1002: DOI detected (levelPath=debug/CitoidService)
[2017-05-29T21:45:34.598Z] DEBUG: citoid/17369 on scb1002: requestFromDOI method (levelPath=debug/CitoidService)
[2017-05-29T21:45:35.053Z] DEBUG: citoid/17369 on scb1002: Resolved DOI 10.1029/94wr00436 to URL http://doi.wiley.com/10.1029/94WR00436; Sending to requestFromURL (levelPath=debug/DOI)
[2017-05-29T21:45:35.054Z] DEBUG: citoid/17369 on scb1002: requestFromURL method (levelPath=debug/CitoidService)
[2017-05-29T21:45:36.636Z] DEBUG: citoid/17369 on scb1002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.637Z] DEBUG: citoid/17369 on scb1002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.656Z] DEBUG: citoid/17369 on scb1002: Zotero request made for: http://doi.wiley.com/10.1029/94WR00436 (levelPath=debug/zotero)
[2017-05-29T21:45:36.659Z] DEBUG: citoid/17369 on scb1002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.659Z] DEBUG: citoid/17369 on scb1002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.736Z] DEBUG: citoid/17369 on scb1002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.736Z] DEBUG: citoid/17369 on scb1002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.780Z]  WARN: citoid/17369 on scb1002: http://127.0.0.1/?DOI=10.1029/94WR00436 is not public, and is disallowed (levelPath=warn/hostIsAllowed)

Super weird. That's not really confusing url with doi though; request from URL is normal after a request from DOI, and we don't make any requests that look like http://127.0.0.1/?DOI=10.1029/94WR00436 . Just looking at that log it looks like wiley is making some weird redirect stuff which we knew was happening before.

We should make sure that even if it chokes during the redirect though that we still use crossref. If it fails when checking a url, citoid doesn't let it proceed to the scraper at all, which is where we request extra info from crossref if we aren't actually able to scrape.

Mvolz renamed this task from Citoid confuses DOI with URL in production to Wiley requests for DOI and some other publishers don't work in production.May 30 2017, 12:59 PM
Mvolz updated the task description. (Show Details)

Oddly, the request completes fine in CODFW (the other DC):

[2017-05-30T13:27:05.324Z] DEBUG: citoid/102520 on scb2002: Adding response function (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.325Z] DEBUG: citoid/102520 on scb2002: DOI detected (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.325Z] DEBUG: citoid/102520 on scb2002: requestFromDOI method (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.874Z] DEBUG: citoid/102520 on scb2002: Resolved DOI 10.1029/94wr00436 to URL http://doi.wiley.com/10.1029/94WR00436; Sending to requestFromURL (levelPath=debug/DOI)
[2017-05-30T13:27:05.875Z] DEBUG: citoid/102520 on scb2002: requestFromURL method (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.875Z] TRACE: citoid/102520 on scb2002: Resolving hostname doi.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:05.876Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:05.877Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.415Z] TRACE: citoid/102520 on scb2002: resolve6Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.416Z] DEBUG: citoid/102520 on scb2002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.417Z] DEBUG: citoid/102520 on scb2002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.431Z] DEBUG: citoid/102520 on scb2002: Zotero request made for: http://doi.wiley.com/10.1029/94WR00436 (levelPath=debug/zotero)
[2017-05-30T13:27:07.432Z] TRACE: citoid/102520 on scb2002: No Zotero translator found, looking for redirects (levelPath=trace/zotero)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: Resolving hostname doi.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: resolve6Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] DEBUG: citoid/102520 on scb2002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.434Z] DEBUG: citoid/102520 on scb2002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.561Z] TRACE: citoid/102520 on scb2002: Resolving hostname onlinelibrary.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.562Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.097Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.195 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.366Z] DEBUG: citoid/102520 on scb2002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:09.366Z] DEBUG: citoid/102520 on scb2002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: Resolving hostname onlinelibrary.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.195 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:11.136Z] DEBUG: citoid/102520 on scb2002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:11.136Z] DEBUG: citoid/102520 on scb2002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:11.736Z] TRACE: citoid/102520 on scb2002: Redirect detected to http://onlinelibrary.wiley.com/doi/10.1029/94WR00436/abstract;jsessionid=60DF91115ABB533A606FA3A5B86C0409.f04t03 (levelPath=trace/zotero)
[2017-05-30T13:27:13.158Z] DEBUG: citoid/102520 on scb2002: Zotero request made for: http://onlinelibrary.wiley.com/doi/10.1029/94WR00436/abstract;jsessionid=60DF91115ABB533A606FA3A5B86C0409.f04t03 (levelPath=debug/zotero)
[2017-05-30T13:27:13.203Z] DEBUG: citoid/102520 on scb2002: Successfully retrieved body from Zotero (levelPath=debug/zotero)
[2017-05-30T13:27:13.839Z] DEBUG: citoid/102520 on scb2002: PubMed query made for: https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?tool=citoid&email=citoid@mediawiki&format=json&ids=10.1029%2F94WR00436 (levelPath=debug/pubmed)
Joe added a subscriber: Joe.May 30 2017, 2:20 PM

So the problem is the eqiad proxy cached a redirect to localhost, likely sent by the remote host during an outage

1496153701.136     45 10.64.16.21 TCP_MISS/302 556 GET http://onlinelibrary.wiley.com/resolve/doi? - HIER_DIRECT/199.171.202.195 text/html [User-Agent: Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)\r\nCookie: OLProdServerID=1027\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 0\r\nConnection: close\r\nHost: onlinelibrary.wiley.com\r\n] [HTTP/1.1 302 Found\r\nDate: Tue, 30 May 2017 14:14:59 GMT\r\nServer: Apache\r\nLocation: http://127.0.0.1/?DOI=10.1111/1467-7660.00159\r\nContent-Length: 229\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r]

So clearing that from the cache should solve the issue.

Mvolz added a comment.May 30 2017, 2:23 PM

@Joe aha, thanks!

How long do these caches last? This problem been going on for a month apparently!

Joe added a comment.May 30 2017, 2:28 PM

Actually it was a dumb comment - the log I pasted clearly reported TCP_MISS/302, so I'm not sure what's happening. Investigating further.

Joe added a comment.May 30 2017, 2:33 PM

In fact, I suppose the problem is our proxy IP in eqiad has been banned. From the proxy machine

curl -v 'http://onlinelibrary.wiley.com/resolve/doi?DOI=10.1029/2005GB002488'
* Hostname was NOT found in DNS cache
*   Trying 199.171.202.195...
* Connected to onlinelibrary.wiley.com (199.171.202.195) port 80 (#0)
> GET /resolve/doi?DOI=10.1029/2005GB002488 HTTP/1.1
> User-Agent: curl/7.38.0
> Host: onlinelibrary.wiley.com
> Accept: */*
> 
< HTTP/1.1 302 Found
< Set-Cookie: OLProdServerID=1024; domain=.wiley.com; path=/
< Date: Tue, 30 May 2017 14:29:32 GMT
* Server Apache is not blacklisted
< Server: Apache
< Location: http://127.0.0.1/?DOI=10.1029/2005GB002488
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://127.0.0.1/?DOI=10.1029/2005GB002488">here</a>.</p>
</body></html>
* Closing connection 0

as you can see, the response redirects to 127.0.0.1.

I've verified from other hosts that can communicate directly and the responses we get are correct.

We might want to contact the admins at wiley.com to get whitelisted possibly.

In the meanwhile, we could try using the codfw proxy, but I suspect we'd only end up getting it banned as well.

Mentioned in SAL (#wikimedia-operations) [2017-05-30T14:53:09Z] <_joe_> failing citoid over to codfw, T165105

We might want to contact the admins at wiley.com to get whitelisted possibly.

@Jdforrester-WMF do we have a way of contacting Wiley? It would be awesome if they could white-list our IPs (we basically need two, one per DC).

In the meanwhile, we could try using the codfw proxy, but I suspect we'd only end up getting it banned as well.

We opted for this work-around for the time being, but it's not going to last us long...

Mvolz added a comment.May 31 2017, 8:01 AM

I've sent a cold e-mail to technical support.

Mvolz lowered the priority of this task from High to Normal.May 31 2017, 9:06 AM
Mvolz claimed this task.Jun 12 2017, 11:20 AM
Mvolz updated the task description. (Show Details)

No response to the e-mail.

They only seem to be blocking our native scraper, probably by UA, and not Zotero (which uses a Mozilla UA) which is why using the urls directly and not trying to follow the redirect from the doi seems to be working okay.

The initial redirect is handled by doi.dx which gives us a public and a correct url, which we could send to Zotero to get a result. But I guess the problem is that we're following the chain of redirects after that to the end, where we're blocked.

We could actually stop in the middle of the redirect chain and check every url to see if private and and send that to Zotero (which doesn't follow redirects at all) - maybe another solution? Although potentially slow.

Mvolz updated the task description. (Show Details)Jun 12 2017, 11:28 AM

@Samwalton9 are you in contact with Wiley through TWL? (Or is someone else?) Might you be able to help us get in touch? Summary: We need our server whitelisted so that we can provide automated lookup of their citation information

Not directly, but I might have someone who does have direct contact. Could you give me a TL;DR of the above? Not sure I understood the details - what is it that needs whitelisting?

Not directly, but I might have someone who does have direct contact. Could you give me a TL;DR of the above? Not sure I understood the details - what is it that needs whitelisting?

We have two data centers that run Citoid. We believe Wiley has blocked one of the two data centre's IP addresses.

Citoid is a webscraper that fills in citation information on Wikipedia. So for instance, a person tries to cite a Wiley paper on Wikipedia using our tool, citoid will take that URL, try to access the page, and then fill in the title of the paper, the authors, etc. so the person doesn't have to fill in that information manually. Does this help?

Yes - thanks! Do you have the IP address, so I can pass it along?

Yes - thanks! Do you have the IP address, so I can pass it along?

@Joe @mobrovac can you send the two IP addresses that we need to be whitelisted to @Samwalton9? Thanks :).

Yes - thanks! Do you have the IP address, so I can pass it along?

@Joe @mobrovac can you send the two IP addresses that we need to be whitelisted to @Samwalton9? Thanks :).

@akosiaris @Joe ping ^

Joe added a comment.Jun 21 2017, 7:29 AM

Hey sorry I was on vacation last week.

The IPs of our proxies are:

webproxy.eqiad.wmnet - 208.80.154.86
webproxy.codfw.wmnet - 208.80.153.53

akosiaris added a comment.EditedJun 21 2017, 8:22 AM

Hey sorry I was on vacation last week.

The IPs of our proxies are:

webproxy.eqiad.wmnet - 208.80.154.86
webproxy.codfw.wmnet - 208.80.153.53

Not that's not correct. It's the url-downloaders that are used for applications, not the webproxies. But sending just these IP address would unnecessarily restrict us in the network operations we can do as we would not be able to use different IP addresses for those services. @Samwalton9 could you please ask them to whitelist our entire address space ? That would be 208.80.152.0/22 for both Datacenters. That IPs used from that space are strictly allocated so they 'll never see any illegitimate traffic. And we will always reacting ASAP to any kind of abuse report.

Thanks - I've sent an email to our contacts at Cochrane, who will hopefully be able to pass the message on to the right people at Wiley. I'll let you know if/when I hear back.

Change 360832 had a related patch set uploaded (by Mvolz; owner: Marielle Volz):
[mediawiki/services/citoid@master] [WIP] Fallback to crossRef for restricted URLs

https://gerrit.wikimedia.org/r/360832

Change 360832 merged by jenkins-bot:
[mediawiki/services/citoid@master] Fallback to crossRef for restricted URLs

https://gerrit.wikimedia.org/r/360832

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:06:29Z] <mobrovac@tin> Started deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:06:29Z] <mobrovac@tin> Started deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:09:21Z] <mobrovac@tin> Finished deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105) (duration: 02m 52s)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:09:21Z] <mobrovac@tin> Finished deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105) (duration: 02m 52s)

Mvolz updated the task description. (Show Details)
Mvolz removed Mvolz as the assignee of this task.

I'm told that Wiley "should be able to provide the necessary info this week."

Thanks for chasing this down @Samwalton9!

GWicke moved this task from blocked to watching on the Services board.Jul 18 2017, 9:06 PM
GWicke edited projects, added Services (watching); removed Services (blocked).
elukey added a subscriber: elukey.Jul 19 2017, 8:02 AM

I received this response today:

I have confirmed with network and Sys-Ops team that none of the IP from the range are banned at firewall or application level. Could you please request customer to try again.

IP range (208.80.152.0/22)

If the issue persist please request them to execute the following 9 commands from datacentre IP (any IP from where he is unable to access WOL) and get back with the result of each command executed to diagnose their connection to OL:

  1. ipconfig/all
  2. nslookup onlinelibrarystatic.wiley.com
  3. tracert onlinelibrarystatic.wiley.com
  4. ping onlinelibrarystatic.wiley.com
  5. pathping onlinelibrarystatic.wiley.com
  6. nslookup onlinelibrary.wiley.com
  7. tracert onlinelibrary.wiley.com
  8. ping onlinelibrary.wiley.com
  9. pathping onlinelibrary.wiley.com These information will help us a lot. Please let us know if you have any further query.

I think we have been unblocked btw

$ curl -i --resolve en.wikipedia.org:443:208.80.154.224 https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x

HTTP/2 200
....
[{"itemType":"journalArticle","creators":[{"firstName":"A. J.","lastName":"Barnett","creatorType":"author"},{"firstName":"P. M.","lastName":"Burgess","creatorType":"author"},{"firstName":"V. P.","lastName":"Wright","creatorType":"author"}],"notes":[],"tags":[],"title":"Icehouse world sea-level behaviour and...

Shall we re-enable the other DC now?

I think we have been unblocked btw

$ curl -i --resolve en.wikipedia.org:443:208.80.154.224 https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x

HTTP/2 200
....
[{"itemType":"journalArticle","creators":[{"firstName":"A. J.","lastName":"Barnett","creatorType":"author"},{"firstName":"P. M.","lastName":"Burgess","creatorType":"author"},{"firstName":"V. P.","lastName":"Wright","creatorType":"author"}],"notes":[],"tags":[],"title":"Icehouse world sea-level behaviour and...

Actually that was the wrong paste. The correct one is

$ curl -i 'http://citoid.svc.eqiad.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x'
HTTP/1.1 200 OK
...

[{"url":"http://doi.wiley.com/10.1046/j.1365-2117.2002.00186.x","itemType":"journalArticle","pages":"417–438","creators":[{"creatorType":"author","lastName":"Barnett","firstName":"A. J."},{"creatorType":"author","lastName":"Burgess","firstName":"P. M."},{"creatorType":"author","lastName":"Wright","firstName":"V. P."}],"title":"Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies","publicationTitle":"Basin Research","volume":"14","issue":"3"}]

What's concerning is that I checked today once more and it seems eqiad and codfw returning different data. e.g.

akosiaris@bast1001:~$ curl 'http://citoid.svc.eqiad.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   576  100   576    0     0    780      0 --:--:-- --:--:-- --:--:--   780
[
  {
    "url": "http://doi.wiley.com/10.1046/j.1365-2117.2002.00186.x",
    "itemType": "journalArticle",
    "pages": "417–438",
    "creators": [
      {
        "creatorType": "author",
        "lastName": "Barnett",
        "firstName": "A. J."
      },
      {
        "creatorType": "author",
        "lastName": "Burgess",
        "firstName": "P. M."
      },
      {
        "creatorType": "author",
        "lastName": "Wright",
        "firstName": "V. P."
      }
    ],
    "title": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies",
    "publicationTitle": "Basin Research",
    "volume": "14",
    "issue": "3"
  }

while codfw returns more data

akosiaris@bast1001:~$ curl 'http://citoid.svc.codfw.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3038  100  3038    0     0   1056      0  0:00:02  0:00:02 --:--:--  1056
[
  {
    "itemType": "journalArticle",
    "creators": [
      {
        "firstName": "A. J.",
        "lastName": "Barnett",
        "creatorType": "author"
      },
      {
        "firstName": "P. M.",
        "lastName": "Burgess",
        "creatorType": "author"
      },
      {
        "firstName": "V. P.",
        "lastName": "Wright",
        "creatorType": "author"
      }
    ],
    "notes": [],
    "tags": [],
    "title": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies",
    "publicationTitle": "Basin Research",
    "volume": "14",
    "issue": "3",
    "ISSN": "1365-2117",
    "url": "http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2117.2002.00186.x/abstract",
    "DOI": "10.1046/j.1365-2117.2002.00186.x",
    "pages": "417–438",
    "date": "2002-09-01",
    "abstractNote": "Late Visean (Asbian–Brigantian) platform carbonates in the British Isles show a pronounced cyclicity marked by the alternation of mainly subtidal carbonates and subaerial exposure surfaces. Whereas the cyclicity of shallow-water limestones of this age has been well-documented, there has been little attempt to understand the controls on larger-scale patterns such as those recognized in Pennsylvanian successions in the U.S.A. The principal aim of this study is to test two contrasting theories of cycle stacking via numerical forward modelling. Earlier studies of Pennsylvanian-early Permian platform carbonates in south-west U.S.A. suggest that cycle stacking patterns were controlled by the interaction of third- and fourth-order sea-level oscillations, with relatively uniform fourth-order oscillations altered mainly by the harmonic effect of lower-order sea-level changes. An alternative model is based on an insolation curve for the Carboniferous calculated using Milankovitch parameters. This model predicts a considerable variability in levels of solar insolation that would have affected the amplitude of fourth-order sea-level changes and cycle composition. Both of these ideas were examined via numerous model runs using CARBONATE 6.0 and a new program, CARBOSMUT. Model results were evaluated through the use of 3 key criteria derived from well-documented outcropping stratigraphies in the U.K.: (1) cycle stacking patterns and the stratigraphic position of major transgressions, (2) stratigraphic position of major faunal changes, (3) degree of development of subaerial exposure surfaces. Computer simulations and comparison with outcrop data suggest that a model invoking the interaction of relatively uniform fourth- (c.100 Kyr) and third-order sea-level oscillations is most appropriate for much of the late Visean, with major lowstands occurring at the mid-Asbian and Asbian–Brigantian boundary. Late Visean cycles are important exploration targets in the Pri-Caspian Basin, Kazakhstan and understanding the controls on stratal patterns is important as a potential exploration tool.",
    "language": "en",
    "libraryCatalog": "Wiley Online Library",
    "accessDate": "2017-08-24",
    "shortTitle": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms"
  }
]

At a first glance this looks like eqiad is not being redirected for some reason to http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2117.2002.00186.x/abstract however looking from the url-downloader host in eqiad the redirect is issued just fine.

@Mvolz any ideas ?

Wiley are checking in to ask whether the problem is now resolved. Can anyone confirm?

Hi all,

I can confirm that the problem is solved, just checked for a DOI of a Wiley
journal.

Thank you all for helping out in this!
Albert.

I don't think it has, but I 'd like a confirmation from @Mvolz. The stuff I posted in T165105#3548361 still holds true and I have a hard time understanding why. There's a clear discrepancy between the responses from the citoid service in the 2 datacenters. If the discrepancy is expected, why is it ? If not, how do we fix it ?

For what is worth, I have trouble seeing how Wiley could be responsible for the above discrepancy.