Some requests for DOIs are failing or very slow; if we have a DOI and the request is taking too long, just use CrossRef data instead.
Closed, ResolvedPublic

There are a very large number of changes, so older changes are hidden. Show Older Changes

These are working on citoid on master directly. Seems to be a problem with the RestBase wrapper.

Mvolz renamed this task from Citoid: several publishers recently missing? to Some search queries that work on citoid directly are broken from restbase.May 25 2017, 10:46 AM
Mvolz added a project: RESTBase.
Mvolz added a subscriber: mobrovac.
Mvolz renamed this task from Some search queries that work on citoid directly are broken from restbase to Some search queries that work on citoid directly are causing unknown hyperswitch errors in restbase.May 25 2017, 10:50 AM
Mvolz raised the priority of this task from Normal to High.
Mvolz updated the task description. (Show Details)
Mvolz added a subscriber: Pchelolo.
Mvolz moved this task from Backlog to Production on the Citoid board.May 25 2017, 11:13 AM
mobrovac renamed this task from Some search queries that work on citoid directly are causing unknown hyperswitch errors in restbase to Citoid confuses DOI with URL in production.May 29 2017, 7:41 PM

For some reason, Citoid in production resolves the given input as a URL instead of a DOI. Issuing a query for localhost:1970/api?format=mediawiki&search=10.1029%2F94WR00436 directly on a production host results in:

{"Error":"Invalid host supplied"}

Citoid logs show that it blocked the request because it thinks it should look for a private IP address:

http://127.0.0.1/?DOI=10.1029/94WR00436 is not public, and is disallowed (levelPath=warn/hostIsAllowed)

Further investigation is needed here.

Full debug log from production:

[2017-05-29T21:45:22.883Z]  INFO: citoid/17369 on scb1002: Worker 17369 listening on 0.0.0.0:1977 (levelPath=info)
[2017-05-29T21:45:34.596Z] DEBUG: citoid/17369 on scb1002: Adding response function (levelPath=debug/CitoidService)
[2017-05-29T21:45:34.598Z] DEBUG: citoid/17369 on scb1002: DOI detected (levelPath=debug/CitoidService)
[2017-05-29T21:45:34.598Z] DEBUG: citoid/17369 on scb1002: requestFromDOI method (levelPath=debug/CitoidService)
[2017-05-29T21:45:35.053Z] DEBUG: citoid/17369 on scb1002: Resolved DOI 10.1029/94wr00436 to URL http://doi.wiley.com/10.1029/94WR00436; Sending to requestFromURL (levelPath=debug/DOI)
[2017-05-29T21:45:35.054Z] DEBUG: citoid/17369 on scb1002: requestFromURL method (levelPath=debug/CitoidService)
[2017-05-29T21:45:36.636Z] DEBUG: citoid/17369 on scb1002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.637Z] DEBUG: citoid/17369 on scb1002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.656Z] DEBUG: citoid/17369 on scb1002: Zotero request made for: http://doi.wiley.com/10.1029/94WR00436 (levelPath=debug/zotero)
[2017-05-29T21:45:36.659Z] DEBUG: citoid/17369 on scb1002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:36.659Z] DEBUG: citoid/17369 on scb1002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.736Z] DEBUG: citoid/17369 on scb1002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.736Z] DEBUG: citoid/17369 on scb1002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-29T21:45:37.780Z]  WARN: citoid/17369 on scb1002: http://127.0.0.1/?DOI=10.1029/94WR00436 is not public, and is disallowed (levelPath=warn/hostIsAllowed)

Super weird. That's not really confusing url with doi though; request from URL is normal after a request from DOI, and we don't make any requests that look like http://127.0.0.1/?DOI=10.1029/94WR00436 . Just looking at that log it looks like wiley is making some weird redirect stuff which we knew was happening before.

We should make sure that even if it chokes during the redirect though that we still use crossref. If it fails when checking a url, citoid doesn't let it proceed to the scraper at all, which is where we request extra info from crossref if we aren't actually able to scrape.

Mvolz renamed this task from Citoid confuses DOI with URL in production to Wiley requests for DOI and some other publishers don't work in production.May 30 2017, 12:59 PM
Mvolz updated the task description. (Show Details)

Oddly, the request completes fine in CODFW (the other DC):

[2017-05-30T13:27:05.324Z] DEBUG: citoid/102520 on scb2002: Adding response function (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.325Z] DEBUG: citoid/102520 on scb2002: DOI detected (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.325Z] DEBUG: citoid/102520 on scb2002: requestFromDOI method (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.874Z] DEBUG: citoid/102520 on scb2002: Resolved DOI 10.1029/94wr00436 to URL http://doi.wiley.com/10.1029/94WR00436; Sending to requestFromURL (levelPath=debug/DOI)
[2017-05-30T13:27:05.875Z] DEBUG: citoid/102520 on scb2002: requestFromURL method (levelPath=debug/CitoidService)
[2017-05-30T13:27:05.875Z] TRACE: citoid/102520 on scb2002: Resolving hostname doi.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:05.876Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:05.877Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.415Z] TRACE: citoid/102520 on scb2002: resolve6Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.416Z] DEBUG: citoid/102520 on scb2002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.417Z] DEBUG: citoid/102520 on scb2002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.431Z] DEBUG: citoid/102520 on scb2002: Zotero request made for: http://doi.wiley.com/10.1029/94WR00436 (levelPath=debug/zotero)
[2017-05-30T13:27:07.432Z] TRACE: citoid/102520 on scb2002: No Zotero translator found, looking for redirects (levelPath=trace/zotero)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: Resolving hostname doi.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] TRACE: citoid/102520 on scb2002: resolve6Async ran: 1,199.171.202.200 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.433Z] DEBUG: citoid/102520 on scb2002: Resolved doi.wiley.com to 1, 199.171.202.200 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.434Z] DEBUG: citoid/102520 on scb2002: doi.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:07.561Z] TRACE: citoid/102520 on scb2002: Resolving hostname onlinelibrary.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:07.562Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.097Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.195 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.366Z] DEBUG: citoid/102520 on scb2002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:09.366Z] DEBUG: citoid/102520 on scb2002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: Resolving hostname onlinelibrary.wiley.com (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: lookupAsync ran: 1 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:09.581Z] TRACE: citoid/102520 on scb2002: resolve4Async ran: 1,199.171.202.195 (levelPath=trace/hostIsAllowed)
[2017-05-30T13:27:11.136Z] DEBUG: citoid/102520 on scb2002: Resolved onlinelibrary.wiley.com to 1, 199.171.202.195 (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:11.136Z] DEBUG: citoid/102520 on scb2002: onlinelibrary.wiley.com is public, and is allowed (levelPath=debug/hostIsAllowed)
[2017-05-30T13:27:11.736Z] TRACE: citoid/102520 on scb2002: Redirect detected to http://onlinelibrary.wiley.com/doi/10.1029/94WR00436/abstract;jsessionid=60DF91115ABB533A606FA3A5B86C0409.f04t03 (levelPath=trace/zotero)
[2017-05-30T13:27:13.158Z] DEBUG: citoid/102520 on scb2002: Zotero request made for: http://onlinelibrary.wiley.com/doi/10.1029/94WR00436/abstract;jsessionid=60DF91115ABB533A606FA3A5B86C0409.f04t03 (levelPath=debug/zotero)
[2017-05-30T13:27:13.203Z] DEBUG: citoid/102520 on scb2002: Successfully retrieved body from Zotero (levelPath=debug/zotero)
[2017-05-30T13:27:13.839Z] DEBUG: citoid/102520 on scb2002: PubMed query made for: https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?tool=citoid&email=citoid@mediawiki&format=json&ids=10.1029%2F94WR00436 (levelPath=debug/pubmed)
Joe added a subscriber: Joe.May 30 2017, 2:20 PM

So the problem is the eqiad proxy cached a redirect to localhost, likely sent by the remote host during an outage

1496153701.136     45 10.64.16.21 TCP_MISS/302 556 GET http://onlinelibrary.wiley.com/resolve/doi? - HIER_DIRECT/199.171.202.195 text/html [User-Agent: Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)\r\nCookie: OLProdServerID=1027\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 0\r\nConnection: close\r\nHost: onlinelibrary.wiley.com\r\n] [HTTP/1.1 302 Found\r\nDate: Tue, 30 May 2017 14:14:59 GMT\r\nServer: Apache\r\nLocation: http://127.0.0.1/?DOI=10.1111/1467-7660.00159\r\nContent-Length: 229\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r]

So clearing that from the cache should solve the issue.

Mvolz added a comment.May 30 2017, 2:23 PM

@Joe aha, thanks!

How long do these caches last? This problem been going on for a month apparently!

Joe added a comment.May 30 2017, 2:28 PM

Actually it was a dumb comment - the log I pasted clearly reported TCP_MISS/302, so I'm not sure what's happening. Investigating further.

Joe added a comment.May 30 2017, 2:33 PM

In fact, I suppose the problem is our proxy IP in eqiad has been banned. From the proxy machine

curl -v 'http://onlinelibrary.wiley.com/resolve/doi?DOI=10.1029/2005GB002488'
* Hostname was NOT found in DNS cache
*   Trying 199.171.202.195...
* Connected to onlinelibrary.wiley.com (199.171.202.195) port 80 (#0)
> GET /resolve/doi?DOI=10.1029/2005GB002488 HTTP/1.1
> User-Agent: curl/7.38.0
> Host: onlinelibrary.wiley.com
> Accept: */*
> 
< HTTP/1.1 302 Found
< Set-Cookie: OLProdServerID=1024; domain=.wiley.com; path=/
< Date: Tue, 30 May 2017 14:29:32 GMT
* Server Apache is not blacklisted
< Server: Apache
< Location: http://127.0.0.1/?DOI=10.1029/2005GB002488
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://127.0.0.1/?DOI=10.1029/2005GB002488">here</a>.</p>
</body></html>
* Closing connection 0

as you can see, the response redirects to 127.0.0.1.

I've verified from other hosts that can communicate directly and the responses we get are correct.

We might want to contact the admins at wiley.com to get whitelisted possibly.

In the meanwhile, we could try using the codfw proxy, but I suspect we'd only end up getting it banned as well.

Mentioned in SAL (#wikimedia-operations) [2017-05-30T14:53:09Z] <_joe_> failing citoid over to codfw, T165105

We might want to contact the admins at wiley.com to get whitelisted possibly.

@Jdforrester-WMF do we have a way of contacting Wiley? It would be awesome if they could white-list our IPs (we basically need two, one per DC).

In the meanwhile, we could try using the codfw proxy, but I suspect we'd only end up getting it banned as well.

We opted for this work-around for the time being, but it's not going to last us long...

Mvolz added a comment.May 31 2017, 8:01 AM

I've sent a cold e-mail to technical support.

Mvolz lowered the priority of this task from High to Normal.May 31 2017, 9:06 AM
Mvolz claimed this task.Jun 12 2017, 11:20 AM
Mvolz updated the task description. (Show Details)

No response to the e-mail.

They only seem to be blocking our native scraper, probably by UA, and not Zotero (which uses a Mozilla UA) which is why using the urls directly and not trying to follow the redirect from the doi seems to be working okay.

The initial redirect is handled by doi.dx which gives us a public and a correct url, which we could send to Zotero to get a result. But I guess the problem is that we're following the chain of redirects after that to the end, where we're blocked.

We could actually stop in the middle of the redirect chain and check every url to see if private and and send that to Zotero (which doesn't follow redirects at all) - maybe another solution? Although potentially slow.

Mvolz updated the task description. (Show Details)Jun 12 2017, 11:28 AM

@Samwalton9 are you in contact with Wiley through TWL? (Or is someone else?) Might you be able to help us get in touch? Summary: We need our server whitelisted so that we can provide automated lookup of their citation information

Not directly, but I might have someone who does have direct contact. Could you give me a TL;DR of the above? Not sure I understood the details - what is it that needs whitelisting?

Not directly, but I might have someone who does have direct contact. Could you give me a TL;DR of the above? Not sure I understood the details - what is it that needs whitelisting?

We have two data centers that run Citoid. We believe Wiley has blocked one of the two data centre's IP addresses.

Citoid is a webscraper that fills in citation information on Wikipedia. So for instance, a person tries to cite a Wiley paper on Wikipedia using our tool, citoid will take that URL, try to access the page, and then fill in the title of the paper, the authors, etc. so the person doesn't have to fill in that information manually. Does this help?

Yes - thanks! Do you have the IP address, so I can pass it along?

Yes - thanks! Do you have the IP address, so I can pass it along?

@Joe @mobrovac can you send the two IP addresses that we need to be whitelisted to @Samwalton9? Thanks :).

Yes - thanks! Do you have the IP address, so I can pass it along?

@Joe @mobrovac can you send the two IP addresses that we need to be whitelisted to @Samwalton9? Thanks :).

@akosiaris @Joe ping ^

Joe added a comment.Jun 21 2017, 7:29 AM

Hey sorry I was on vacation last week.

The IPs of our proxies are:

webproxy.eqiad.wmnet - 208.80.154.86
webproxy.codfw.wmnet - 208.80.153.53

akosiaris added a comment.EditedJun 21 2017, 8:22 AM

Hey sorry I was on vacation last week.

The IPs of our proxies are:

webproxy.eqiad.wmnet - 208.80.154.86
webproxy.codfw.wmnet - 208.80.153.53

Not that's not correct. It's the url-downloaders that are used for applications, not the webproxies. But sending just these IP address would unnecessarily restrict us in the network operations we can do as we would not be able to use different IP addresses for those services. @Samwalton9 could you please ask them to whitelist our entire address space ? That would be 208.80.152.0/22 for both Datacenters. That IPs used from that space are strictly allocated so they 'll never see any illegitimate traffic. And we will always reacting ASAP to any kind of abuse report.

Thanks - I've sent an email to our contacts at Cochrane, who will hopefully be able to pass the message on to the right people at Wiley. I'll let you know if/when I hear back.

Change 360832 had a related patch set uploaded (by Mvolz; owner: Marielle Volz):
[mediawiki/services/citoid@master] [WIP] Fallback to crossRef for restricted URLs

https://gerrit.wikimedia.org/r/360832

Change 360832 merged by jenkins-bot:
[mediawiki/services/citoid@master] Fallback to crossRef for restricted URLs

https://gerrit.wikimedia.org/r/360832

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:06:29Z] <mobrovac@tin> Started deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:06:29Z] <mobrovac@tin> Started deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:09:21Z] <mobrovac@tin> Finished deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105) (duration: 02m 52s)

Mentioned in SAL (#wikimedia-operations) [2017-07-04T15:09:21Z] <mobrovac@tin> Finished deploy [citoid/deploy@9d22567]: Fallback to crossRef (T165105) and use MarcXML (T165105) (duration: 02m 52s)

Mvolz updated the task description. (Show Details)
Mvolz removed Mvolz as the assignee of this task.

I'm told that Wiley "should be able to provide the necessary info this week."

Thanks for chasing this down @Samwalton9!

GWicke moved this task from blocked to watching on the Services board.Jul 18 2017, 9:06 PM
GWicke edited projects, added Services (watching); removed Services (blocked).
elukey added a subscriber: elukey.Jul 19 2017, 8:02 AM

I received this response today:

I have confirmed with network and Sys-Ops team that none of the IP from the range are banned at firewall or application level. Could you please request customer to try again.

IP range (208.80.152.0/22)

If the issue persist please request them to execute the following 9 commands from datacentre IP (any IP from where he is unable to access WOL) and get back with the result of each command executed to diagnose their connection to OL:

  1. ipconfig/all
  2. nslookup onlinelibrarystatic.wiley.com
  3. tracert onlinelibrarystatic.wiley.com
  4. ping onlinelibrarystatic.wiley.com
  5. pathping onlinelibrarystatic.wiley.com
  6. nslookup onlinelibrary.wiley.com
  7. tracert onlinelibrary.wiley.com
  8. ping onlinelibrary.wiley.com
  9. pathping onlinelibrary.wiley.com These information will help us a lot. Please let us know if you have any further query.

I think we have been unblocked btw

$ curl -i --resolve en.wikipedia.org:443:208.80.154.224 https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x

HTTP/2 200
....
[{"itemType":"journalArticle","creators":[{"firstName":"A. J.","lastName":"Barnett","creatorType":"author"},{"firstName":"P. M.","lastName":"Burgess","creatorType":"author"},{"firstName":"V. P.","lastName":"Wright","creatorType":"author"}],"notes":[],"tags":[],"title":"Icehouse world sea-level behaviour and...

Shall we re-enable the other DC now?

I think we have been unblocked btw

$ curl -i --resolve en.wikipedia.org:443:208.80.154.224 https://en.wikipedia.org/api/rest_v1/data/citation/zotero/10.1046%2Fj.1365-2117.2002.00186.x

HTTP/2 200
....
[{"itemType":"journalArticle","creators":[{"firstName":"A. J.","lastName":"Barnett","creatorType":"author"},{"firstName":"P. M.","lastName":"Burgess","creatorType":"author"},{"firstName":"V. P.","lastName":"Wright","creatorType":"author"}],"notes":[],"tags":[],"title":"Icehouse world sea-level behaviour and...

Actually that was the wrong paste. The correct one is

$ curl -i 'http://citoid.svc.eqiad.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x'
HTTP/1.1 200 OK
...

[{"url":"http://doi.wiley.com/10.1046/j.1365-2117.2002.00186.x","itemType":"journalArticle","pages":"417–438","creators":[{"creatorType":"author","lastName":"Barnett","firstName":"A. J."},{"creatorType":"author","lastName":"Burgess","firstName":"P. M."},{"creatorType":"author","lastName":"Wright","firstName":"V. P."}],"title":"Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies","publicationTitle":"Basin Research","volume":"14","issue":"3"}]

What's concerning is that I checked today once more and it seems eqiad and codfw returning different data. e.g.

akosiaris@bast1001:~$ curl 'http://citoid.svc.eqiad.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   576  100   576    0     0    780      0 --:--:-- --:--:-- --:--:--   780
[
  {
    "url": "http://doi.wiley.com/10.1046/j.1365-2117.2002.00186.x",
    "itemType": "journalArticle",
    "pages": "417–438",
    "creators": [
      {
        "creatorType": "author",
        "lastName": "Barnett",
        "firstName": "A. J."
      },
      {
        "creatorType": "author",
        "lastName": "Burgess",
        "firstName": "P. M."
      },
      {
        "creatorType": "author",
        "lastName": "Wright",
        "firstName": "V. P."
      }
    ],
    "title": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies",
    "publicationTitle": "Basin Research",
    "volume": "14",
    "issue": "3"
  }

while codfw returns more data

akosiaris@bast1001:~$ curl 'http://citoid.svc.codfw.wmnet:1970/api?format=zotero&search=10.1046%2Fj.1365-2117.2002.00186.x' | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3038  100  3038    0     0   1056      0  0:00:02  0:00:02 --:--:--  1056
[
  {
    "itemType": "journalArticle",
    "creators": [
      {
        "firstName": "A. J.",
        "lastName": "Barnett",
        "creatorType": "author"
      },
      {
        "firstName": "P. M.",
        "lastName": "Burgess",
        "creatorType": "author"
      },
      {
        "firstName": "V. P.",
        "lastName": "Wright",
        "creatorType": "author"
      }
    ],
    "notes": [],
    "tags": [],
    "title": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms: integration of numerical forward modelling and outcrop studies",
    "publicationTitle": "Basin Research",
    "volume": "14",
    "issue": "3",
    "ISSN": "1365-2117",
    "url": "http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2117.2002.00186.x/abstract",
    "DOI": "10.1046/j.1365-2117.2002.00186.x",
    "pages": "417–438",
    "date": "2002-09-01",
    "abstractNote": "Late Visean (Asbian–Brigantian) platform carbonates in the British Isles show a pronounced cyclicity marked by the alternation of mainly subtidal carbonates and subaerial exposure surfaces. Whereas the cyclicity of shallow-water limestones of this age has been well-documented, there has been little attempt to understand the controls on larger-scale patterns such as those recognized in Pennsylvanian successions in the U.S.A. The principal aim of this study is to test two contrasting theories of cycle stacking via numerical forward modelling. Earlier studies of Pennsylvanian-early Permian platform carbonates in south-west U.S.A. suggest that cycle stacking patterns were controlled by the interaction of third- and fourth-order sea-level oscillations, with relatively uniform fourth-order oscillations altered mainly by the harmonic effect of lower-order sea-level changes. An alternative model is based on an insolation curve for the Carboniferous calculated using Milankovitch parameters. This model predicts a considerable variability in levels of solar insolation that would have affected the amplitude of fourth-order sea-level changes and cycle composition. Both of these ideas were examined via numerous model runs using CARBONATE 6.0 and a new program, CARBOSMUT. Model results were evaluated through the use of 3 key criteria derived from well-documented outcropping stratigraphies in the U.K.: (1) cycle stacking patterns and the stratigraphic position of major transgressions, (2) stratigraphic position of major faunal changes, (3) degree of development of subaerial exposure surfaces. Computer simulations and comparison with outcrop data suggest that a model invoking the interaction of relatively uniform fourth- (c.100 Kyr) and third-order sea-level oscillations is most appropriate for much of the late Visean, with major lowstands occurring at the mid-Asbian and Asbian–Brigantian boundary. Late Visean cycles are important exploration targets in the Pri-Caspian Basin, Kazakhstan and understanding the controls on stratal patterns is important as a potential exploration tool.",
    "language": "en",
    "libraryCatalog": "Wiley Online Library",
    "accessDate": "2017-08-24",
    "shortTitle": "Icehouse world sea-level behaviour and resulting stratal patterns in late Visean (Mississippian) carbonate platforms"
  }
]

At a first glance this looks like eqiad is not being redirected for some reason to http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2117.2002.00186.x/abstract however looking from the url-downloader host in eqiad the redirect is issued just fine.

@Mvolz any ideas ?

Wiley are checking in to ask whether the problem is now resolved. Can anyone confirm?

Hi all,

I can confirm that the problem is solved, just checked for a DOI of a Wiley
journal.

Thank you all for helping out in this!
Albert.

I don't think it has, but I 'd like a confirmation from @Mvolz. The stuff I posted in T165105#3548361 still holds true and I have a hard time understanding why. There's a clear discrepancy between the responses from the citoid service in the 2 datacenters. If the discrepancy is expected, why is it ? If not, how do we fix it ?

For what is worth, I have trouble seeing how Wiley could be responsible for the above discrepancy.

Acamicamacaraca moved this task from Waiting on Deploy to Production on the Citoid board.

Since there's "some other publishers" in the title, here are two examples of non working ones:

  • 10.1152/ajpendo.00361.2010 (physiology.org)
  • 10.1177/0883073809337162 (journals.sagepub.com)

While they fail on Wikipedia because of the 20 seconds timeout, they work on the REST API (https://www.mediawiki.org/api/rest_v1/#!/Citation/getCitation) after about 33 seconds for both.

From Twitter above:

  • 10.1080/03014223.1975.9517878 (tandfonline.com)

The delay for this one is about 43 seconds.

Couldn't the 20 seconds timeout be increased to, say, 60 seconds?
https://phabricator.wikimedia.org/diffusion/ECEX/browse/HEAD/modules/ve.ui.CiteFromIdInspector.js;383a79c531a1534a48cdaf6428bfe8d7308d5596$87

I'd rather wait a few more seconds than spend 5 minutes to fill all the fields.

As far as http://www.tandfonline.com/doi/abs/10.1080/03014223.1975.9517878 goes

$ curl 'http://citoid.svc.codfw.wmnet:1970/api?format=zotero&search=10.1080%2F03014223.1975.9517878' | jq '.'
[
  {
    "url": "http://www.tandfonline.com/doi/abs/10.1080/03014223.1975.9517878",
    "itemType": "journalArticle",
    "pages": "265–363",
    "creators": [
      {
        "creatorType": "author",
        "lastName": "Gaskin",
        "firstName": "D. E."
      }
    ],
    "title": "Revision of the New Zealand Crambini (Lepidoptera: Pyralidae: Crambinae)",
    "publicationTitle": "New Zealand Journal of Zoology",
    "volume": "2",
    "issue": "3"
  }
]

which is the same as

$ curl 'http://citoid.svc.eqiad.wmnet:1970/api?format=zotero&search=10.1080%2F03014223.1975.9517878' | jq '.'
[
  {
    "url": "http://www.tandfonline.com/doi/abs/10.1080/03014223.1975.9517878",
    "itemType": "journalArticle",
    "pages": "265–363",
    "creators": [
      {
        "creatorType": "author",
        "lastName": "Gaskin",
        "firstName": "D. E."
      }
    ],
    "title": "Revision of the New Zealand Crambini (Lepidoptera: Pyralidae: Crambinae)",
    "publicationTitle": "New Zealand Journal of Zoology",
    "volume": "2",
    "issue": "3"
  }
]

So this is not the same as the issue I reported in T165105#3548361, which still holds true btw.

For what is worth, I think we are not blocked or anything like it but rather have some issues in our infrastructure, so this is probably NO longer the task to report problems like the ones reported above.

Mvolz added a comment.Feb 1 2018, 1:55 PM

From Twitter above:

That's definitely an option, but I think as a first pass, particularly if we have DOI, we can get good metadata from crossRef and we should actually just skip resolving the doi, then using Zotero, and merging the results, which is what we currently do. Or race them and pick whichever is faster. 34 seconds is a long time for user to wait; I even get annoyed at the 20 seconds sometimes.

Mvolz renamed this task from Wiley requests for DOI and some other publishers don't work in production to Some requests for DOIs are failing or very slow; if we have a DOI and the request is taking too long, just use CrossRef data instead..Feb 1 2018, 1:57 PM
Mvolz claimed this task.
Mvolz updated the task description. (Show Details)Feb 1 2018, 2:00 PM

I'd rather wait a few more seconds than spend 5 minutes to fill all the fields.

That's definitely an option, but I think as a first pass, particularly if we have DOI, we can get good metadata from crossRef and we should actually just skip resolving the doi, then using Zotero, and merging the results, which is what we currently do. Or race them and pick whichever is faster. 34 seconds is a long time for user to wait; I even get annoyed at the 20 seconds sometimes.

There are also operational issues with very long timeouts. They reserve resources and not release them for use by other users, effectively denying them service if in any kind of system stress. And that's without actually providing a benefit to the first group of users. At least that's what research suggests. Per [1], 40 percent of consumers will wait no more than three seconds for a web page to render and that's numbers from 2009. We can probably assume very few people will wait >20 or even >30 secs for something to happen in the page. And given that might very well endanger the experience for other editors, it's probably prudent to not increase the timeout more.

[1] https://www.akamai.com/us/en/about/news/press/2009-press/akamai-reveals-2-seconds-as-the-new-threshold-of-acceptability-for-ecommerce-web-page-response-times.jsp

Change 421876 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[mediawiki/services/citoid@master] [POC] Race crossRef with native scraper

https://gerrit.wikimedia.org/r/421876

Mvolz added a comment.Apr 23 2018, 9:22 AM

Change 421876 had a related patch set uploaded (by Mvolz; owner: Mvolz):
[mediawiki/services/citoid@master] [POC] Race crossRef with native scraper

https://gerrit.wikimedia.org/r/421876

POC which just races them. Crossref will *always* be faster since the complete method does crossRef as well, unless crossRef goes down and fails completely (which happened Thursday!) so I guess it's not entirely pointless.

So I guess what needs to be decided is a) do we give up on trying to get the other metadata. It typically includes pmid, pmc, language, and abstract, which aren't necessarily that important. This is the behaviour of the wikitext ref tool bar, fyi, which doesn't bother with citoid if a doi is provided.

Or should we try to give the response time an explicit shorter timeout somewhere?

Perhaps we should provide different end points in Citoid for these two behaviours and let clients decide if they want the full response, or a shorter/quicker metadata look-up?

Perhaps we should provide different end points in Citoid for these two behaviours and let clients decide if they want the full response, or a shorter/quicker metadata look-up?

Ideally, yes, particularly if we have have bots that use it - but for the current use case it's less important. Maybe we should just do it setting it manually in the config and then revisit if we need to?

Set the config manually to what, and for what? :P

Set the config manually to what, and for what? :P

Sorry, setting the timeout :). see WIP https://gerrit.wikimedia.org/r/#/c/421876/

Change 421876 merged by jenkins-bot:
[mediawiki/services/citoid@master] Force early return for some DOIs

https://gerrit.wikimedia.org/r/421876

Mentioned in SAL (#wikimedia-operations) [2018-06-29T10:18:06Z] <mobrovac@deploy1001> Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853

Mentioned in SAL (#wikimedia-operations) [2018-06-29T10:21:32Z] <mobrovac@deploy1001> Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 (duration: 03m 26s)

Mvolz closed this task as Resolved.Sun, Jul 1, 12:57 PM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptSun, Jul 1, 12:57 PM