Page MenuHomePhabricator
Create Task
Maniphest T141833

500 errors from feed endpoint
Closed, ResolvedPublic
Actions

Description

There's a non-insignificant rate of 500 errors coming from the Mobile-Content-Service for request for the feed. Here's a logstash dashboard showing the errors.

Some examples:

Also there're some 500 errors for mobile-sections requests:

Details

SubjectRepoBranchLines +/-
Remove setMathFormulaImageMaxWidthmediawiki/services/mobileappsmaster+6 -28
Escape special characters before feeding into RegExpmediawiki/services/mobileappsmaster+27 -4
Featured image: handle cases where extmetadata is emptymediawiki/services/mobileappsmaster+19 -4
Most-read endpoint: Handle MWAPI responses without normalizationsmediawiki/services/mobileappsmaster+23 -3
Customize query in gerrit

Related Objects

Event Timeline

• Pchelolo created this task.Aug 1 2016, 11:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2016, 11:23 PM
• Pchelolo updated the task description. (Show Details)Aug 1 2016, 11:34 PM
• mobrovac triaged this task as High priority.Aug 2 2016, 1:46 PM
• mobrovac added projects: User-mobrovac, Services.

Full error trace:

[2016-08-02T13:44:46.796Z] ERROR: mobileapps/739 on scb1001: 500: internal_error (message="500: internal_error", status=500, type=internal_error, detail="Cannot read property 'length' of undefined", levelPath=error/500, request_id=467904c6-58b7-11e6-a452-58cc4587c4fc)
    stack: TypeError: Cannot read property 'length' of undefined
        at Object.mUtil.adjustMemberKeys (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/lib/mobile-util.js:133:28)
        at Request.<anonymous> (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/lib/feed/most-read.js:71:15)
        at Request.tryCatcher (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/util.js:16:23)
        at Promise._settlePromiseFromHandler (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:504:31)
        at Promise._settlePromise (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:561:18)
        at Promise._settlePromise0 (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:606:10)
        at Promise._settlePromises (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:685:18)
        at Promise._fulfill (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:630:18)
        at Promise._resolveCallback (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:425:57)
        at Promise._settlePromiseFromHandler (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:516:17)
        at Promise._settlePromise (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:561:18)
        at Promise._settlePromise0 (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:606:10)
        at Promise._settlePromises (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:685:18)
        at Promise._fulfill (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:630:18)
        at Promise._resolveCallback (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:425:57)
        at Promise._settlePromiseFromHandler (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:516:17)
    --
    request: {
      "url": "/ja.wikipedia.org/v1/feed/featured/2016/06/16",
      "headers": {
        "user-agent": "curl/7.38.0",
        "x-request-id": "467904c6-58b7-11e6-a452-58cc4587c4fc"
      },
      "method": "GET",
      "params": {
        "0": "/ja.wikipedia.org/v1/feed/featured/2016/06/16"
      },
      "query": {},
      "remoteAddress": "127.0.0.1",
      "remotePort": 56078
    }
• mobrovac added a comment.Aug 2 2016, 1:51 PM

And here's the stack trace for Sun_O))):

[2016-08-02T13:50:48.390Z] ERROR: mobileapps/802 on scb1001: 500: internal_error (message="500: internal_error", status=500, type=internal_error, levelPath=error/500, request_id=1e0dbb9e-58b8-11e6-9740-ad138e56ec6b)
    stack: SyntaxError: Invalid regular expression: /Sunn.?O)))/: Unmatched ')'
        at RegExp (native)
        at new RegExp (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/core-js/modules/es6.regexp.constructor.js:26:11)
        at pickPronunciationFilePageUrl (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/lib/parseProperty.js:50:27)
        at Object.parsePronunciation (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/lib/parseProperty.js:59:31)
        at buildLead (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/routes/mobile-sections.js:98:30)
        at buildAll (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/routes/mobile-sections.js:113:15)
        at /srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/src/routes/mobile-sections.js:147:20
        at tryCatcher (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/util.js:16:23)
        at Promise._settlePromiseFromHandler (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:504:31)
        at Promise._settlePromise (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:561:18)
        at Promise._settlePromise0 (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:606:10)
        at Promise._settlePromises (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:685:18)
        at Promise._fulfill (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:630:18)
        at Promise._resolveCallback (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:425:57)
        at Promise._settlePromiseFromHandler (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:516:17)
        at Promise._settlePromise (/srv/deployment/mobileapps/deploy-cache/revs/4f042c5e5eaa44bb25b634693feb3a9ba953997c/node_modules/bluebird/js/release/promise.js:561:18)
    --
    detail: Invalid regular expression: /Sunn.?O)))/: Unmatched ')'
    --
    request: {
      "url": "/vi.wikipedia.org/v1/page/mobile-sections/Sunn_O)))",
      "headers": {
        "user-agent": "curl/7.38.0",
        "x-request-id": "1e0dbb9e-58b8-11e6-9740-ad138e56ec6b"
      },
      "method": "GET",
      "params": {
        "0": "/vi.wikipedia.org/v1/page/mobile-sections/Sunn_O)))"
      },
      "query": {},
      "remoteAddress": "127.0.0.1",
      "remotePort": 35840
    }
gerritbot subscribed.Aug 2 2016, 2:38 PM

Change 302437 had a related patch set uploaded (by Mholloway):
Featured image: handle cases where extmetadata is empty

https://gerrit.wikimedia.org/r/302437

gerritbot added a comment.Aug 2 2016, 3:05 PM

Change 302444 had a related patch set uploaded (by Mholloway):
Most-read endpoint: Handle MWAPI responses without normalizations

https://gerrit.wikimedia.org/r/302444

gerritbot added a comment.Aug 2 2016, 3:22 PM

Change 302444 merged by jenkins-bot:
Most-read endpoint: Handle MWAPI responses without normalizations

https://gerrit.wikimedia.org/r/302444

gerritbot added a comment.Aug 2 2016, 3:35 PM

Change 302458 had a related patch set uploaded (by Mholloway):
Escape special characters before feeding into RegExp

https://gerrit.wikimedia.org/r/302458

• Mholloway added a comment.Aug 2 2016, 3:43 PM

Stack trace for the 2016/05/23 errors, which are different from the 2016/06/16 error:

"name":"service-mobileapp-node","hostname":"wmf1256.local","pid":16471,"level":50,"message":"500: internal_error","stack":"TypeError: Cannot read property 'payload' of undefined\n
  at /Users/mholloway/mobileapps/routes/aggregated.js:47:34\n
  at tryCatcher (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/util.js:16:23)\n    
  at Promise._settlePromiseFromHandler (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:504:31)\n
  at Promise._settlePromise (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:561:18)\n
  at Promise._settlePromise0 (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:606:10)\n
  at Promise._settlePromises (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:685:18)\n
  at Promise._fulfill (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:630:18)\n
  at PropertiesPromiseArray.PromiseArray._resolve (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise_array.js:125:19)\n
  at PropertiesPromiseArray._promiseFulfilled (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/props.js:78:14)\n
  at Promise._settlePromise (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:566:26)\n
  at Promise._settlePromise0 (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:606:10)\n
  at Promise._settlePromises (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:685:18)\n
  at Promise._fulfill (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:630:18)\n
  at Promise._resolveCallback (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:425:57)\n
  at Promise._settlePromiseFromHandler (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:516:17)\n
  at Promise._settlePromise (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:561:18)","status":500,"type":"internal_error","detail":"Cannot read property 'payload' of undefined","levelPath":"error/500","request_id":"9dfdf021-58c7-11e6-b859-7e80f4ef408a","request":{"url":"/en.wikipedia.org/v1/feed/featured/2016/05/23","headers":{"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36","x-request-id":"9dfdf021-58c7-11e6-b859-7e80f4ef408a"},"method":"GET","params":{"0":"/en.wikipedia.org/v1/feed/featured/2016/05/23"},"query":{},"remoteAddress":"127.0.0.1","remotePort":63162},"msg":"500: internal_error","time":"2016-08-02T15:41:46.144Z","v":0}
gerritbot added a comment.Aug 2 2016, 4:18 PM

Change 302437 merged by jenkins-bot:
Featured image: handle cases where extmetadata is empty

https://gerrit.wikimedia.org/r/302437

• Pchelolo added a comment.Aug 2 2016, 6:35 PM

Found one more: https://de.wikipedia.org/api/rest_v1/page/mobile-sections/Verallgemeinerter_Laplace-Operator

• Mholloway added a comment.Aug 2 2016, 6:50 PM

Hmm, this one looks like a bug in Domino, somewhat similar to T124446: TypeError: Cannot set property 'text-decoration' of undefined:

"name":"service-mobileapp-node","hostname":"wmf1256.local","pid":18667,"level":50,"message":"500: internal_error",
"stack":"TypeError: Cannot read property 'type' of undefined\n
  at TokenStreamBase.LA (/Users/mholloway/mobileapps/node_modules/domino/lib/cssparser.js:810:29)\n
  at TokenStreamBase.advance (/Users/mholloway/mobileapps/node_modules/domino/lib/cssparser.js:683:20)\n
  at Parser.additions._readDeclarations (/Users/mholloway/mobileapps/node_modules/domino/lib/cssparser.js:3402:42)\n
  at Parser.additions.parseStyleAttribute (/Users/mholloway/mobileapps/node_modules/domino/lib/cssparser.js:3581:22)\n
  at parseStyles (/Users/mholloway/mobileapps/node_modules/domino/lib/CSSStyleDeclaration.js:23:10)\n
  at Object.create._parsed.get (/Users/mholloway/mobileapps/node_modules/domino/lib/CSSStyleDeclaration.js:38:28)\n
  at Object.create.setProperty.value (/Users/mholloway/mobileapps/node_modules/domino/lib/CSSStyleDeclaration.js:126:22)\n
  at Object.defineProperty.set (/Users/mholloway/mobileapps/node_modules/domino/lib/CSSStyleDeclaration.js:304:12)\n
  at Object.setMathFormulaImageMaxWidth (/Users/mholloway/mobileapps/lib/transformations/setMathFormulaImageMaxWidth.js:17:40)\n
  at Object.transforms.addRequiredMarkup (/Users/mholloway/mobileapps/lib/transforms.js:233:33)\n
  at Request.<anonymous> (/Users/mholloway/mobileapps/lib/parsoid-access.js:153:24)\n
  at Request.tryCatcher (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/util.js:16:23)\n
  at Promise._settlePromiseFromHandler (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:504:31)\n
  at Promise._settlePromise (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:561:18)\n
  at Promise._settlePromise0 (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:606:10)\n
  at Promise._settlePromises (/Users/mholloway/mobileapps/node_modules/bluebird/js/release/promise.js:685:18)","status":500,"type":"internal_error","detail":"Cannot read property 'type' of undefined","levelPath":"error/500","request_id":"0d05d231-58e1-11e6-96b1-233285f092c5","request":{"url":"/de.wikipedia.org/v1/page/mobile-sections/Verallgemeinerter_Laplace-Operator","headers":{"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36","x-request-id":"0d05d231-58e1-11e6-96b1-233285f092c5"},"method":"GET","params":{"0":"/de.wikipedia.org/v1/page/mobile-sections/Verallgemeinerter_Laplace-Operator"},"query":{},"remoteAddress":"127.0.0.1","remotePort":62014},"msg":"500: internal_error","time":"2016-08-02T18:43:49.490Z","v":0
• Mholloway added a comment.Aug 2 2016, 10:55 PM

Filed https://github.com/fgnass/domino/issues/86.

gerritbot added a comment.Aug 3 2016, 5:14 PM

Change 302458 merged by jenkins-bot:
Escape special characters before feeding into RegExp

https://gerrit.wikimedia.org/r/302458

• Pchelolo added a comment.Aug 11 2016, 7:38 PM

I guess everything is done on this one? I haven't seen these errors for a while now. Can we close?

• mobrovac added a comment.Aug 12 2016, 12:13 PM
In T141833#2545166, @Pchelolo wrote:

I guess everything is done on this one? I haven't seen these errors for a while now. Can we close?

I'd rather leave it open until https://github.com/fgnass/domino/issues/86 is resolved.

gerritbot added a comment.Aug 17 2016, 11:23 AM

Change 305225 had a related patch set uploaded (by Mholloway):
Remove setMathFormulaImageMaxWidth

https://gerrit.wikimedia.org/r/305225

gerritbot added a project: Patch-For-Review.Aug 17 2016, 11:23 AM
• bearND moved this task from Incoming to Backlog on the Mobile-Content-Service board.Aug 19 2016, 6:42 PM
gerritbot added a comment.Aug 23 2016, 1:49 PM

Change 305225 merged by jenkins-bot:
Remove setMathFormulaImageMaxWidth

https://gerrit.wikimedia.org/r/305225

• bearND closed this task as Resolved.Aug 26 2016, 6:09 PM
• bearND claimed this task.
• bearND moved this task from Backlog to To Deploy on the Mobile-Content-Service board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL