Page MenuHomePhabricator
Create Task
Maniphest T296421

Function calls on Not Wikilambda produce Internal Server Error
Closed, ResolvedPublic
Actions

Description

In T290777#7526408, @Jdforrester-WMF wrote:

AFAICS all function calls on NotWikilambda are erroring right now with an 'Internal Server Error', e.g. https://notwikilambda.toolforge.org/wiki/Z777 and manually-created calls even to Echo or If.

Related Objects

Event Timeline

LucasWerkmeister created this task.Nov 24 2021, 5:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 24 2021, 5:31 PM
LucasWerkmeister added a comment.Nov 24 2021, 5:31 PM

It’s this “kad” nonsense again (I vaguely remember issues with it, they went away at some point):

kubectl logs pod/function-evaluator-6dc9fddbf5-jtjsg
> service-runner

internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module 'kad'
Require stack:
- /srv/function-evaluator/node_modules/limitation/lib/kad_backend.js
- /srv/function-evaluator/node_modules/limitation/index.js
- /srv/function-evaluator/node_modules/service-runner/lib/ratelimiter.js
- /srv/function-evaluator/node_modules/service-runner/lib/worker.js
- /srv/function-evaluator/node_modules/service-runner/lib/master.js
- /srv/function-evaluator/node_modules/service-runner/service-runner.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/srv/function-evaluator/node_modules/limitation/lib/kad_backend.js:4:26)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/srv/function-evaluator/node_modules/limitation/index.js:18:18)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/srv/function-evaluator/node_modules/limitation/lib/kad_backend.js',
    '/srv/function-evaluator/node_modules/limitation/index.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/ratelimiter.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/worker.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/master.js',
    '/srv/function-evaluator/node_modules/service-runner/service-runner.js'
  ]
}
npm ERR! code 1
npm ERR! path /srv/function-evaluator
npm ERR! command failed
npm ERR! command sh -c service-runner
LucasWerkmeister mentioned this in T290777: NotWikiLambda shows failures on many tests with Guzzle errors.Nov 24 2021, 5:32 PM
LucasWerkmeister added a comment.Nov 24 2021, 5:35 PM

Hm, in the function-evaluator package-lock, limitation has a dependency on wikimedia-kad-fork, but in node_modules/limitation/package.json it’s still a dependency on kad (albeit on the version wikimedia/kad#master), and at runtime, its lib/kad_backend.js still requires kad, so it’s not going to find whatever this wikimedia-kad-fork is.

Stashbot added a comment.Nov 24 2021, 5:39 PM

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:39:31Z] <wm-bot> <lucaswerkmeister> deleted function-evaluator deployment T296421

Stashbot added a comment.Nov 24 2021, 5:47 PM

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:47:27Z] <wm-bot> <lucaswerkmeister> ran npm ci in function-evaluator (in webservice node12 shell), for some reason node_modules/limitation was on 0.2.2 despite package-lock.json having 0.2.3 (T296421)

Stashbot added a comment.Nov 24 2021, 5:50 PM

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:50:03Z] <wm-bot> <lucaswerkmeister> recreated function-evaluator deployment (still on commit 8731f2ff6e, no changes) T296421

LucasWerkmeister added a comment.Nov 24 2021, 5:54 PM

Okay, that should take care of the function-evaluator. But Z777 still doesn’t work, and I’m realizing now that the function-evaluator was probably never the problem to begin with? After all, this function call probably doesn’t require evaluating any actual code.

Screenshot 2021-11-24 at 18-52-28 No label defined in this language or accepted fallbacks Z777 Function call (Z7) - Not Wik[...].png (378×496 px, 24 KB)

The function-orchestrator logs have this error:

{"name":"function-orchestrator","hostname":"function-orchestrator-68f9857d8b-7gq6l","pid":87,"level":50,"message":"500: internal_error","stack":"FetchError: request to http://mediawiki-web:8080/w/api.php?action=wikilambda_fetch&format=json&zids=Z7%7CZ9 failed, reason: getaddrinfo ENOTFOUND mediawiki-web\n    at ClientRequest.<anonymous> (/srv/function-orchestrator/node_modules/node-fetch/lib/index.js:1461:11)\n    at ClientRequest.emit (events.js:314:20)\n    at Socket.socketErrorListener (_http_client.js:427:9)\n    at Socket.emit (events.js:314:20)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)","status":500,"type":"internal_error","detail":"request to http://mediawiki-web:8080/w/api.php?action=wikilambda_fetch&format=json&zids=Z7%7CZ9 failed, reason: getaddrinfo ENOTFOUND mediawiki-web","request_id":"f6c387c0-4d4e-11ec-8221-9d942a513f33","request":{"url":"/1/v1/evaluate/","headers":{"user-agent":"GuzzleHttp/7","content-type":"application/json","content-length":"242","x-request-id":"f6c387c0-4d4e-11ec-8221-9d942a513f33"},"method":"POST","params":{"0":"/1/v1/evaluate/"},"query":{},"remoteAddress":"[redacted]","remotePort":"[redacted]"},"levelPath":"error/500","msg":"500: internal_error","time":"2021-11-24T17:50:08.564Z","v":0}

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

LucasWerkmeister added a comment.Nov 24 2021, 5:59 PM
In T296421#7527680, @LucasWerkmeister wrote:

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

ahem

Stashbot added a comment.Nov 24 2021, 6:01 PM

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T18:01:11Z] <wm-bot> <lucaswerkmeister> renamed $wgWikiLocation to $wgWikiLambdaWikiAPILocation in local settings T296421, and also changed it to internal hostname (notwikilambda:8000), no need to go through the Toolforge proxy

LucasWerkmeister closed this task as Resolved.Nov 24 2021, 6:01 PM

Works now as far as I can tell.

Jdforrester-WMF added a comment.Nov 24 2021, 6:20 PM
In T296421#7527692, @LucasWerkmeister wrote:
In T296421#7527680, @LucasWerkmeister wrote:

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

ahem

Oh oops, in my head you'd not over-ridden the default value so didn't need anything changed. Sorry!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits