Page MenuHomePhabricator

Function calls on Not Wikilambda produce Internal Server Error
Closed, ResolvedPublic

Description

AFAICS all function calls on NotWikilambda are erroring right now with an 'Internal Server Error', e.g. https://notwikilambda.toolforge.org/wiki/Z777 and manually-created calls even to Echo or If.

Event Timeline

It’s this “kad” nonsense again (I vaguely remember issues with it, they went away at some point):

kubectl logs pod/function-evaluator-6dc9fddbf5-jtjsg
> service-runner

internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module 'kad'
Require stack:
- /srv/function-evaluator/node_modules/limitation/lib/kad_backend.js
- /srv/function-evaluator/node_modules/limitation/index.js
- /srv/function-evaluator/node_modules/service-runner/lib/ratelimiter.js
- /srv/function-evaluator/node_modules/service-runner/lib/worker.js
- /srv/function-evaluator/node_modules/service-runner/lib/master.js
- /srv/function-evaluator/node_modules/service-runner/service-runner.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/srv/function-evaluator/node_modules/limitation/lib/kad_backend.js:4:26)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/srv/function-evaluator/node_modules/limitation/index.js:18:18)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/srv/function-evaluator/node_modules/limitation/lib/kad_backend.js',
    '/srv/function-evaluator/node_modules/limitation/index.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/ratelimiter.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/worker.js',
    '/srv/function-evaluator/node_modules/service-runner/lib/master.js',
    '/srv/function-evaluator/node_modules/service-runner/service-runner.js'
  ]
}
npm ERR! code 1
npm ERR! path /srv/function-evaluator
npm ERR! command failed
npm ERR! command sh -c service-runner

Hm, in the function-evaluator package-lock, limitation has a dependency on wikimedia-kad-fork, but in node_modules/limitation/package.json it’s still a dependency on kad (albeit on the version wikimedia/kad#master), and at runtime, its lib/kad_backend.js still requires kad, so it’s not going to find whatever this wikimedia-kad-fork is.

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:39:31Z] <wm-bot> <lucaswerkmeister> deleted function-evaluator deployment T296421

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:47:27Z] <wm-bot> <lucaswerkmeister> ran npm ci in function-evaluator (in webservice node12 shell), for some reason node_modules/limitation was on 0.2.2 despite package-lock.json having 0.2.3 (T296421)

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T17:50:03Z] <wm-bot> <lucaswerkmeister> recreated function-evaluator deployment (still on commit 8731f2ff6e, no changes) T296421

Okay, that should take care of the function-evaluator. But Z777 still doesn’t work, and I’m realizing now that the function-evaluator was probably never the problem to begin with? After all, this function call probably doesn’t require evaluating any actual code.

Screenshot 2021-11-24 at 18-52-28 No label defined in this language or accepted fallbacks Z777 Function call (Z7) - Not Wik[...].png (378×496 px, 24 KB)

The function-orchestrator logs have this error:

{"name":"function-orchestrator","hostname":"function-orchestrator-68f9857d8b-7gq6l","pid":87,"level":50,"message":"500: internal_error","stack":"FetchError: request to http://mediawiki-web:8080/w/api.php?action=wikilambda_fetch&format=json&zids=Z7%7CZ9 failed, reason: getaddrinfo ENOTFOUND mediawiki-web\n    at ClientRequest.<anonymous> (/srv/function-orchestrator/node_modules/node-fetch/lib/index.js:1461:11)\n    at ClientRequest.emit (events.js:314:20)\n    at Socket.socketErrorListener (_http_client.js:427:9)\n    at Socket.emit (events.js:314:20)\n    at emitErrorNT (internal/streams/destroy.js:92:8)\n    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)","status":500,"type":"internal_error","detail":"request to http://mediawiki-web:8080/w/api.php?action=wikilambda_fetch&format=json&zids=Z7%7CZ9 failed, reason: getaddrinfo ENOTFOUND mediawiki-web","request_id":"f6c387c0-4d4e-11ec-8221-9d942a513f33","request":{"url":"/1/v1/evaluate/","headers":{"user-agent":"GuzzleHttp/7","content-type":"application/json","content-length":"242","x-request-id":"f6c387c0-4d4e-11ec-8221-9d942a513f33"},"method":"POST","params":{"0":"/1/v1/evaluate/"},"query":{},"remoteAddress":"[redacted]","remotePort":"[redacted]"},"levelPath":"error/500","msg":"500: internal_error","time":"2021-11-24T17:50:08.564Z","v":0}

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

ahem

Mentioned in SAL (#wikimedia-cloud) [2021-11-24T18:01:11Z] <wm-bot> <lucaswerkmeister> renamed $wgWikiLocation to $wgWikiLambdaWikiAPILocation in local settings T296421, and also changed it to internal hostname (notwikilambda:8000), no need to go through the Toolforge proxy

Works now as far as I can tell.

I assume mediawiki-web:8080 is a default host name that we are failing to configure somewhere.

ahem

Oh oops, in my head you'd not over-ridden the default value so didn't need anything changed. Sorry!