Page MenuHomePhabricator

Cxserver container: Container does not send fatal errors to docker logs via stdout?
Closed, InvalidPublic

Description

See parent task.

krenair@deployment-docker-cxserver01:~$ time sudo /usr/bin/docker run -p 8080:8080 -v /etc/mediawiki-services-cxserver/:/etc/mediawiki-services-cxserver --name alex-test docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-05-08-064536-production -c /etc/mediawiki-services-cxserver/config.yaml

real	0m13.426s
user	0m0.044s
sys	0m0.028s
krenair@deployment-docker-cxserver01:~$ cat /etc/mediawiki-services-cxserver/config.yaml
logging:
  level: warn
  name: cxserver
  streams:
  - host: deployment-logstash2.deployment-prep.eqiad.wmflabs
    port: 12201
    type: gelf
  - level: info
    path: "/etc/mediawiki-services-cxserver/main.log"
    type: file
metrics:
  host: labmon1001.eqiad.wmnet
  name: cxserver
  port: 8125
  type: statsd
num_workers: ncpu
ratelimiter:
  listen:
    address: 127.0.0.1
    port: 3051
  seeds:
  - 127.0.0.1
  type: kademlia
services:
- conf:
    cors: "*"
    dictionary:
      JsonDict:
        languages: src/config/JsonDict.yaml
    jwt:
      algorithms:
      - HS256
      secret: H.9F-.|,SC#Von%?Tklo"a<uPFOcxI$> (this is a dummy secret)
    languages: src/config/languages.yaml
    mt:
      Apertium:
        api: http://apertium-beta.wmflabs.org
        languages: src/config/Apertium.wikimedia.yaml
      Google:
        api: https://www.googleapis.com/language/translate/v2
        key: dummykey
        languages: src/config/Google.yaml
      LingoCloud:
        account: wikimedia
        api: https://api.interpreter.caiyunai.com/v1
        key: dummykey
        languages: src/config/LingoCloud.yaml
      Matxin:
        api: http://matxin.elhuyar.eus/API
        key: dummykey
        languages: src/config/Matxin.yaml
      TestClient:
        languages: src/config/TestClient.yaml
      Yandex:
        api: https://translate.yandex.net
        key: trnsl.1.1.20150216T113622Z.2ccb71d878ea0b58.5ac57322acecfa2d9001aad077fdcff21c206a43 (this is a dummy secret)
        languages: src/config/Yandex.yaml
      Youdao:
        api: https://openapi.youdao.com/api
        appKey: dummykey
        appSecret: dummysecret
        languages: src/config/Youdao.yaml
      defaults: src/config/mt-defaults.wikimedia.yaml
    mwapi_req:
      body: "{{request.body}}"
      headers:
        host: "{{request.params.domain}}"
        user-agent: "{{user-agent}}"
      method: post
      query: "{{ default(request.query, {}) }}"
      uri: http://deployment-mediawiki-07.deployment-prep.eqiad.wmflabs/w/api.php
    no_proxy_list:
    - wmflabs
    port: 8080
    proxy: 
    ratelimiter_key: X-Client-IP
    ratelimiter_rate: 10
    restbase_req:
      body: "{{request.body}}"
      headers: "{{request.headers}}"
      method: "{{request.method}}"
      query: "{{ default(request.query, {}) }}"
      uri: http://deployment-restbase02.deployment-prep.eqiad.wmflabs:7231/{{domain}}/v1/{+path}
    user_agent: CXServer/WMF
  module: src/app.js
  name: cxserver
worker_heap_limit_mb: 300
worker_heartbeat_timeout: 7500

Event Timeline

Krenair created this task.May 15 2019, 5:49 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 15 2019, 5:49 PM
Krenair renamed this task from Cxserver container: Container runs silently for 15 seconds and then stops to Cxserver container: Container runs silently for 13-14 seconds and then stops.EditedMay 15 2019, 5:52 PM

Looks like after I fiddle with the permissions it does log errors to /etc/mediawiki-services-cxserver/main.log, but shouldn't something like this reach docker logs?

{"name":"cxserver","hostname":"a034b105e363","pid":47,"level":60,"err":{"message":"","name":"Error","stack":"Error: Cannot find module '/srv/service/node_modules/src/app.js'\n    at Function.Module._resolveFilename (module.js:469:15)\n    at Function.Module._load (module.js:417:25)\n    at Module.require (module.js:497:17)\n    at require (internal/module.js:20:19)\n    at Worker._requireModule (/srv/service/node_modules/service-runner/lib/base_service.js:363:30)\n    at Worker._requireModule (/srv/service/node_modules/service-runner/lib/base_service.js:383:29)\n    at Worker._requireModule (/srv/service/node_modules/service-runner/lib/base_service.js:383:29)\n    at P.map (/srv/service/node_modules/service-runner/lib/worker.js:171:25)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at MappingPromiseArray._promiseFulfilled (/srv/service/node_modules/bluebird/js/release/map.js:61:38)\n    at MappingPromiseArray.PromiseArray._iterate (/srv/service/node_modules/bluebird/js/release/promise_array.js:114:31)\n    at MappingPromiseArray.init (/srv/service/node_modules/bluebird/js/release/promise_array.js:78:10)\n    at MappingPromiseArray._asyncInit (/srv/service/node_modules/bluebird/js/release/map.js:30:10)\n    at _drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:142:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:131:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:147:5)","code":"MODULE_NOT_FOUND","moduleName":"src/app.js","levelPath":"fatal/service-runner/worker"},"msg":"Cannot find module '/srv/service/node_modules/src/app.js'","time":"2019-05-15T17:52:14.865Z","v":0}
{"name":"cxserver","hostname":"a034b105e363","pid":1,"level":60,"message":"startup failed, exiting master","worker_pid":47,"exit_code":1,"levelPath":"fatal/service-runner/master","msg":"startup failed, exiting master","time":"2019-05-15T17:52:15.882Z","v":0}

Edit: With some config fixes relating to filesystem paths I got it to run, will make this task about the logging

Krenair renamed this task from Cxserver container: Container runs silently for 13-14 seconds and then stops to Cxserver container: Container does not send fatal errors to docker logs via stdout?.May 15 2019, 6:04 PM
streams:
- host: deployment-logstash2.deployment-prep.eqiad.wmflabs
  port: 12201
  type: gelf
- level: info
  path: "/etc/mediawiki-services-cxserver/main.log"
  type: file

needs an extra stanza

- type: stdout

that should do it.

Krenair closed this task as Invalid.May 16 2019, 12:41 PM

... thanks :)

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptMay 16 2019, 12:41 PM

Doing s/module: src\/app.js/module: app.js/ should fix your issue.

I did module: ../app.js to solve that, was fine once I managed to get it writing error logs.