Page MenuHomePhabricator

Logging config is causing errors in production
Closed, ResolvedPublic

Description

After this debug patch [1] it looks like the error config is raising errors in production for the aggregate endpoints:

TypeError: Illegal invocation
    at handleWriteReq (node:internal/stream_base_commons:61:21)
    at writeGeneric (node:internal/stream_base_commons:149:15)
    at Socket._writeGeneric (node:net:874:11)
    at Socket._write (node:net:886:8)
    at writeOrBuffer (node:internal/streams/writable:391:12)
    at _write (node:internal/streams/writable:332:10)
    at Socket.Writable.write (node:internal/streams/writable:336:10)
    at NamedLevelStdout._write (/srv/service/node_modules/service-runner/lib/logger.js:23:25)
    at writeOrBuffer (node:internal/streams/writable:391:12)
    at _write (node:internal/streams/writable:332:10)
    at NamedLevelStdout.Writable.write (node:internal/streams/writable:336:10)
    at Logger._emit (/srv/service/node_modules/bunyan/lib/bunyan.js:923:22)
    at Logger.warn (/srv/service/node_modules/bunyan/lib/bunyan.js:1045:24)
    at Logger._log (/srv/service/node_modules/service-runner/lib/logger.js:331:26)
    at Logger.log (/srv/service/node_modules/service-runner/lib/logger.js:365:18)
    at /srv/service/lib/util.js:461:36

It is reproducible locally after this config diff:

 # Logger info                                                                                                                                   │
 logging:                                                                                                                                        │
-  level: trace                                                                                                                                  │
+  level: warn                                                                                                                                   │
   streams:                                                                                                                                      │
     - type: stdout # log to stdout                                                                                                              │
       named_levels: true # emit log level name instead of index. e.g. INFO vs 30                                                                │

This must be related with the service-runner upgrade and specifically this service runner commit:
https://github.com/wikimedia/service-runner/commit/d3fc15b95b23cba792a01a8fcdf41c5a36786a05

[1] https://gerrit.wikimedia.org/r/c/mediawiki/services/wikifeeds/+/933111

Details

Event Timeline

This error is only caused with logging.streams.type = stdout

Change 933861 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Fix logging init for cloned requests

https://gerrit.wikimedia.org/r/933861

Change 933861 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Fix logging init for cloned requests

https://gerrit.wikimedia.org/r/933861