WikiFunctions function-orchestrator-main-orchestrator started logging response as a json object which occupies over 1300 fields in the legacy logstash indexes. This causes logs from other producers to be dropped due to exceeding the index limit of 2048 fields.
To mitigate, we have dropped the field in logstash.
T180051: Reduce the number of fields declared in elasticsearch by logstash
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| evaluate: Don't log the JSON object, it's too big for prod | repos/abstract-wiki/wikifunctions/function-orchestrator!56 | jforrester | T343176 | main |
jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/56
evaluate: Don't log the JSON object, it's too big for prod
kindrobot merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/56
evaluate: Don't log the JSON object, it's too big for prod
Change 944304 had a related patch set uploaded (by Jforrester; author: Jforrester):
[operations/deployment-charts@master] wikifunctions: Bump to image without stupendous output logging
Change 944304 merged by jenkins-bot:
[operations/deployment-charts@master] wikifunctions: Bump to image without stupendous output logging
OK, this is deployed; sorry for the noise! You should be able to re-enable us (but please shout if it didn't work or there are other problems).
Change 944194 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] Revert "logstash remove wikifunctions response field"
Change 944194 merged by Cwhite:
[operations/puppet@production] Revert "logstash remove wikifunctions response field"
The mitigation is removed and the issue has not yet reappeared. Thanks!
Optimistically resolving for now. We'll continue to watch for issues and report back if we find a problem.