A request took down both zotero and citoid, probably by taking up too much memory:
https://grafana.wikimedia.org/d/000000620/xxxx-zotero-debugging-kubernetes?orgId=1&from=1579716341745&to=1579717330571&fullscreen&panelId=41
https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue%3A0)%2Ctime%3A(from%3A'2020-01-22T18%3A10%3A06.199Z'%2Cmode%3Aabsolute%2Cto%3A'2020-01-22T18%3A20%3A06.201Z')) for the time,
https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=h@4d6d298&_a=h@806e79b for the search (type:citoid) - the "share" function doesn't seem to work correctly.
Sample citoid crash:
```
{
"_index": "logstash-2020.01.22",
"_type": "citoid",
"_id": "AW_Og9d1KWrIH1QReyyj",
"_version": 1,
"_score": null,
"_source": {
"source_host": "10.192.64.28",
"err_levelPath": "fatal/service-runner/unhandled",
"level": "EMERGENCY",
"line": "22",
"pid": 16,
"type": "citoid",
"message": "Maximum call stack size exceeded",
"version": "1.0",
"normalized_message": "Maximum call stack size exceeded",
"tags": [
"input-gelf-12201",
"deprecated-input",
"es",
"gelf",
"normalized_message_untrimmed"
],
"err_message": "",
"file": "/srv/service/node_modules/domutils/lib/traversal.js",
"@timestamp": "2020-01-22T18:29:02.551Z",
"err_name": "RangeError",
"host": "citoid-production-685469f44d-nnmn8",
"@version": "1",
"gelf_level": "0",
"err_stack": "RangeError: Maximum call stack size exceeded\n at Object.exports.getName (/srv/service/node_modules/domutils/lib/traversal.js:22:27)\n at tag (/srv/service/node_modules/css-select/lib/general.js:21:11)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:87:6)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)\n at findAll (/srv/service/node_modules/domutils/lib/querying.js:90:27)"
},
"fields": {
"@timestamp": [
1579717742551
]
},
"sort": [
1579717742551
]
}
```
We don't actually know what the request was yet as the kibana logs don't display it (it should be in the raw citoid logs.)