We recently saw problems in production (see T129237) because a patch (https://gerrit.wikimedia.org/r/#/c/274638/) created unanticipated server load.

This is preventable. We should be load-testing changes in the service before deploying them to production. One option is to use appservice.wmflabs.org for this (unless there's some reason we shouldn't be doing this on a labs machine).

Some testing options for exploration:
https://jmeter.apache.org/
https://loader.io/
http://gatling.io/
https://github.com/alexfernandez/loadtest

See also:
https://strongloop.com/strongblog/monitoring-stress-testing-nodejs-apps/