Around the time that I did a trial deployment to ores1001, that machine became sad and memory use climbed to the ceiling repeatedly. This is a dedicated box, so we're almost certainly to blame.
Check the exact timing of this issue relative to my deployment and rollback, and see if any diagnostics are still available.
- 2018-05-09 22:02 memory nosedives
- 2018-05-09 22:45 memory bottoms out and begins to sawtooth from 17GB to 0GB
- 2018-05-10 01:49-ish, memory pressure seems to ease although sawtooth still goes deeper than usual, down to 6GB.
- 2018-05-10 20:46 hits bottom again
- 2018-05-11 00:45 hits bottom
- 2018-05-11 01:33 seems to rebound to normal levels, although more bottoming-out occurs
Other servers show similar deep spikes, but not the severe memory pressure of ores1001 on 2018-05-09.
ores1001 deployment timeline:
- 2018-05-01 21:48 git-lfs test (rORESDEPLOY4601497c4f43)
- 2018-05-01 21:54 git-lfs test (52347e0)
- 2018-05-01 22:01 rolled back (bf182e2)
- 2018-05-09 22:05 git-lfs test 1 (c0db102)
- 2018-05-09 22:09 git-lfs test 2 (c0db102)
- 2018-05-09 22:20 git-lfs test 3 (2a09939)
- 2018-05-09 23:06 drafttopic trial deployed (bf1e2b1)
- 2018-05-09 23:24 rolled back (bf182e2)