Currently using varnish without Swift gives odd behavior when it comes to purging. Some thumbnail sizes are purged (need to find out how) and others are entirely missed because Swift isn't there to keep track of their existence. I presume whatever mechanism is finding some of them doesn't work with thumbor at all anyway.
We simply need a persistent data store to keep track of requested thumbnail sizes, for purging purposes.