We need something like an etcd cluster (or similar alternative) controlling the active cache lists at the various layers (pybal/LVS -> varnish, varnish -> varnish). Ideally the puppet nodelists would populate the basics nodelists in etcd (currently in hieradata/common/cache/*.yaml), and we'd have the data structure set up so that nodes can be runtime-depooled via etcd updates at runtime independently of that, and then wrap some tooling around it to easily depool a given cache node in both the frontend and backend senses globally. From there it would become much easier to script up daemon/host restarts with automatic depooling, and we could also have nodes self-(de|re)pool around clean reboots on their own as well via initscripts->etcd.
In code terms, the pybal integration could be direct, while varnish would probably have data-updated-triggered regeneration of a VCL fragment + reload-vcl.