It is possible for us to query the local storage cache (mw.loader.store) to determine if VE has been loaded before by the client.
If it has then about 300k out of 500k of the modules (compressed sizes) will be in local storage. The remaining modules (ext.visualEditor.core & oojs-ui-widgets) will end up in the browser cache (after the second load).
If this is the case, then we can take about 200ms off VE load time by loading all these modules into memory in the background after the page has loaded.
If we think this is too aggressive, we could consider loading them when the 'edit' button is hovered, or some other heuristic (e.g. user has edited in the past 24 hours).