I have upgraded Jenkins which loaded just fine. Then updated a bunch of plugins including ArtifactDeployer to 0.31. That version breaks lazy loading and cause the whole history to be loaded on Jenkins initialization.
Filled as https://issues.jenkins-ci.org/browse/JENKINS-24140
Workaround: I have manually downgraded the plugin to previous version.