Timo mentioned a cool idea being used in VisualEditor, if you've already scrolled down the page by the time crap at the top is ready to load, it prevents a page bump by calculating its own height and subtracting that as a negative scroll offset.
We could use this same method to prevent banner bump if the page has been scrolled.
Disclaimer: I've seen this not work in VE, so take their example code with a grain of salt.