Currently you have to drag the yellow/blue triangles, as the space above and below is clickable, however it needn't be: Clicking on a revision that is already selected is either a no-op, or sets the markers to the same place which would show the user a blank diff.
Making the whole of the selected column draggable would also increase the target area, making the tool easier to use.
It should also never be possible to set the diff markers to the same revision as an empty diff is useless. I thought I filed a task about that, but I can't find it.
@Esanders thanks for all your awesome reviews, comments and suggestions for the revision slider! I just talked to @Jan_Dittrich, our UX person, about your suggestion. We completely agree that the target area for the arrows is too small right now. However, making the whole column draggable would lead to some design inconsistencies. For example, it would not be possible to see the tooltip of the revision anymore. Granted, most of the information is below in the diff header, but not all. Therefore @Jan_Dittrich will come up with a suggestion how to increase the target area in another manner.
There was also a task about prohibiting empty diffs: T141254 I closed this, as logically, the tool is behaving correctly.
This is the ticket for making it easier to hit the arrows: T143091: making revisionslider pointer easier to hit by mouse or finger
Why are the two mutually exclusive? You can make the whole bar draggable and still show the tooltip on hover.