Fix display of active toggles when hovering over them

Since we change the display text color to be white,
the button should not be white

We should eventually revisit the display and the color,
for both display modes.

Bug: T274616
Change-Id: Iec6510c4870ce9df87dabf872ed22f563f8376d9