Since we're filtering the history, the user doesn't actually get the number they asked for (50, 100, etc.). This isn't the end of the world; if they click through the 'next' links they will see everything they're intended to see, but it's still wrong.
This can be fixed by letting HistoryPager do the re-querying as needed.
See also T108289: Contributions apparently could have wrong pagination, which should probably be fixed similarly but in ContributionsQuery (since ContribsPager is in core).