The current name implies it behaves like a JS Set, i.e. that elements are unique. In fact we support an annotation being stored multiple times, and this is sometimes semantically valid (although not supported by our UI):

<p><big><big>Very big text</big></big></p>

[
  [ 'V', [ bigAnn, bigAnn ] ],
  [ 'e', [ bigAnn, bigAnn ] ],
  ...
]

When AnnotationSet was originally introduced (as a sub-class of OrderedHashSet) it did prevent you pushing the same has twice:
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/22162/3/modules/ve/ve.OrderedHashSet.js

At some point we dropped that restriction, although de-duplication does appear in other method still (e.g. addSet, and add under certain coniditions).

We either need to make uniqueness a properly enforced feature, in which case we need to consider how to handle duplicate annotations (e.g. alienated?), or support duplicate entries.