As documented in T3310, MW does not allow extension tags with the same tag name to be nested. An extension tag is treated as containing unstructured plain text up to the next terminating tag. This was a deliberate design choice due to the fact that many extensions (math, timeline, score, etc.) do not use XML-style markup and have no concept of nesting. The bug has been open since 2005 with mixed feelings expressed by several participants on whether it should be fixed.
However, Parsoid allows nesting of extension tags: <ref><ref></ref></ref> etc. So, that is a compatibility break.
Nesting should be allowed either for both parsers or for neither. I'm not aware of any technical reason as to why they should be different, it's just that the Parsoid implementors had a different opinion on T3310 than the core maintainers.