I can't even select as much as you.
When I draw a selection with the mouse past a reference or a "reference needed", the current selection collapses and starts as a new selection after the reference.
If I use the keyboard with "cursor right", the same happens.
Only if I use the keyboard with "cursor down", I have a chance to select something including references.

This issue appears to be selecting across block images. Bisect blames this commit from 4 years ago: https://gerrit.wikimedia.org/r/#/c/VisualEditor/VisualEditor/+/201357/.

Removing the following CSS seems to fix the issue, though it will probably break something else:

.ve-ce-focusableNode * {
 -moz-user-select: none;
}

This doesn't fix it for me, testing on the core demo page "Image".

This works when selecting across a single block image, but not when selecting across a paragraph slug.

Also discovered this: T217047

It appears that the moz-user-select-none hack (T70537) causes Firefox to generate multi-range selections, which we explicitly try to avoid:
https://github.com/wikimedia/VisualEditor/blob/wmf/1.33.0-wmf.18/src/ce/ve.ce.Surface.js#L3205
Supporting these would be a nightmare, so we probably have to avoid the user-select-not hack.

Change 492698 had a related patch set uploaded (by Esanders; owner: Esanders):
[VisualEditor/VisualEditor@master] Remove -moz-user-select:none from hack and buttonElement

https://gerrit.wikimedia.org/r/492698

Change 492698 merged by jenkins-bot:
[VisualEditor/VisualEditor@master] Remove -moz-user-select:none from hack and buttonElement

https://gerrit.wikimedia.org/r/492698

Change 493507 had a related patch set uploaded (by Esanders; owner: Esanders):
[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (226ff19e5)

https://gerrit.wikimedia.org/r/493507

Change 493507 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (226ff19e5)

https://gerrit.wikimedia.org/r/493507