The two paths to fix this I can think of are:

1. Getting the list of gallery modes from PHP, rather than hard-coding it.
2. Adding a JS hook at extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWGalleryDialog.js (here) for adding extra gallery modes.

The latter feels much simpler to implement.

Getting the list of gallery modes from PHP seems easy enough too. We already do that with the default options: https://codesearch.wmcloud.org/search/?q=GalleryOptions&i=fosho&files=&excludeFiles=&repos=Extension:VisualEditor. You could just run the hook there and ship the result in the same way.

The only tricky piece would be providing the localisation messages (to display the name of each mode in the dropdown when editing a gallery). I would try doing it this, to dynamically define 'messages' in the ResourceLoader module: https://codesearch.wmcloud.org/search/?q=addOptionalDependencies&i=nope&files=&excludeFiles=&repos=Extension:DiscussionTools.

On the other hand… We also have some small special cases hard-coded for several of the core gallery modes: https://codesearch.wmcloud.org/search/?q=mode%20%3D%3D%3D&i=fosho&files=.*Gallery.*&excludeFiles=&repos=Extension:VisualEditor. I don't know how important they are, and whether custom modes would work perfectly without those. Perhaps this is a reason to go with a JS hook (or hooks).

If you're interested in writing patches for this (using either approach), I'd be happy to review them.