As shown in the proof-of-concept patch https://gerrit.wikimedia.org/r/357796 in order to allow extensions to register custom data types, extensions should have a way to add custom "experts" for those data types, so UI widgets would accept values of the extension-provided data type.
It seems "core" Wikibase front-end code should expose an extension point for extensions. This would possibly be implemented using front-end hooks.