Currently callback URLs for OAuth apps are parsed by UrlUtils which means only protocols present in $wgUrlProtocols are allowed. Which means that to allow an app using a callback URL with a custom protocol (common for mobile apps), we 1) need to make a configuration change, 2) that will also allow it to be used in links in wikitext, which is probably unwanted.
We should find a way to parse URLs without UrlUtils, or make UrlUtils ignore $wgUrlProtocols; and we should emphasize during review that the app is using a custom protocol (so OAuth admins can ensure that the protocol legitimately belongs to the app). That probably also means never doing auto-approval when a custom protocol is used.