UW checks whether mw.Title.newFromText() is successful, and that function just returns false whether the title is too long or has invalid characters or whatever else, so I would say this has to be solved in core. (Not sure how though - maybe by mw.Title providing a validator that uses the same logic but returns error codes?)

Could possibly be due to length checking being with file extension in Special:Upload but without it in UW.

Or (UTF-16) characters vs. bytes? Haven't looked at the code, just guessing.

So maybe the logic would be:
Use newFromTitle() to check the title, if newFromTitle() returns false, use a validator function on the title to check what error it was? newFromTitle() could use the validator to check the title internally so we wouldn't have two copies of the "valid title" checks.