I fixed it in EditNoticesOnMobile. If there's anything in that hack that might inspire a patch, copy-paste whatever you can use. It's WTFPL after all.

Edit: it should actually be simpler to fix. If a symbolic size was set and it's not "full", never set width/height to 100% but set them to 85%. And don't remove the classes for a windowed popup or add those for a fullscreen popup.

My best attempt at a workaround:

// Subclass of OO.ui.MessageDialog that never goes into fullscreen mode
function NoFullscreenMessageDialog( config ) {
	OO.ui.MessageDialog.call( this, config );
}
OO.inheritClass( NoFullscreenMessageDialog, OO.ui.MessageDialog );
// Override to never return 'full'
NoFullscreenMessageDialog.prototype.getSize = function () {
	return 'large';
};

OO.ui.getWindowManager().addWindows( { 'noFullscreenMessage': new NoFullscreenMessageDialog() } );

// Copy of OO.ui.confirm() using the customized dialog
// (you could also call openWindow() directly in your code if you find that clear enough)
function noFullscreenConfirm( text, options ) {
	return OO.ui.getWindowManager().openWindow( 'noFullscreenMessage', $.extend( {
		message: text
	}, options ) ).closed.then( function ( data ) {
		return !!( data && data.action === 'accept' );
	} );
};

noFullscreenConfirm( 'test' );

I'm not sure if making this change in OOUI would be desirable (I really don't have an opinion), but it would affect all interfaces, which might rely on the current behavior, so I'm not going to work on it without a solid reason.

Obviously nothing actually relies on that as this only affects windows which have a symbolic size specified, so any such window is already known to become windowed on a large enough screen.

Even without changing the default behavior, an option could be added to prevent windows from becoming fullscreen.

The hack above relies on max-width being set, which OO.ui currently does but may not be guaranteed in the future. It also barely makes the window a window: there's only 1em of space at the top and bottom and the width remains 100%.