It's possible to serve WebP conditionally with some rewrite rules or reverse proxy, use 404 transform with media handlers overridden:

if ( MW_ENTRY_POINT === 'thumb_handler' ) {
	global $wgMediaHandlers;
	$wgMediaHandlers['image/webp'] = WebPThumb\WebPHandler::class;
	$wgMediaHandlers['image/png'] =  WebPThumb\PNGHandler::class;
}

(I created a PoC extension locally with the onMediaWikiServices hook, but LocalSettings.php etc. may also work.)

namespace WebPThumb;

class WebPHandler extends \WebPHandler {

	public function getThumbType( $ext, $mime, $params = null ) {
		return [ 'webp', 'image/webp' ];
	}
}

For incoming requests, strip .png postfix for webp files, add .webp postfix for png files, then pass the request to the thumb_handler.php endpoint.

The only issue is that the Content-Type header of the response is application/x-wiki.

This is because $wgTrivialMimeDetection is set to true by ThumbnailEntryPoint and the StreamFile::contentTypeFromPath() function doesn't consider webp as a valid extension in this case.

The thumb_handler.php endpoint should be able to stream webp with the correct Content-Type header image/webp.