As it will be embedded within MediaWiki, chances are we can just let the existing MediaWiki SessionManager code handle it.

I imagine we'll still call Setup.php and that takes care of authentication. OTOH there should probably be a way for API endpoints to declare themselves sessionless.

If by "API endpoints" you mean the REST endpoint as a whole before REST routing, that's possible as is done for load.php and a few others already.

If you mean the individual routed endpoints, that seems like it could be tricky as you'd need to somehow get the endpoint configuration before calling Setup.php, which in turn would make it hard to use MediaWiki's configuration and extension-loading mechanism to define the endpoints.

Individual endpoints, yeah. Session setup takes a nontrivial amount of time and most GET endpoints won't need it (the action API tends to mix public and private data, e.g. show redacted usernames if you have the permission to see them; in the REST API we probably don't want that as it makes responses uncacheable). I don't think it's hard, just an an extension point in Setup.php (pretty much everything else is loaded by that point), but something to think about.

A somewhat related issue is that currently any request with a session cookie on it bypasses Varnish; we won't want that, since it kind of defeats the point of using REST, OTOH we also don't want to leak private information every time some API module forgets to set a Vary header, so what's the right level of handling that?

This probably makes more sense as a subtask of T221737: REST API Infrastructure in MediaWiki now.