Some examples which use popup for social login: Uber, Reddit, Twitter, TikTok, LinkedIn, Pinterest.

FWIW Google's OAuth client library (which supports popup and top-level redirect mode) says "Key Point: For the best balance of usability and security using the Popup mode UX flow with Authorization code model is recommended."
They don't elaborate so I'm not sure whether they consider the redirect mode less usable or less secure. (I guess at a minumum it would be less usable from the developer's POV, since you need to implement a redirection target in your web app, while the popup can be handled entirely via the client library.)

We probably won't use the popup approach in the end, so I will cut this investigation short, but I wanted to note what I have already learned. I had a closer look at https://archive.org/donate and the two popup auth providers available there. Observations:

PayPal will fall back to an iframe interface when the browser blocks popups completely. It's interesting that they're not worried about the security of it (maybe it's rare enough to block cookies that it's not worth worrying about, but then it's interesting that they bothered implementing a fallback). Google will not, instead it fall backs to a top-level navigation.

Google Pay prefers using the Payment Request API, which is not supported by Firefox, and results in a different kind of popup (it's a separate browser window, but it's "attached" to the parent window like an alert() modal dialog). This is neat, but this API is not an option for us.

I really liked the design of PayPal's backdrop in case the user loses the popup window. It shows a clear modal overlay that provides a fallback experience (…although it didn't work for me), and is designed with "soft" borders so that they don't have to worry about aligning the popup closely with it.

Google Pay just dims the whole page with a modal overlay, and brings the popup window to the front when the overlay is clicked. This doesn't seem that intuitive to me.

Lastly, both PayPal and Google Pay are able to detect when the user closes the popup window, and exit out of the modal backdrop on the parent window. We should definitely add this feature if we ever come back to this.

For the other websites, I took a few screenshots for reference. Some of these integrations aren't very good looking.

Most of them also neglect to close the login popup when the user leaves the login page without logging in, which was quite annoying for me when testing, but it's also not a good user experience.

Website ↓	Auth provider →	Google	Apple	Other
https://archive.org/donate			–	PayPal:
https://www.reddit.com/				–
https://uber.com/				–
https://twitter.com/				–
https://tiktok.com/				Facebook: doesn't work. Twitter:
https://linkedin.com/				–
https://pinterest.com/			–	Facebook:

I also did a bit of competitive analysis as a part of our design decision brief.

I'm going to expand on this just a bit further and then I'll add my examples inline instead of Google Slides for reference here as well!