Would it be bad to make the src attribute optional in the templatestyles tag, such that if <templatestyles/> is put on page Template:Name, it attempts to load sanitized CSS from Template:Name/styles.css?
If that's not acceptable, would it be acceptable to default to the template's page name, such that <templatestyles src="/styles.css"/> would suffice?
That depends on how Parsoid handles extension tags. I think that instead of loading Template:Name/styles.css, it would try to load TranscludingPage/styles.css, or possibly Api/styles.css if Parsoid doesn't pass a title in. If that's indeed the case, it's probably not worth special-casing it in Parsoid when we can just make the src parameter required.
Parsoid relies on the php preprocessor for template expansion, so it wouldn't necessarily know which template a extension tag comes from when there's nesting.