I've analysed this file, it seems simply librsvg not support style="font-size: smaller" and font-size="smaller".

Looking at https://bugzilla.gnome.org/page.cgi?id=browse.html&product=librsvg I could not find a dedicated ticket - if this is an upstream issue, do you plan to upstream it?

If I get this correctly, it's not implemented in upstream, see:
https://git.gnome.org/browse/librsvg/tree/rsvg-css.c#n152
https://git.gnome.org/browse/librsvg/tree/rsvg-css.c#n200
https://git.gnome.org/browse/librsvg/tree/rsvg-css.c#n259

In T88833#1238198, @Aklapper wrote:

Looking at https://bugzilla.gnome.org/page.cgi?id=browse.html&product=librsvg I could not find a dedicated ticket - if this is an upstream issue, do you plan to upstream it?

Thats it, thanks for taking a look in the code (so I change the task desc.). And yes I'll do it. (It would be even less noticeable when the text would not entirely disappear.)

For anyone trying to reproduce this; the SVG linked in the task description has been changed to no longer use font-size="smaller", so it renders fine, but the underlying bug in librsvg is not addressed yet.

This got fixed in upstream in librsvg 2.42.3.

https://gitlab.gnome.org/GNOME/librsvg/-/issues/108
Confirming fixed in 2.44.10.