Well for anyone interested I finally gave up. While benchmarking apps showed a 15%-20% performance penalty (which sucks but for me is livable) some games I tried (like Grim Dawn) went from 60fps native to 15fps. The cause of both (the 20% and the gaming degradation) is apparently a bug with the Linux KVM Hypervisor and Ryzen.
The other Linux hypervisor (Xen) was supposed to get me much better performance (due to not having that bug) but the version that comes with Ubuntu doesn’t support Ryzen, and building the newest versions wasn’t working for me for whatever reasons.
Since the KVM bug is a software bug that can be fixed, I’ll probably try again in another 6-8 months (I have the hardware so why not) but for now it’s windows native with a Linux VM for me.