I’m having the same problem. The issue seems to be with the surface area Renoise has to render to: The bigger, the worse the performance. I get the same poor performance in a maximized window as I get if it is full-screen.
I don’t know how Renoise’s UI is rendered, but if it is not using the GPU, that may be the source of the problems as rendering on a 5K display as the iMac 27" I have with a CPU is going to be extremely costly. And it’s not like my iMac is a slouch:
What’s weird, though, is that disabling HiDPI mode doesn’t improve the performance (at least not observably). I get the same poor performance regardless of HiDPI mode, it’s only the size of Renoise’s window that affects the performance afaict.