The “piano part” of Renoise is hardwired. But since this suggestion is about customization, I’d suggest that a tool that captures keyboard input while it’s dialog is focused.
This way, you still would have access to the full Renoise keyboard and it would allow you to do “vim-style” editing where key sequences trigger certain actions.
Would be very powerful if combined with a suggestion like this one (blast from the past!):