I want some buttons to trigger instruments on a specific track […] and some to be mapped to effects
Yes, that would indeedbe a nice way to use this tool.
Wondering about the note → effect mappings. How are you doing this? Because, one thing I don’t really understand about Renoise is why we can’t MIDI-map individual notes to parameters.
In the Renoise MIDI mapping dialog, you can assign a note but then you have to control the parameter using the notevelocity. In practical terms, when playing something by hand this is really hard to do (and some pad controllers doesn’t even transmit velocity). I guess xRules is really handy here!
As for the multiple instance setup, it really depends on the audio/midi drivers that are installed on your system. Here, I can open multiple instances and they can all receive, send MIDI just fine. Lucky me 
But apart from the device-driver issue, running multiple instances is actually a challenge to do right. Most often, you’d want each copy of Renoise to run with a slightly different configuration - see this topic for a more in-depth description of how that could be done.