VST Instrument control via the macros? Seems officially unsupported.
There is a way around, though. Put an init doofer into a track. Place one instrument automation device into it. Copy the doofer into clipboard. Paste it into a instrument effects chain. Link doofer controls to parameters you want to map. Map doofer controls to macro dial.
Right now you’re supposed to map VST instruments on track level I think. So don’t whine in case in some future version of renoise this trick won’t work anymore, or if you encounter bugs or other breakage. I also find it interesting, that the inst automation device doesn’t have the instrument selector inside intrument effects, so it also could be official to use it in future versions?
For VST effects just click the little triangle-pointing-up button between the on/off checkbox and the ext editor button, the parameter should show up & be mappable to macros or midi.