Are there other “Guitar Rig 5.dll” files in you plugin path? Instantiate FAILED (MagicMatch) basically means that this is not a VST plugin dll, but some other system dll file.
Try setting the VST path in Renoise to the folder where exactly the plugin is located to ensure that it’s the right one.
Have you tried clearing/deleting Renoise’s VST cache files (while Renoise is not open). See Renoise -> Help -> “Show Preferences Folder…” for the folder location.
If all this doesn’t help, please send us the full log file…