Oooh no, Audiounit preset data and VST preset data isof course not compatible. You cannot change it to VST spontaneously 
Not sure what you wanted to achieve here, can you explain to me? What you can do is the following:
-
Converting songs using Windows VSTs to a Mac working variant, only if you have the same plugin installed on both systems, with the same version, simply by renaming the PluginIdentifier node
-
You can convert a preset from the Audiounit version to the VST version of a plugin, if the plugin provides an own preset mechanism/browser, so saving there the preset and then load the vst version and load the preset there
-
Workaround the current AU bug, so your pre-10.13-on-macos-saved will load normally, until this bug will be fixed (maybe another topic, refer here)
You can always save the individual plugin’s format by opening the preset context menu on the plugin and select “export preset” or “import preset”. Then you will see that AU-plugins save in their own .aupreset format, while VST uses a preset structure called .fxp and for banks .fxb. In the renoise song this data will be xml bin encoded (I would prefer to have those fxp/aupreset data directly there), so it may look similar but it is not at all.