Suppose this only happens in very specific cases. It would be good to indicate in detail what steps you take to find the fault.
I know there are certain states that are not registered within the undo / redo. But some seem to be on purpose. For example, I think I remember that silencing or not a track is not registered. Maybe there are more situations.
However, there may be particular cases without a good registry, and could be considered a bug.
I’m also creating own VSTs with Synthedit. Maybe it is an older 32 bit plugin created with an older Synthedit build? Because the older builds were very limited in functionality and had a lot bugs and also the exported VSTs were full of bugs. My own Synthedit plugins made with latest builds are all working fine with the undo in Renoise.
BTW: did you try to start Renoise as admin? this can also solve several problems with Synthedit VSTs. Because they use external files (.sem) which are packed inside the dll and will be automatically unpacked to a separate folder into the location where the dll is placed. They will be unpacked when you load the plugin the first time inside a DAW. Starting Renoise as admin gives the plugin all permissions it needs to write these files to disk without blocking something. VSTs made with latest builds don’t unpack these files anymore. Here you get a complete folder which already contains the dll, all needed graphical image files and also the needed sem files. Here you have to copy the whole folder to your VST plugins folder and not just the dll.