Argggg, i hoped it could be that simple (it was the case for OpenMPT) !
Thanks vV, but yes the /3GB flag is living in my boot.ini since many years.
Ok, let’s go for some details then 
System : WinXP 32bit, 4BG installed ram (a little bit more than 3GB usable).
Renoise 2.7.2 (same behaviour with 2.6.1).
The song i’m currently working on uses Independance Pro vsti (in fact here it’s called Engine2, the free version of the engine distributed with sound libraries) and kontakt 4, with a total of 24 loaded instruments.
After the song is loaded, my ram goes from ~3100MB to ~1100MB (no problem here - about 2GB used, hence my suspicion about the /LARGEADDRESSAWARE).
I can work for a few time (random), and then get the kind of error in renoise5.jpg, among other errors (Failed to validate the document… the memory is getting low… failed to initialize midi actions… Ouch the vst plugin ‘XXX’ crashed in VST event/function ‘dispatch’. Thread GUI…).
I cannot load any additional instrument (see renoise10.jpg when i try to load a 5.5mb instrument) without vsti crash because of low memory.
I don’t know if either independance pro, kontakt or renoise are responsible, i will try to work on a smaller (memory wise) song !
Thanks in advance if anyone can help locate the real problem.
As a side note, i have also the same problems with all my security programs (Anti-Malware & Online Armor) disabled (in fact not disabled, not running at all).
It seems the log file cannot be attached to the post, so here is a session extract :
============================================================
Version : Renoise V2.7.2 (Jun 30 2011)
Date : 2011-09-20
Time : 13:09:16
OS : WinXP Service Pack 3
CPU: Found 1 enabled unit(s) with 2 core(s) / 2 logical processor(s) per unit. 2 cores are enabled in total.
Application: Showing the splash window…
Application: Initializing the API…
QuickTime: Initializing…
QuickTime: Successfully initialized…
QuickTime: Running QuickTime Version: 7418000
Graphport: Initializing Freeimage…
GraphPort: Initializing DirectX…
GraphPort: Initializing the Font Engine…
MIDI: Initializing DirectMusic…
MIDI: Enumerating DirectMusic (WDM) MIDI devices…
MIDI: Adding WDM in-device ‘Juli@ MIDI (WDM)’
MIDI: Adding WDM out-device ‘Juli@ MIDI (WDM)’
MIDI: Skipping softsynth WDM out-device ‘Juli@ Ch12’ (not supported)
MIDI: Skipping softsynth WDM out-device ‘Juli@ Ch34’ (not supported)
MIDI: Skipping softsynth WDM out-device ‘Juli@ Ch1234’ (not supported)
MIDI: Skipping softsynth WDM out-device ‘E-MU 0404 | USB’ (not supported)
MIDI: Skipping emulated WDM device ‘Mappeur MIDI Microsoft [Émulé]’
MIDI: Skipping emulated WDM device ‘PÚriphÚrique audio USB [Émulé]’
MIDI: Skipping emulated WDM device ‘Juli@ MIDI [Émulé]’
MIDI: Skipping emulated WDM device ‘SynthÚ. SW table de sons GS Mic [Émulé]’
MIDI: Skipping emulated WDM device ‘E-MU 0404 | USB [Émulé]’
MIDI: Skipping emulated WDM device ‘Juli@ MIDI [Émulé]’
MIDI: Skipping emulated WDM device ‘E-MU 0404 | USB [Émulé]’
MIDI: Skipping emulated WDM device ‘PÚriphÚrique audio USB [Émulé]’
MIDI: Skipping softsynth WDM out-device ‘Microsoft Synthesizer’ (not supported)
MIDI: Enumerating Windows MME MIDI devices…
MIDI: Adding MME in-device ‘Juli@ MIDI’
MIDI: Adding MME in-device ‘E-MU 0404 | USB’
MIDI: Adding MME in-device ‘Périphérique audio USB’
MIDI: Adding MME out-device ‘Périphérique audio USB’
MIDI: Adding MME out-device ‘Juli@ MIDI’
MIDI: Adding MME out-device ‘Synthé. SW table de sons GS Mic’
MIDI: Adding MME out-device ‘E-MU 0404 | USB’
MIDI: Finished device enumeration
ReWire Slave: ReWire engine link (Local Machine) is already installed and up to date…
ReWire Slave: ReWire engine link (Current User) is already installed and up to date…
ReWire Slave: ReWire Panel was initialized…
ReWire Slave: ReWire Slave mode failed to init (no master is running)…
ReWire Master: Successfully opened
DspDevices: Registering native DSP effects…
DspDevices: Registering native plugin DSP effects…
Application: Creating the main application…
Application: Start running…
Application: Loading the preferences…
Application: Init…
VSTPlugs: Searching for new VST plugins in ‘C:\Program Files\Steinberg’…
VSTPlugs: Searching for new VST plugins in ‘D:\Applications\Kontakt 4 player\VST plugin’…
DirectSound: Initializing…
DirectSound: Using SampleRate 48000
DirectSound: Primary Buffer Format: FormatTag=1, Channels=2, SamplesPerSec=48000, BitsPerSample=16, BlockAlign=4
DirectSound: CreateObjects … OK
DirectSound: Using MaxProcessingBlockSize: 11776
DirectSound: Output Buffer Format: BufferSize=96000, FormatTag=1, Channels=2, SamplesPerSec=48000, BitsPerSample=16, BlockAlign=4
DirectSound: CreateBuffer … OK
DirectSound: Up and running
MIDI: Opening MME Midi-In device ‘Périphérique audio USB’
Application: Creating a new document…
MIDI: Loading MIDI actions from file ‘D:\Applications\Renoise\Scripts\GlobalMidiActions.lua’…
Osc: Loading OSC actions from file ‘D:\Applications\Renoise\Scripts\GlobalOscActions.lua’…
Timer: Seems safe to use the ‘QueryPerformance’ counters…
Player: Constructing…
Player: Creating the slave threads…
Player: Start running…
GUI: Creating the Document GUI…
GUI: Successfully constructed
Application: A new document was created.
GraphPort: Initializing in WindowedMode
GraphPort: Successfully initialized
Application: Init OK
Application: Enter MainLoop…
Application: Releasing the old document…
MIDI: Loading MIDI actions from file ‘D:\Applications\Renoise\Scripts\GlobalMidiActions.lua’…
Osc: Loading OSC actions from file ‘D:\Applications\Renoise\Scripts\GlobalOscActions.lua’…
Player: Constructing…
Player: Creating the slave threads…
Player: Start running…
GUI: Creating the Document GUI…
GUI: Successfully constructed
Application: Loading ‘template2.xrns’…
MIDI: Loading MIDI actions from file ‘D:\Applications\Renoise\Scripts\GlobalMidiActions.lua’…
Osc: Loading OSC actions from file ‘D:\Applications\Renoise\Scripts\GlobalOscActions.lua’…
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\Engine 2.dll
VstPlugs: Analyzing VST plugin ‘Best Service: Engine 2’, Unique ID:1651729774, Version:2400
VstPlugs: Instantiate OK
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\Kontakt 4 16out.dll
VstPlugs: Analyzing VST plugin ‘Native Instruments GmbH: Kontakt 4 16out’, Unique ID:1315531574, Version:2400
VstPlugs: Forced static processing buffers (256 Samples) for plug ‘Native Instruments GmbH: Kontakt 4 16out’
VstPlugs: Disabled auto suspending for plug ‘Native Instruments GmbH: Kontakt 4 16out’
VstPlugs: Instantiate OK
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\Engine 2.dll
VstPlugs: Analyzing VST plugin ‘Best Service: Engine 2’, Unique ID:1651729774, Version:2400
VstPlugs: Instantiate OK
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\PSP_StereoEnhancer.dll
VstPlugs: Analyzing VST plugin ‘PSP: StereoEnhancer’, Unique ID:1349538661, Version:2200
VstPlugs: Instantiate OK
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\AIR.dll
VstPlugs: Plug is a SynthEdit plug (SE version: 111001)…
VstPlugs: Analyzing VST plugin ‘Yichi Wang: AIR’, Unique ID:1295016756, Version:2300
VstPlugs: Instantiate OK
VstPlugs: Trying to instantiate C:\Program Files\Steinberg\Vstplugins\Eqium 2ch.dll
VstPlugs: Analyzing VST plugin ‘Elemental Audio Systems: Eqium 2ch’, Unique ID:1916023601, Version:2
VstPlugs: Instantiate OK
Player: Constructing…
Player: Creating the slave threads…
Player: Start running…
GUI: Creating the Document GUI…
GUI: Successfully constructed
Application: Successfully loaded ‘template2.xrns’.
VstPlugs: Caught exception in VSTPlug::Dispatch
Warning Message: Ouch. The VST plugin ‘Native Instruments GmbH: Kontakt 4 16out (from Native Instruments GmbH)’ crashed in VST event/function: ‘Dispatch’, Thread: ‘GUI’
Warning Message: IMPORTANT: This is a fatal error. Please save the current document under a !new! name to prevent losing important data. Renoise will silently ignore any following errors from this plug-in.
Warning Message: DO NOT IGNORE THIS ERROR; DO NOT CONTINUE USING THE PLUGIN. THIS CAN DESTROY YOUR SONG OR CAUSE OTHER RANDOM CRASHES AND PROBLEMS!
Warning Message: It’s most likely that this is a bug in the VST plugin, so please report this problem to the authors of this plugin in order to get the problem fixed!
CrashLog: Handling Exception! Code : C0000005
CrashLog: 34FF77E3: ??? +00000
CrashLog: 34FF3051: ??? +00000
CrashLog: 35163675: ??? +00000
CrashLog: 355F8EA9: ??? +00000
CrashLog: 355F8F1A: ??? +00000
CrashLog: 7E398734: GetDC +0006D
CrashLog: 7E398816: GetDC +0014F
CrashLog: 7E3989CD: GetWindowLongW +00127
CrashLog: 7E398A10: DispatchMessageW +0000F
CrashLog: 00A51FD3: xmlUCSIsCat +3CEDF4
CrashLog: 00BC0BDC: xmlUCSIsCat +53D9FD
CrashLog: 00B5EC0E: xmlUCSIsCat +4DBA2F
CrashLog: 004AC4BE: ??? +00000
CrashLog: 00B50115: xmlUCSIsCat +4CCF36
CrashLog: 00B509C0: xmlUCSIsCat +4CD7E1
CrashLog: 00A379ED: xmlUCSIsCat +3B480E
CrashLog: 00A39CCC: xmlUCSIsCat +3B6AED
CrashLog: 00A4922F: xmlUCSIsCat +3C6050
CrashLog: 00A4D991: xmlUCSIsCat +3CA7B2
CrashLog: 00D08FDC: xmlUCSIsCat +685DFD
CrashLog: 00B5F11B: xmlUCSIsCat +4DBF3C
CrashLog: 00B5F1CB: xmlUCSIsCat +4DBFEC
CrashLog: 005E0004: ??? +00000
CrashLog: 7C817077: RegisterWaitForInputIdle +00049
Application: Caught an unhandled exception (Thread: GUI)!
Application: Saving a backup…
VstPlugs: Caught exception in VSTPlug::Dispatch
Error Message: A fatal error or crash occurred (unhandled exception in thread: GUI).
Error Message: A backup of the current document was saved at
Error Message: ‘C:\Documents and Settings\Eric\Application Data\Renoise\V2.7.2\CrashBackups\template2_9_20_2011__13_16_crash.xrns’…
Error Message: This either happened because of a bug in Renoise, or because of a bug in one of its loaded components (plugins). Please contact bug@renoise.com and report this problem, so that it can be fixed.
Error Message: Note: It’s very important that we know exactly what has happened (what you were doing before this message popped up), or the problem cannot be replicated/analyzed. Please include a description of what you were doing and which components were being used…
Application: Terminating…
DirectSound: Stop Polling…
DirectSound: Releasing Juli@ Ch12…
MIDI: Shut down: Closing all acquired MIDI devices…
MIDI: Closing MME Midi-In device ‘Périphérique audio USB’
MIDI: Shutting down DirectMusic…