Out of Memory Problems - 3GB XP & Large Samples

sys: wxp32sp2 4gigram q6600@4x3ghz

i get “out of memory” warnings in renoise, even if i have some gigabytes of ram free.
so i tested something around, and it seems that my renoise 2.1 only can have a maximum of allocated ram with a amount of about 1.3 gigybte.
(but… i can have two such renoise sessions running simultaneously!)

what i have done for testing:
disabling all virtuel ram disc’s, so wxp have 3gig of ram. 500mb is used by wxp.

i created a 100mb wav file, and tried to load this wav again and again, just 4 testing how much ram renoise can handle.

renoise can load 6 x 100mb of wav files, the sevent can not be loaded (tested with drag&drop and later with copy/paste instrument, both not working)
after the sixth 100mb wav file the renoise intern errormessage appears (see
screenshot), BUT i can load this 1.2gb song in a new renoise instance.
please give some hints or a bugfix, i have some tracks which must be finished in two weeks which needs some big instruments
(not wav’s… i know reaper)

here is the screenshot:

memorybug.png

ps.: if i save the song which needs 1.2 gig of ram with 6x the same wav file the xrns is 500mb, why such bad pack ratio?

edit:
the best at the end:

after reboot renoise won’t load the 500mb xrns file which allocate 1.2 gig of ram (even if this is consisting of 6x100 mb ^^ ) , every instrument with this 100mb wav file is commented during loading with “not enough ram…”

but i still can drag&drop this wav in renoise.
(instruments are 32 bit, intern? but it should load ^^ )

(to be exaxt this wav file is 95mb, 44khz, 16bit, stereo, BUT i found this bug during loading some intakt/kompakt instruments, which needs about 70-150mb each)

I’ve got a similar problem with RAM usage on Renoise. I never tried to load such big WAV files but problem starts when I try to load some VST Romplers. For example, even when I load 1GB piano VST plus two sets of loops on Stylus RMX, I’m getting GUI error message from Renoise. And it’s all when 2GB of ram remains free. This looks like the host has problem with allocating bigger amount of RAM for working.

regarding moltenlove report, it has to be noted that 32bit applications (such as Renoise) cannot allocate more than 3GB of RAM, regardless of how much RAM is available from the system. also, the allocation of a VST DLL can have memory usage peaks which can give problems. you have to take in count that loading a 1GB patch into a VST sampler does not necessarily mean that only 1GB will be used: more likely, 1GB will be added to the RAM which is necessary for the VST sampler to be allocated.

of course, in normal scenarios, where the 3GB limit is far to be reached, nothing bad should happen.

on with the problem pcvf reported, this is of course strange, but what I did not fully understand is how are these samples loaded: do you load the into a Kompakt instance each? or did you have the error when, after loading 5*100MB files, you tried to load a 75MB Kompakt instance?
your are right about sample conversion, though: Renoise will save the files as 32Bit files, and XRNS files are actually renamed ZIP files, so you should not be surprised if a 500MB XRNS needs 1.2GB of RAM to be loaded: try opening the XRNS file with WinRAR, extract the samples from the file and see how much they occupy on the HD.

Yes, thats indeed strange, but up to the OS to decide. We don’t “guess” if we could run out of memory, but just ask the OS to give us some memory. If this fails, well, then we show an out of memory message.
Why the OS doesn’t give Renoise any more memory, has beside of other things also to do with the memory layout/fragmentation. There might be enough memory in summary left, if you count all free spaces in memory, but not a continuously large free block which we need to store a sample.

All we can do here is:

• recommend using the 3GB startup option for windows. See for example http://dwf.blogs.com/beyond_the_paper/2009…dows-vista.html or http://www.google.com/search?q=3GB+startup…lient=firefox-a

• get rid of the problem by allowing to “stream” large samples directly in Renoise, read them from disk instead of pushing them to memory

• until we support large sample streaming, use external VST or ReWire slaves/masters which can do so…

concerning the 3GB switch in boot.ini:
i was using it myself for a long time to have more RAM available for apps, until recently i entirely destroyed my WinXP installation due to the /3GB option.
the sideeffect of that setting is, that your windows registry may not exceed a size of 64MB anymore - if it does, windows will let you know that with a message that no further registry alterations will be taken care of and every “write” to it will be ignored.

the time it took me to find out about why the message is suddenly appearing on my installation ( couple of hours ) after boot, was sufficient to destroy my entire OS.
suddenly i was getting a lot of explorer errors and i couldn’t even get into system configuration specific areas like the user account control or the IPsec settings anymore.

how can i found the size of the registry ?

@taktik & it-alien

i used for this test simply one wav file, which i direct drag&dropped into renoise, without any vsti.

but i found this bug with vsti’s which won’t load. its a very dirty bug; here is a list which i have made some time ago for internal purposes; actual i have real problems with ram in renoise, 1.3 gig is the max!

so if i can only load 5 or 6 of this instruments, its a serious failure, i think.

regarding to fragmented ram: i tested this also after a clean boot.
i also can have 2xrenoise with 1.2gigabyte ram simultanouisly, i mentioned it above.

But do you have the 3GB “boot.ini” option enabled?
The fragmentation happens only in Renoise - the processed you run, not in “real”, so restarting Renoise might help, restarting Windows won’t make a difference…

no, i need the info how i found out the size of my registry.
beside this i doesnt though about; i dont get 2 gbyte for renoise ^^ so 3gbyte is to far away^^

but i’ll test and post the result.

is this streaming memory problem a result of my 2x2gbyte ram’s ?

okay, i added /3GB and success; now i can load 11x100mb wav file, which makes a load of 2.3 gbyte (virtual size) in the taskmanager.

for me its a solution, i hope no other programms are now in troubble.

after some days with tested /3gb switch in boot.ini i have to say: the system becams very unstable. i have had about ten times or more bluscreens, mostly my ati card driver was the cause. i used catalyst driver 9.4 and now 9.6, both without catalyst control center, i use ati tray tools.

i also tried the /3gb with /USERVA=2990, it doesnt help.

for example: if i start grid wihout /3g i get ingame about 90fps, with the /3g option i have 5 frames and massive grafik errors in the game (d3d). with setted the /userva the game gives a blescreen.

so, taktik, the problem is played back to the pogrammer 8)

You can’t blame the application programmer for the bad memory management the OS performs.
An application requests Windows for memory if it needs some, as soon as Windows says “I don’t have more for you” the deal is over and Renoise simply won’t get more. You can ask Taktik to implement any particular armory to threat Windows and demand more, but it will never work unfortunately.

You might want to check the System Paging file options and see how large the paging file is.
Usually it should be 1.5x the real memory, though it does not matter much when Windows only addresses up to 2GB per application.

If you could ReWire one of your plugins if it was a Stand-Alone app (which some plugin developers also offer) this might also give Renoise and your plugin some breathing space.

@vV

i dont wanna blame taktik, i simply tried to told him that the /3gb is not a very good and stable solution.
i have read around: 1.8gb per programm is the maximum which wxp will support in best cases, but i can reach only 1.2 in renoise; the problem seems to be that gigabytes of ram are freally free, but are “fragmented” ? what does this mean technically?

so if a fragment is less than 100mb , and i want to load somethin in renoise that is bigger than 100mb, you get a memory out message from the OS ?

i doesnt really know about the technics behind; but is it possible to allow renoise to use virtual paged memory? would this be not fragmented? i use a ramdisc for pagefile, and for me it would be a solution.

Fragmented memory is like you have a table that you want to put some plates on it.
The table can hold let’s say 25 dishes and there are already 18 dishes on the table, yet they are positioned in a way that you cannot add a dish in between them, so you have to reorder the dishes on the table first before you have a space big enough to add another few dishes.

In graphics it looks similar like this:

For hardrives this is a common problem, but this happens to memory as well.
There are programs that can defragment RAM memory (Like MemTurbo). These kind of programs will improve some of the memory usage but cannot intervene in reserved memory spaces already assigned to programs.

thx 8)
i know what fragmented means, but i doesnt understand why after a fresh boot the 2.5 gigs of ram, which are empty, are fragmented. does this mean a programm (windows driver, gui…) load his code aynwhere at random in the memory? thats not possible, i think.
no other programms would run properly, all other programms would have than the “not enough unfragmented memory” problem.

i will test “mem turbo” and try to load more than the 1.2 gigs of samples (which are effective only 600 mb… and thats what is so bad)

Sorry for bothering you with technical details. Don’t care about this. As long as Renoise is a 32bit application and as long as you are using 32bit VSTs in a 32bit OS, you will quite easily run out of memory when using large samples or a huge amount of small samples with sample VSTs.

Does it really make a difference if you can load up to 1.2 or 1.8 GB of samples?

So the only solution for this is, as said above, for now, to use Direct from Disk streaming sample VSTs. Don’t load everything into memory.
We will in future try to support this in the Renoise internal sampler as well, and will also support 64bit VST and OSs. But until most plugins are only available as 32bit plugins, this doesn’t make any sense, cause we would need to run them “emulated”, which is slow as hell, or you could not use them at all. You can not load a 32bit plugin (and most plugins are 32bit) in a 64bit application without emulation.

One way to already now use 64bit plugins in a 64bit version of windows with a 32bit Renoise, is to use jbridge: http://jstuff.wordpress.com/jbridge/
This will “emulate” 32bit to 64bit plugins and vice versa, so on a 64bit windows, you could run a 32bit Renoise and load 64bit versions of the plugins (if there are some)

Too many 32bit and 64bits in this post. Sorry, but this and will be just complicated.
Don’t know how to explain this more easily…

yes, for me ;(
i already use DfD plugins for all native instruments… but they need the above listed amount of ram with this feature.

and no; its not to technically for me, i try to motivate you to find unusual solutions
(einfach mal den kampgeist des scenecoders entfachen ich arbeite seit nunmehr seit 1992 mit solch talentierten programmieren wie dir zusammen, und ich weiss wenn einer sagt: “das geht nicht, weil…” , gehts meist DOCH, wenn die motivation stimmt sieh es so: dann kann die line mit “professionelles audiosystem” , oder was da stand, wieder in den loadscreen rein weil ich nutz renoise tatsächlich ausschließlich zum arbeiten, und wenn mir was auffällt, was mich beim arbeiten arg behindert, dann sag ich das hier; wurde bisher zwar fast immer ignoriert, aber ich geb nicht auf ^^
auch ist mir das nicht zu technisch, ich hab schon aufm c64 in assembler musikroutinen in den speicher gehackt. bitte behandelt mich nich wie einen ewig unzufriedenen meckerer, der keinen plan hat

Jbridge is only a stopgap though taktik as most developers of DAWs are now developing their own native built in wrapper on the road to 64bit

Taktik is right here and if anybody imagines that their entire range of plugins they use now will be ported to 64bit they are kidding themselves unless they only use plugins from the most active devs around

Truth be told on windows that SE has garned somewhat of a terrible reputation and not because SE is sloppy as a dev environment but because most of the plugins made in SE will never be recompiled with the fixes that have made there way into SE

The same thing will happen with 64bit too and a lot of loved plugins will never make it to 64bit

Having spouted all that crap i would love to see a 64 bit version of Renoise because i see no point at all in stream from disk unless the entire sample engine is updated so that we can make multi zoned multi layered multi random and round robin style stuff
For me personally i cant even see why anyone would use the sample engine in Renoise to make a huge instrument and not just use a sampler

Bungle

I had the “out of memory” error today for the first time. I was trying to load an 88MB mp3 (DJ mix) so I could edit it in the sample editor. Seemed a bit weird as I had 3GB of free ram at the time. I’ll see if it works after a reboot.

It’s not really a problem though because i’ve never run out of memory when making tunes.

88MB mp3, how much was the compression ratio? 88MB of MP3 will be expanded in Renoise to full blown 32-bit float you know?

ahh that explains it then! I thought it was strange that i’d never run out of memory with wavs, but have with mp3s

Thanks a lot